Skip to content

Commit b43d7eb

Browse files
ahegdeNRhrai-nr
ahegdeNR
and
hrai-nr
authored
[NR-383508] Sync develop branch with main branch (#24)
* modified lambda-template with /src as codeUri. Region wise S3 gets added as part of CI/CD * modified how the yaml file changes by adding a script * added lambda trigger script and some common stack script to validate stack creation * added cloudwatch trigger script * added ci/cd automation to run tests during merge * fixed missing permission to fetch secrets from github secrets * fixed constants for scripts * moved license key to github secrets * fixed filter pattern json format in template parameters * fixes * modified bucket name and log group name, build all templates * code refactoring and fixes * fixes * fixes * updated permissions for common script * removed other templates from getting packaged and published as part of this task * modified parameters file name in triggers * removed parallelization of script * removed parallelization of script * fixes * fixes * fixes * add parallelization to running scripts * modified e2e tests to run during push workflow * review fixes, comments and retries * review fixes, comments and retries * created events for cloudwatch, s3 and validating logs are present in new relic * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * parsing requestId from message and stamping it as attribute (#12) RequestId changes for Text and JSON formatted log messages from AWS cloudwatch * fixes * fixes * fixes * testing on pull request * testing on pull request * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * parallelised template build and made it run once only for the tests * fixes * send slack notification on failure * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * added e2e tests to run once every month as cron job * [NR-352205] E2E tests to validate metrics with logging integration templates (#22) * added tests for metrics and lambda stacks * fixes * reverted change made to test * parallelized running tests * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * fixes * reverting previous changes post testing * removed chmod command * moving version number to a different file named version.go * updated entity_synthesis_param.cfg file with version change and new line * [NR-383508] Fix validate template code in CI/CD (#29) * fixed validate template step and lambda-template.yaml validate errors * removed slack notification and made e2e tests run on pull request * added develop branch to test e2e tests * removing changes made for testing --------- Co-authored-by: hrai-nr <[email protected]>
1 parent ad086ef commit b43d7eb

20 files changed

+1613
-87
lines changed

‎.github/workflows/pull_request.yaml

+4-3
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ name: Pull Request Workflow
33
on:
44
pull_request:
55
branches:
6+
- develop
67
- main
78

89
jobs:
@@ -14,7 +15,7 @@ jobs:
1415

1516
- name: Install AWS SAM CLI
1617
run: |
17-
pip install aws-sam-cli
18+
pip install aws-sam-cli==1.134.0
1819
1920
- name: Install golint
2021
run: |
@@ -56,7 +57,7 @@ jobs:
5657
5758
- name: Validate SAM Templates
5859
run: |
59-
for template in $(find . -name ".yaml" -o -name ".yml"); do
60+
for template in $(find . -maxdepth 1 \( -name "*.yaml" -o -name "*.yml" \)); do
6061
echo "Validating template: $template"
6162
sam validate --template-file "$template" --region us-east-2 --lint
6263
done
@@ -124,7 +125,7 @@ jobs:
124125
go tool cover -html=coverage/coverage.out -o coverage/coverage.html
125126
126127
- name: Upload Coverage Report
127-
uses: actions/upload-artifact@v3
128+
uses: actions/upload-artifact@v4
128129
with:
129130
name: coverage-report
130131
path: src/coverage/coverage.html

‎.github/workflows/release-lambda-code.yaml

+2-2
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ jobs:
3434
cd ..
3535
3636
- name: Upload artifact
37-
uses: actions/upload-artifact@v3
37+
uses: actions/upload-artifact@v4
3838
with:
3939
name: log-forwarder-zip
4040
path: |
@@ -82,7 +82,7 @@ jobs:
8282

8383
steps:
8484
- name: Download Artifact
85-
uses: actions/download-artifact@v3
85+
uses: actions/download-artifact@v4
8686
with:
8787
name: log-forwarder-zip
8888
path: ./build-artifacts

‎.github/workflows/release-template-files.yaml

+17
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,23 @@ jobs:
2626
run: |
2727
pip install aws-sam-cli
2828
29+
# modify lambda CodeUri to production as s3 bucket destination
30+
- name: Set up Python
31+
uses: actions/setup-python@v2
32+
with:
33+
python-version: '3.x'
34+
35+
- name: Install ruamel
36+
run: |
37+
python -m pip install --upgrade pip
38+
pip install ruamel.yaml
39+
40+
- name: Run Python Script to Update lambda-template.yaml
41+
run: python workflow-scripts/update_lambda_template.py
42+
43+
- name: Verify the changes
44+
run: cat lambda-template.yaml
45+
2946
- name: Configure AWS Credentials
3047
uses: aws-actions/configure-aws-credentials@v3
3148
with:

‎.github/workflows/run-e2e-tests.yaml

+241
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
name: E2E Test Workflow
2+
3+
on:
4+
pull_request_review:
5+
types:
6+
- submitted
7+
schedule:
8+
- cron: '0 0 1 * *'
9+
10+
jobs:
11+
build-templates:
12+
if: github.event.review.state == 'approved'
13+
runs-on: ubuntu-latest
14+
permissions:
15+
id-token: write
16+
contents: write
17+
strategy:
18+
matrix:
19+
TEMPLATE_FILE: [lambda-template, logging-lambda-firehose-template, logging-lambda-metric-polling, logging-lambda-metric-stream, logging-firehose-metric-polling, logging-firehose-metric-stream, logging-lambda-firehose-metric-polling, logging-lambda-firehose-metric-stream]
20+
steps:
21+
- name: Checkout code
22+
uses: actions/checkout@v4
23+
24+
- name: Setup Go
25+
uses: actions/setup-go@v5
26+
with:
27+
go-version: '1.22'
28+
29+
- name: Install AWS SAM CLI
30+
run: |
31+
pip install aws-sam-cli
32+
33+
- name: Configure AWS Credentials
34+
uses: aws-actions/configure-aws-credentials@v3
35+
with:
36+
role-to-assume: ${{ secrets.AWS_E2E_ROLE }}
37+
aws-region: us-east-1
38+
39+
- name: Build SAM Application
40+
env:
41+
S3_BUCKET: unified-lambda-e2e-test-templates
42+
run: |
43+
sam build -u --template-file "${{ matrix.TEMPLATE_FILE }}.yaml" --build-dir ".aws-sam/build/${{ matrix.TEMPLATE_FILE }}"
44+
sam package --s3-bucket "$S3_BUCKET" --template-file ".aws-sam/build/${{ matrix.TEMPLATE_FILE }}/template.yaml" --output-template-file ".aws-sam/build/${{ matrix.TEMPLATE_FILE }}/${{ matrix.TEMPLATE_FILE }}.yaml"
45+
46+
- name: Upload Artifact
47+
uses: actions/upload-artifact@v4
48+
with:
49+
name: ${{ matrix.TEMPLATE_FILE }}.yaml
50+
path: .aws-sam/build/${{ matrix.TEMPLATE_FILE }}/${{ matrix.TEMPLATE_FILE }}.yaml
51+
52+
- name: Send failure notification to Slack
53+
if: always()
54+
uses: ravsamhq/notify-slack-action@v1
55+
with:
56+
status: ${{ job.status }}
57+
notify_when: 'failure'
58+
env:
59+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
60+
61+
run-e2e-tests-cloudwatch:
62+
needs: [build-templates]
63+
runs-on: ubuntu-latest
64+
permissions:
65+
id-token: write
66+
contents: write
67+
strategy:
68+
matrix:
69+
test-case: [test_logs_with_filter_pattern, test_logs_for_secret_manager, test_logs_for_invalid_log_group]
70+
steps:
71+
- name: Checkout code
72+
uses: actions/checkout@v4
73+
74+
- name: Download Artifact
75+
uses: actions/download-artifact@v4
76+
with:
77+
path: build-artifacts
78+
79+
- name: Setup Go
80+
uses: actions/setup-go@v5
81+
with:
82+
go-version: '1.22'
83+
84+
- name: Install AWS SAM CLI
85+
run: |
86+
pip install aws-sam-cli
87+
88+
- name: Configure AWS Credentials
89+
uses: aws-actions/configure-aws-credentials@v3
90+
with:
91+
role-to-assume: ${{ secrets.AWS_E2E_ROLE }}
92+
aws-region: us-east-1
93+
94+
- name: Run e2e tests for cloudwatch
95+
env:
96+
NEW_RELIC_USER_KEY: ${{ secrets.NEW_RELIC_USER_KEY }}
97+
NEW_RELIC_LICENSE_KEY: ${{ secrets.NEW_RELIC_LICENSE_KEY }}
98+
run: |
99+
cd e2e-tests
100+
./lambda-cloudwatch-trigger.sh ${{ matrix.test-case }}
101+
102+
- name: Send failure notification to Slack
103+
if: always()
104+
uses: ravsamhq/notify-slack-action@v1
105+
with:
106+
status: ${{ job.status }}
107+
notify_when: 'failure'
108+
env:
109+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
110+
111+
run-e2e-tests-s3:
112+
needs: [build-templates]
113+
runs-on: ubuntu-latest
114+
permissions:
115+
id-token: write
116+
contents: write
117+
strategy:
118+
matrix:
119+
test-case: [test_logs_for_prefix, test_logs_for_secret_manager, test_logs_for_invalid_bucket_name]
120+
steps:
121+
- name: Checkout code
122+
uses: actions/checkout@v4
123+
124+
- name: Download Artifact
125+
uses: actions/download-artifact@v4
126+
with:
127+
path: build-artifacts
128+
129+
- name: Setup Go
130+
uses: actions/setup-go@v5
131+
with:
132+
go-version: '1.22'
133+
134+
- name: Install AWS SAM CLI
135+
run: |
136+
pip install aws-sam-cli
137+
138+
- name: Configure AWS Credentials
139+
uses: aws-actions/configure-aws-credentials@v3
140+
with:
141+
role-to-assume: ${{ secrets.AWS_E2E_ROLE }}
142+
aws-region: us-east-1
143+
144+
- name: Run e2e tests for s3
145+
env:
146+
NEW_RELIC_USER_KEY: ${{ secrets.NEW_RELIC_USER_KEY }}
147+
NEW_RELIC_LICENSE_KEY: ${{ secrets.NEW_RELIC_LICENSE_KEY }}
148+
run: |
149+
cd e2e-tests
150+
./lambda-s3-trigger.sh ${{ matrix.test-case }}
151+
152+
- name: Send failure notification to Slack
153+
if: always()
154+
uses: ravsamhq/notify-slack-action@v1
155+
with:
156+
status: ${{ job.status }}
157+
notify_when: 'failure'
158+
env:
159+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
160+
161+
run-e2e-tests-lambda-firehose-metrics:
162+
needs: [build-templates]
163+
runs-on: ubuntu-latest
164+
permissions:
165+
id-token: write
166+
contents: write
167+
strategy:
168+
matrix:
169+
test-case: [test_for_lambda_firehose_stack, test_for_firehose_metric_polling_stack, test_for_firehose_metric_streaming_stack, test_for_lambda_metrics_polling_stack, test_for_lambda_metrics_streaming_stack, test_for_lambda_firehose_metric_polling_stack, test_for_lambda_firehose_metric_streaming_stack]
170+
steps:
171+
- name: Checkout code
172+
uses: actions/checkout@v4
173+
174+
- name: Download Artifact
175+
uses: actions/download-artifact@v4
176+
with:
177+
path: build-artifacts
178+
179+
- name: Setup Go
180+
uses: actions/setup-go@v5
181+
with:
182+
go-version: '1.22'
183+
184+
- name: Install AWS SAM CLI
185+
run: |
186+
pip install aws-sam-cli
187+
188+
- name: Configure AWS Credentials
189+
uses: aws-actions/configure-aws-credentials@v3
190+
with:
191+
role-to-assume: ${{ secrets.AWS_E2E_ROLE }}
192+
aws-region: us-east-1
193+
194+
- name: Run e2e tests for lambda and metrics stack
195+
env:
196+
NEW_RELIC_USER_KEY: ${{ secrets.NEW_RELIC_USER_KEY }}
197+
NEW_RELIC_LICENSE_KEY: ${{ secrets.NEW_RELIC_LICENSE_KEY }}
198+
run: |
199+
cd e2e-tests/
200+
./lambda-with-metrics-test.sh ${{ matrix.test-case }}
201+
202+
- name: Send failure notification to Slack
203+
if: always()
204+
uses: ravsamhq/notify-slack-action@v1
205+
with:
206+
status: ${{ job.status }}
207+
notify_when: 'failure'
208+
env:
209+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
210+
211+
clean-up:
212+
needs: [run-e2e-tests-cloudwatch, run-e2e-tests-s3, run-e2e-tests-lambda-firehose-metrics]
213+
runs-on: ubuntu-latest
214+
permissions:
215+
id-token: write
216+
contents: write
217+
steps:
218+
- name: Install AWS SAM CLI
219+
run: |
220+
pip install aws-sam-cli
221+
222+
- name: Configure AWS Credentials
223+
uses: aws-actions/configure-aws-credentials@v3
224+
with:
225+
role-to-assume: ${{ secrets.AWS_E2E_ROLE }}
226+
aws-region: us-east-1
227+
228+
- name: Delete Resources
229+
env:
230+
S3_BUCKET: unified-lambda-e2e-test-templates
231+
run:
232+
aws s3 rm "s3://$S3_BUCKET" --recursive
233+
234+
- name: Send failure notification to Slack
235+
if: always()
236+
uses: ravsamhq/notify-slack-action@v1
237+
with:
238+
status: ${{ job.status }}
239+
notify_when: 'failure'
240+
env:
241+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

‎e2e-tests/common-scripts/entity_synthesis_param.cfg

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Entity synthesis Parameters , aws related params are dropped in pipeline.
2+
instrumentation_provider=aws
3+
instrumentation_name=lambda
4+
instrumentation_version=1.1.0

0 commit comments

Comments
 (0)