Skip to content

Commit 0e84cc0

Browse files
michaelmoussamichaelmoussa
committed
Initial commit
0 parents  commit 0e84cc0

15 files changed

+3360
-0
lines changed

‎.gitignore

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
*.js
2+
*.d.ts
3+
node_modules
4+
5+
# CDK asset staging directory
6+
.cdk.staging
7+
cdk.out

‎.npmignore

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*.ts
2+
!*.d.ts
3+
4+
# CDK asset staging directory
5+
.cdk.staging
6+
cdk.out

‎CODE_OF_CONDUCT.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
## Code of Conduct
2+
3+
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
4+
For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
5+
[email protected] with any additional questions or comments.

‎CONTRIBUTING.md

+61
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Contributing Guidelines
2+
3+
Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional
4+
documentation, we greatly value feedback and contributions from our community.
5+
6+
Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
7+
information to effectively respond to your bug report or contribution.
8+
9+
10+
## Reporting Bugs/Feature Requests
11+
12+
We welcome you to use the GitHub issue tracker to report bugs or suggest features.
13+
14+
When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already
15+
reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
16+
17+
* A reproducible test case or series of steps
18+
* The version of our code being used
19+
* Any modifications you've made relevant to the bug
20+
* Anything unusual about your environment or deployment
21+
22+
23+
## Contributing via Pull Requests
24+
Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
25+
26+
1. You are working against the latest source on the *master* branch.
27+
2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
28+
3. You open an issue to discuss any significant work - we would hate for your time to be wasted.
29+
30+
To send us a pull request, please:
31+
32+
1. Fork the repository.
33+
2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
34+
3. Ensure local tests pass.
35+
4. Commit to your fork using clear commit messages.
36+
5. Send us a pull request, answering any default questions in the pull request interface.
37+
6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
38+
39+
GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
40+
[creating a pull request](https://help.github.com/articles/creating-a-pull-request/).
41+
42+
43+
## Finding contributions to work on
44+
Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.
45+
46+
47+
## Code of Conduct
48+
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
49+
For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
50+
[email protected] with any additional questions or comments.
51+
52+
53+
## Security issue notifications
54+
If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
55+
56+
57+
## Licensing
58+
59+
See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
60+
61+
We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes.

‎LICENSE

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
3+
Permission is hereby granted, free of charge, to any person obtaining a copy of
4+
this software and associated documentation files (the "Software"), to deal in
5+
the Software without restriction, including without limitation the rights to
6+
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
7+
the Software, and to permit persons to whom the Software is furnished to do so.
8+
9+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
10+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
11+
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
12+
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
13+
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
14+
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

‎README.md

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
## RDS Snapshot Export to S3 Pipeline
2+
3+
This repository creates the automation necessary to export Amazon RDS snapshots to S3 for a specific database whenever an automated snapshot is created.
4+
5+
## Usage
6+
7+
1. Install the [Amazon Cloud Development Kit](https://aws.amazon.com/cdk/) (CDK).
8+
2. Clone this repository and `cd` into it.
9+
3. Modify the arguments to the `RdsSnapshotExportPipelineStack` constructor in `$/bin/cdk.ts` according to your environment.
10+
* `dbName`: This RDS database must already exist.
11+
* `rdsEventId`: This should be `RdsEventId.DB_AUTOMATED_AURORA_SNAPSHOT_CREATED` for Amazon Aurora databases or `RdsEventId.DB_AUTOMATED_SNAPSHOT_CREATED` otherwise.
12+
* `s3BucketName`: An S3 bucket with the provided name will be created automatically for you.
13+
4. Execute the following:
14+
* `npm install`
15+
* `npm run cdk bootstrap`
16+
* `npm run cdk deploy`
17+
5. Open up your `<dbName>-rds-snapshot-exporter` function in the [AWS Lambda](https://console.aws.amazon.com/lambda/home) console and configure a test event using the contents of [$/event.json](./event.json) as a template.
18+
* **NOTE:** The example content is a *subset* of an SNS event notification containing the minimum valid event data necessary to successfully trigger the Lambda function's execution. You should modify the `<SNAPSHOT_NAME>` value within the `Message` key to match an existing RDS snapshot (e.g. `rds:<dbName>-YYYY-MM-DD-hh-mm`). You may also need to modify the `MessageId` if you are attempting to export the same snapshot more than once.
19+
6. Click the **Test** button to start an export.
20+
21+
You can check on the progress of the export in the [Exports in Amazon S3](https://console.aws.amazon.com/rds/home#snapshots-list:tab=exporttos3) listing. When that is finished, you can use the [AWS Glue Crawler](https://console.aws.amazon.com/glue/home#catalog:tab=crawlers) that was created for you to crawl the export, then use [Amazon Athena](https://console.aws.amazon.com/athena/home#query) to perform queries on the exported snapshot.
22+
23+
## Cleanup
24+
25+
Execute `npm run cdk destroy` to delete resources pertaining to this example.
26+
27+
You will also need to delete the following manually:
28+
* The S3 bucket that was created to store the snapshot exports.
29+
* The [CDKToolkit CloudFormation Stack](https://console.aws.amazon.com/cloudformation/home#/stacks?filteringText=CDKToolkit) created by `npm run cdk bootstrap`.
30+
* The `cdktoolkit-stagingbucket-<...>` bucket.
31+
32+
## License
33+
34+
This library is licensed under the MIT-0 License. See the [LICENSE](./LICENSE) file.

‎assets/exporter/main.py

+62
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
import json
2+
import logging
3+
import os
4+
import re
5+
6+
import boto3
7+
8+
"""
9+
Evaluates whether or not the triggering event notification is for an automated
10+
snapshot of the desired DB_NAME, then initiates an RDS snapshot export to S3
11+
task of that snapshot if so.
12+
13+
The function returns the response from the `start_export_task` API call if
14+
it was successful. The function execution will fail if any errors are produced
15+
when making the API call. Otherwise, if the triggering event does not correspond
16+
to the RDS_EVENT_ID or DB_NAME we are expecting to see, the function will return
17+
nothing.
18+
"""
19+
20+
logger = logging.getLogger()
21+
logger.setLevel(os.getenv("LOG_LEVEL", logging.INFO))
22+
23+
24+
def handler(event, context):
25+
if event["Records"][0]["EventSource"] != "aws:sns":
26+
logger.warning(
27+
"This function only supports invocations via SNS events, "
28+
"but was triggered by the following:\n"
29+
f"{json.dumps(event)}"
30+
)
31+
return
32+
33+
logger.debug("EVENT INFO:")
34+
logger.debug(json.dumps(event))
35+
36+
message = json.loads(event["Records"][0]["Sns"]["Message"])
37+
38+
if message["Event ID"].endswith(os.environ["RDS_EVENT_ID"]) and re.match(
39+
"^rds:" + os.environ["DB_NAME"] + "-\d{4}-\d{2}-\d{2}-\d{2}-\d{2}$",
40+
message["Source ID"],
41+
):
42+
export_task_identifier = event["Records"][0]["Sns"]["MessageId"]
43+
account_id = boto3.client("sts").get_caller_identity()["Account"]
44+
response = boto3.client("rds").start_export_task(
45+
ExportTaskIdentifier=(
46+
message["Source ID"][4:27] + '-' + event["Records"][0]["Sns"]["MessageId"]
47+
),
48+
SourceArn=f"arn:aws:rds:{os.environ['AWS_REGION']}:{account_id}:snapshot:{message['Source ID']}",
49+
S3BucketName=os.environ["SNAPSHOT_BUCKET_NAME"],
50+
IamRoleArn=os.environ["SNAPSHOT_TASK_ROLE"],
51+
KmsKeyId=os.environ["SNAPSHOT_TASK_KEY"],
52+
)
53+
response["SnapshotTime"] = str(response["SnapshotTime"])
54+
55+
logger.info("Snapshot export task started")
56+
logger.info(json.dumps(response))
57+
else:
58+
logger.info(f"Ignoring event notification for {message['Source ID']}")
59+
logger.info(
60+
f"Function is configured to accept {os.environ['RDS_EVENT_ID']} "
61+
f"notifications for {os.environ['DB_NAME']} only"
62+
)

‎bin/cdk.ts

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/usr/bin/env node
2+
import 'source-map-support/register';
3+
import * as cdk from '@aws-cdk/core';
4+
import { RdsSnapshotExportPipelineStack, RdsEventId } from '../lib/rds-snapshot-export-pipeline-stack';
5+
6+
const app = new cdk.App();
7+
new RdsSnapshotExportPipelineStack(app, 'RdsSnapshotExportToS3Pipeline', {
8+
dbName: '<existing-rds-database-name>',
9+
rdsEventId: RdsEventId.DB_AUTOMATED_SNAPSHOT_CREATED,
10+
s3BucketName: '<desired-s3-bucket-name>',
11+
});

‎cdk.context.json

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"@aws-cdk/core:enableStackNameDuplicates": "true",
3+
"aws-cdk:enableDiffNoFail": "true"
4+
}

‎cdk.json

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"app": "npx ts-node bin/cdk.ts"
3+
}

‎event.json

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"Records": [
3+
{
4+
"EventSource": "aws:sns",
5+
"Sns": {
6+
"Type": "Notification",
7+
"MessageId": "00000000-0000-0000-0000-000000000000",
8+
"Message": "{\"Event Source\":\"db-snapshot\",\"Source ID\":\"<SNAPSHOT_NAME>\",\"Event ID\":\"http://docs.amazonwebservices.com/AmazonRDS/latest/UserGuide/USER_Events.html#RDS-EVENT-0091\"}"
9+
}
10+
}
11+
]
12+
}

0 commit comments

Comments
 (0)