Skip to content
This repository was archived by the owner on Jul 16, 2024. It is now read-only.

Commit 2f6c1a9

Browse files
vgkowskiAutomation
vgkowski
and
Automation
authored
Feature/data generator (#109)
* feature: data generator Co-authored-by: Automation <[email protected]>
1 parent 39bb810 commit 2f6c1a9

35 files changed

+3466
-621
lines changed

‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -24,3 +24,4 @@ core/node_modules/
2424
*.log
2525
doc/site
2626
__init__.py
27+
core/.github

‎.vscode/launch.json

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"configurations": [
3+
{
4+
"type": "aws-sam",
5+
"request": "direct-invoke",
6+
"name": "data-generator-crawler-start-wait:crawler-start-wait.on_update (python3.8)",
7+
"invokeTarget": {
8+
"target": "code",
9+
"projectRoot": "${workspaceFolder}/core/src/lambdas/data-generator-crawler-start-wait",
10+
"lambdaHandler": "crawler-start-wait.on_update"
11+
},
12+
"lambda": {
13+
"runtime": "python3.8",
14+
"payload": {},
15+
"environmentVariables": {}
16+
}
17+
},
18+
{
19+
"type": "aws-sam",
20+
"request": "direct-invoke",
21+
"name": "data-generator-crawler-start-wait:crawler-start-wait.on_update (python3.8)",
22+
"invokeTarget": {
23+
"target": "code",
24+
"projectRoot": "${workspaceFolder}/core/src/lambdas/data-generator-crawler-start-wait",
25+
"lambdaHandler": "crawler-start-wait.on_update"
26+
},
27+
"lambda": {
28+
"runtime": "python3.8",
29+
"payload": {},
30+
"environmentVariables": {}
31+
}
32+
}
33+
]
34+
}

‎CONTRIBUTING.md

-65
Original file line numberDiff line numberDiff line change
@@ -43,71 +43,6 @@ If you'd like to contribute, but don't have a project in mind, look at the [open
4343

4444
In addition to written content, we really appreciate new examples and code samples for our documentation, such as examples for different platforms or environments, and code samples in additional languages.
4545

46-
## Contribution best practices
47-
48-
### Projen setup for Core components
49-
50-
1. Go into `core` folder
51-
2. Install projen locally. Sometimes `projen` is installed globally and building throwns an error, deleting global `node_modules` folder solves the issue
52-
53-
`npm install projen`
54-
3. Build the core artificats
55-
56-
`npx projen build`
57-
4. Only run unit test
58-
59-
`npx projen test`
60-
61-
### Git branch strategy
62-
63-
Create a branch based on the type of contribution:
64-
* build: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
65-
* ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
66-
* docs: Documentation only changes
67-
* feat: A new feature
68-
* fix: A bug fix
69-
* perf: A code change that improves performance
70-
* refactor: A code change that neither fixes a bug nor adds a feature
71-
* style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
72-
* test: Adding missing tests or correcting existing tests
73-
74-
### Git commit strategy
75-
76-
Put relevant type of contribution in the commit message. This message is used to determine the release type when merged into the `main` branch:
77-
* build: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
78-
* ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
79-
* docs: Documentation only changes
80-
* feat: A new feature. Merging in `main` creates a new minor version
81-
* fix: A bug fix. Merging in `main` creates a new patched version
82-
* perf: A code change that improves performance
83-
* refactor: A code change that neither fixes a bug nor adds a feature
84-
* style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
85-
* test: Adding missing tests or correcting existing tests
86-
* BREAKING CHANGE: introduces a breaking API change. Merging in `main` creates a new major version
87-
88-
`<type>: <description>`
89-
90-
### Documentation
91-
92-
* Typedoc (https://typedoc.org/guides/doccomments/) for code comments and API documentation in the framework (./core). See the `ExampleProps` interface and `Example` class [here](https://github.com/aws-samples/aws-analytics-reference-architecture/blob/main/core/src/example.ts)
93-
* Pdoc3 (https://pdoc3.github.io/pdoc/) for code comments and API documentation of reference architectures (./refarch)
94-
95-
### Coding best practices and patterns
96-
97-
Please refer to the official CDK guidelines (https://github.com/aws/aws-cdk/blob/master/docs/DESIGN_GUIDELINES.md) for coding. An Example Construct is available in the repository to help you onboard.
98-
99-
#### Customizable Constructs defaults parameters
100-
101-
AWS CDK Constructs are abstractions with built-in best practices that can be overriden with custom configurations to allow extensibility. Use optional Props parameters in the interface and defaults Props values in the CDK Construct constructor. Document default values in the Props interface documentation.
102-
103-
See the `ExampleProps` interface and `Example` class [here](https://github.com/aws-samples/aws-analytics-reference-architecture/blob/main/core/src/example.ts). `name` and `value` are optional parameters of the interface. In the `Example` class constructor, `name` and `value` existance are tested and default parameters are used if they are not provided to the Props.
104-
105-
#### Pre-defined CDK objects
106-
107-
Use Typescript `public static readonly` objects to store pre-defined objects like available Datasets.
108-
109-
TODO: add example
110-
11146
## Code of conduct
11247

11348
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). For more information, see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact [[email protected]](mailto:[email protected]) with any additional questions or comments.

‎CONTRIB_FAQ.md

+88
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Contribution best practices
2+
3+
## Project build
4+
5+
### How to setup Projen for Core components?
6+
7+
1. Go into `core` folder
8+
2. Install projen locally. Sometimes `projen` is installed globally and building throwns an error, deleting global `node_modules` folder solves the issue
9+
10+
`npm install projen`
11+
3. Build the core artificats
12+
13+
`npx projen build`
14+
4. Only run unit test
15+
16+
`npx projen test`
17+
18+
## Git strategy
19+
20+
### What Git branch strategy to use?
21+
22+
Create a branch based on the type of contribution:
23+
* build: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
24+
* ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
25+
* docs: Documentation only changes
26+
* feat: A new feature
27+
* fix: A bug fix
28+
* perf: A code change that improves performance
29+
* refactor: A code change that neither fixes a bug nor adds a feature
30+
* style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
31+
* test: Adding missing tests or correcting existing tests
32+
33+
### Which message to use in Git commit?
34+
35+
Put relevant type of contribution in the commit message. This message is used to determine the release type when merged into the `main` branch:
36+
* build: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
37+
* ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
38+
* docs: Documentation only changes
39+
* feat: A new feature. Merging in `main` creates a new minor version
40+
* fix: A bug fix. Merging in `main` creates a new patched version
41+
* perf: A code change that improves performance
42+
* refactor: A code change that neither fixes a bug nor adds a feature
43+
* style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
44+
* test: Adding missing tests or correcting existing tests
45+
* BREAKING CHANGE: introduces a breaking API change. Merging in `main` creates a new major version
46+
47+
`<type>: <description>`
48+
49+
## Documentation
50+
51+
### How to document code?
52+
53+
* Typedoc (https://typedoc.org/guides/doccomments/) for code comments and API documentation in the framework (./core). See the `ExampleProps` interface and `Example` class [here](https://github.com/aws-samples/aws-analytics-reference-architecture/blob/main/core/src/example.ts)
54+
* Pdoc3 (https://pdoc3.github.io/pdoc/) for code comments and API documentation of reference architectures (./refarch)
55+
56+
## CDK best practices
57+
58+
### What are the coding best practices and patterns with CDK
59+
60+
Please refer to the official CDK guidelines (https://github.com/aws/aws-cdk/blob/master/docs/DESIGN_GUIDELINES.md) for coding. An Example Construct is available in the repository to help you onboard.
61+
62+
#### How to implement Constructs with defaults but customizable parameters?
63+
64+
AWS CDK Constructs are abstractions with built-in best practices that can be overriden with custom configurations to allow extensibility. Use optional Props parameters in the interface and defaults Props values in the CDK Construct constructor. Document default values in the Props interface documentation.
65+
66+
See the `ExampleProps` interface and `Example` class [here](https://github.com/aws-samples/aws-analytics-reference-architecture/blob/main/core/src/example.ts). `name` and `value` are optional parameters of the interface. In the `Example` class constructor, `name` and `value` existance are tested and default parameters are used if they are not provided to the Props.
67+
68+
#### How to provide pre-defined CDK objects?
69+
70+
Use Typescript `public static readonly` objects to store pre-defined objects like available Datasets.
71+
72+
See the `Dataset` class using static variables call the class constructor [here](./core/src/dataset.ts).
73+
74+
### How to implement Singleton pattern for an AWS CDK resource
75+
76+
The singleton pattern can be implemented using the unique ID of the AWS CDK node. Instead of creating a new resource from the AWS CDK Construct, a static `getOrCreate` method is used to retrieve the resource by search for the unique ID in the AWS CDK Scope. If no resource exists, the method creates a new one.
77+
78+
See the `SingletonBucket` Construct [here](./core/src/singleton-bucket.ts).
79+
80+
## Testing
81+
82+
#### How to test CDK Constructs logic with a deployment in AWS Account?
83+
84+
AWS CDK logic can be tested by deploying a stack that instantiates AWS CDK components. Projen is configured to provide extra actions `test:deploy` and `test:destroy` to deploy in a testing account and destroy. [`./core/src/integ.default.ts`](./core/src/integ.default.ts) can be customized to deploy Constructs.
85+
86+
### How to test private class members or methods in Typescript?
87+
88+
Typescript allows to access private members and methods using this syntax `customDataset["sqlTable"]()`. See example of testing `sqlTable()`private method from `Dataset` class [here](./core/test/dataset.test.ts)

‎core/.projen/deps.json

+107-7
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"dependencies": [
33
{
44
"name": "@aws-cdk/assert",
5-
"version": "^1.119",
5+
"version": "^1.119.0",
66
"type": "build"
77
},
88
{
@@ -95,39 +95,139 @@
9595
"version": "github:xmldom/xmldom#0.7.0",
9696
"type": "bundled"
9797
},
98+
{
99+
"name": "@aws-cdk/aws-athena",
100+
"version": "^1.119.0",
101+
"type": "peer"
102+
},
103+
{
104+
"name": "@aws-cdk/aws-events-targets",
105+
"version": "^1.119.0",
106+
"type": "peer"
107+
},
108+
{
109+
"name": "@aws-cdk/aws-events",
110+
"version": "^1.119.0",
111+
"type": "peer"
112+
},
113+
{
114+
"name": "@aws-cdk/aws-glue",
115+
"version": "^1.119.0",
116+
"type": "peer"
117+
},
98118
{
99119
"name": "@aws-cdk/aws-iam",
100-
"version": "^1.119",
120+
"version": "^1.119.0",
121+
"type": "peer"
122+
},
123+
{
124+
"name": "@aws-cdk/aws-kinesis",
125+
"version": "^1.119.0",
126+
"type": "peer"
127+
},
128+
{
129+
"name": "@aws-cdk/aws-lambda",
130+
"version": "^1.119.0",
131+
"type": "peer"
132+
},
133+
{
134+
"name": "@aws-cdk/aws-logs",
135+
"version": "^1.119.0",
101136
"type": "peer"
102137
},
103138
{
104139
"name": "@aws-cdk/aws-s3",
105-
"version": "^1.119",
140+
"version": "^1.119.0",
141+
"type": "peer"
142+
},
143+
{
144+
"name": "@aws-cdk/aws-stepfunctions-tasks",
145+
"version": "^1.119.0",
146+
"type": "peer"
147+
},
148+
{
149+
"name": "@aws-cdk/aws-stepfunctions",
150+
"version": "^1.119.0",
106151
"type": "peer"
107152
},
108153
{
109154
"name": "@aws-cdk/core",
110-
"version": "^1.119",
155+
"version": "^1.119.0",
156+
"type": "peer"
157+
},
158+
{
159+
"name": "@aws-cdk/custom-resources",
160+
"version": "^1.119.0",
111161
"type": "peer"
112162
},
113163
{
114164
"name": "constructs",
115165
"version": "^3.2.27",
116166
"type": "peer"
117167
},
168+
{
169+
"name": "@aws-cdk/aws-athena",
170+
"version": "^1.119.0",
171+
"type": "runtime"
172+
},
173+
{
174+
"name": "@aws-cdk/aws-events-targets",
175+
"version": "^1.119.0",
176+
"type": "runtime"
177+
},
178+
{
179+
"name": "@aws-cdk/aws-events",
180+
"version": "^1.119.0",
181+
"type": "runtime"
182+
},
183+
{
184+
"name": "@aws-cdk/aws-glue",
185+
"version": "^1.119.0",
186+
"type": "runtime"
187+
},
118188
{
119189
"name": "@aws-cdk/aws-iam",
120-
"version": "^1.119",
190+
"version": "^1.119.0",
191+
"type": "runtime"
192+
},
193+
{
194+
"name": "@aws-cdk/aws-kinesis",
195+
"version": "^1.119.0",
196+
"type": "runtime"
197+
},
198+
{
199+
"name": "@aws-cdk/aws-lambda",
200+
"version": "^1.119.0",
201+
"type": "runtime"
202+
},
203+
{
204+
"name": "@aws-cdk/aws-logs",
205+
"version": "^1.119.0",
121206
"type": "runtime"
122207
},
123208
{
124209
"name": "@aws-cdk/aws-s3",
125-
"version": "^1.119",
210+
"version": "^1.119.0",
211+
"type": "runtime"
212+
},
213+
{
214+
"name": "@aws-cdk/aws-stepfunctions-tasks",
215+
"version": "^1.119.0",
216+
"type": "runtime"
217+
},
218+
{
219+
"name": "@aws-cdk/aws-stepfunctions",
220+
"version": "^1.119.0",
126221
"type": "runtime"
127222
},
128223
{
129224
"name": "@aws-cdk/core",
130-
"version": "^1.119",
225+
"version": "^1.119.0",
226+
"type": "runtime"
227+
},
228+
{
229+
"name": "@aws-cdk/custom-resources",
230+
"version": "^1.119.0",
131231
"type": "runtime"
132232
}
133233
],

‎core/.projen/tasks.json

+16
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,22 @@
216216
"exec": "jsii-docgen"
217217
}
218218
]
219+
},
220+
"test:deploy": {
221+
"name": "test:deploy",
222+
"steps": [
223+
{
224+
"exec": "cdk deploy --app=./lib/integ.default.js"
225+
}
226+
]
227+
},
228+
"test:destroy": {
229+
"name": "test:destroy",
230+
"steps": [
231+
{
232+
"exec": "cdk destroy --app=./lib/integ.default.js"
233+
}
234+
]
219235
}
220236
},
221237
"env": {

0 commit comments

Comments
 (0)