fix: unique lambda function name in prebundled function (#395)

Author: vgkowski

.gitignore

@@ -21,6 +21,7 @@ refarch/aws-native/dwh/dwh_loader_layer
 refarch/aws-native/streaming/streaming_cdk/lambda-layer
 refarch/aws-native/streaming/stream-processing/target
 core/node_modules/
+core/src/integ.default.ts
 *.log
 doc/site
 __init__.py

CONTRIB_FAQ.md

@@ -120,6 +120,52 @@ Put relevant type of contribution in the commit message. This message is used to
 * Typedoc (https://typedoc.org/guides/doccomments/) for code comments and API documentation in the framework (./core). See the `ExampleProps` interface and `Example` class [here](https://github.com/aws-samples/aws-analytics-reference-architecture/blob/main/core/src/example.ts)
 * Pdoc3 (https://pdoc3.github.io/pdoc/) for code comments and API documentation of reference architectures (./refarch)
 
+Examples can be found in the current code documentation used by [Construct Hub](https://constructs.dev/packages/aws-analytics-reference-architecture). For example here is the code comment for the DataLakeStorage construct:
+
+```
+/**
+ * A CDK Construct that creates the storage layers of a data lake composed of Amazon S3 Buckets.
+ *
+ * This construct is based on 3 Amazon S3 buckets configured with AWS best practices:
+ *  * S3 buckets for Raw/Cleaned/Transformed data,
+ *  * data lifecycle optimization/transitioning to different Amazon S3 storage classes
+ *  * server side buckets encryption managed by KMS customer key
+ *  * Default single KMS key
+ *  * SSL communication enforcement
+ *  * access logged to an S3 bucket
+ *  * All public access blocked
+ *
+ * By default the transitioning rules to Amazon S3 storage classes are configured as following:
+ *  * Raw data is moved to Infrequent Access after 30 days and archived to Glacier after 90 days
+ *  * Clean and Transformed data is moved to Infrequent Access after 90 days and is not archived
+ *
+ * Objects and buckets are automatically deleted when the CDK application is detroyed.
+ *
+ * For custom requirements, consider using {@link AraBucket}.
+ *
+ * Usage example:
+ * ```typescript
+ * import * as cdk from 'aws-cdk-lib';
+ * import { DataLakeStorage } from 'aws-analytics-reference-architecture';
+ *
+ * const exampleApp = new cdk.App();
+ * const stack = new cdk.Stack(exampleApp, 'DataLakeStorageStack');
+ *
+ * new DataLakeStorage(stack, 'MyDataLakeStorage', {
+ *  rawInfrequentAccessDelay: 90,
+ *  rawArchiveDelay: 180,
+ *  cleanInfrequentAccessDelay: 180,
+ *  cleanArchiveDelay: 360,
+ *  transformInfrequentAccessDelay: 180,
+ *  transformArchiveDelay: 360,
+ * });
+ * ```
+ */
+```
+
+And [how it renders in Construct Hub](https://constructs.dev/packages/aws-analytics-reference-architecture/v/2.1.0/api/DataLakeStorage?lang=python).
+
+
 ## CDK best practices
 
 ### What are the coding best practices and patterns with CDK
@@ -267,13 +313,18 @@ Thus, we choose to prebundle Lambda function by installing all dependencies on p
 
 ## Testing
 
-#### How to test CDK Constructs logic with a deployment in AWS Account?
+### What are the required tests to implement?
 
-AWS CDK logic can be tested by deploying a stack that instantiates AWS CDK components. Projen is configured to provide extra actions `test:deploy` and `test:destroy` to deploy in a testing account and destroy. [`./core/src/integ.default.ts`](./core/src/integ.default.ts) can be customized to deploy Constructs.
+It's required to implement 3 types of test:
+ * Unit tests (in `core/test/unit`): validate the CloudFormation that is generated by CDK
+ * CDK-nag tests (in `core/test/unit/cdk-nag`): validate the CDK resources with [CDK-nag](https://github.com/cdklabs/cdk-nag) and the [AWS Solutions rule pack](https://github.com/cdklabs/cdk-nag/blob/main/RULES.md#awssolutions)
+ * End-to-end tests (in `core/test/e2e`): validate the CDK resources can be deployed in an account, and the resources are running as expected. **It's strongly recommended to test the deployment of 2 resources to validate the unicity of CDK IDs and resource names**
 
-### How to test private class members or methods in Typescript?
+### How to test CDK Constructs logic with a deployment in AWS Account?
+
+AWS CDK logic can be tested by deploying a stack that instantiates AWS CDK components. Projen is configured to provide extra actions `test:deploy` and `test:destroy` to deploy in a testing account and destroy. [`./core/src/integ.default.ts`](./core/src/integ.default.ts) can be customized to deploy Constructs.
 
-Typescript allows to access private members and methods using this syntax `customDataset["sqlTable"]()`. See example of testing `sqlTable()`private method from `Dataset` class [here](./core/test/dataset.test.ts)
+Remember to not commit any personal information like account IDs and role name in this file.
 
 ### How to test the core components library in local
 
@@ -283,12 +334,14 @@ Typescript allows to access private members and methods using this syntax `custo
 npx projen package
 ```
 
- * For Python, create a new AWS CDK application in Python outside of this project and install a local dependency pointing to the `wheel` file in `core/dist/python`. In the `setup.py`, modify the dependencies like this
+ * For Python, create a new AWS CDK application in Python outside of this project and install a local dependency pointing to the `wheel` file in `core/dist/python`. In the `requirements.txt`, modify the dependencies like this
 
-```python
-    install_requires=[
-        "aws-cdk.core==1.130.0",
-        f"aws_analytics_reference_architecture @ file:///localhost<LOCAL_PATH>/aws-analytics-reference-architecture/core/dist/python/aws_analytics_reference_architecture-0.0.0-py3-none-any.whl",
-    ],
 ```
+aws-cdk.core==2.27.0
+constructs>=10.0.0,<11.0.0
+<LOCAL_PATH>/aws-analytics-reference-architecture/core/dist/python/aws_analytics_reference_architecture-0.0.0-py3-none-any.whl
+```
+
+### How to test private class members or methods in Typescript?
 
+Typescript allows to access private members and methods using this syntax `customDataset["sqlTable"]()`. See example of testing `sqlTable()`private method from `Dataset` class [here](./core/test/dataset.test.ts)

core/src/common/pre-bundled-function.ts

@@ -3,7 +3,7 @@
 
 import * as path from 'path';
 import { Effect, ManagedPolicy, PolicyStatement, Role, ServicePrincipal } from 'aws-cdk-lib/aws-iam';
-import { Code, Function, FunctionProps, ILayerVersion } from 'aws-cdk-lib/aws-lambda';
+import { Code, Function, FunctionProps } from 'aws-cdk-lib/aws-lambda';
 import { Aws } from 'aws-cdk-lib';
 // import { PreBundledLayer } from './pre-bundled-layer';
 import { Construct } from 'constructs';
@@ -15,9 +15,7 @@ import { Construct } from 'constructs';
  */
 export interface PreBundledFunctionProps extends Partial<FunctionProps> {
   codePath: string;
-  name: string;
   lambdaPolicyStatements?: PolicyStatement[];
-  lambdaLayers?: ILayerVersion[];
 }
 
 /**
@@ -41,14 +39,13 @@ export interface PreBundledFunctionProps extends Partial<FunctionProps> {
  *
  * new PreBundledFunction(this, 'PreBundledFunction', {
  *   codePath: 'construct-dir/resources/lambdas/lambda_dir',
+ *   lambdaPolicyStatements: findFilePathsFnPolicy,
  *   // you can use any property available in Function CDK Construct including
- *   name: 'myFunctionName',
  *   memorySize: 1024,
  *   runtime: Runtime.PYTHON_3_8,
  *   handler: 'lmabda-file-name.handler',
  *   logRetention: RetentionDays.ONE_WEEK,
  *   timeout: Duration.minutes(15),
- *   lambdaPolicyStatements: findFilePathsFnPolicy,
  * });
  * ```
  */
@@ -73,7 +70,7 @@ export class PreBundledFunction extends Function {
     let assetPath = path.join(__dirname, `../${props.codePath}`);
 
     functionProps.code = Code.fromAsset(assetPath);
-    functionProps.functionName = props.name.slice();
+    functionProps.functionName = `${scope.node.id}${id}`;
 
     let lambdaPolicyStatement: PolicyStatement[] = [];
 
@@ -111,7 +108,7 @@ export class PreBundledFunction extends Function {
       assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
       description: 'Role used by lambda in ARA',
       managedPolicies: [lambdaExecutionRolePolicy],
-      roleName: 'LambdaExecutionRole' + functionProps.functionName,
+      //roleName: 'LambdaExecutionRole' + functionProps.functionName,
     });
 
     let logRetentionLambdaPolicyStatement: PolicyStatement[] = [];
@@ -140,26 +137,16 @@ export class PreBundledFunction extends Function {
       assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
       description: 'Role used by lambda to modify log retention',
       managedPolicies: [logRetentionLambdaExecutionRolePolicy],
-      roleName: 'LogRetLambdaExec' + functionProps.functionName,
+      //roleName: 'LogRetLambdaExec' + functionProps.functionName,
     });
 
     functionProps.role = lambdaExecutionRole;
     functionProps.logRetentionRole = logRetentionLambdaExecutionRole;
 
-
-    // const runtimes = Object.values([Runtime.PYTHON_3_8, Runtime.PYTHON_3_7, Runtime.PYTHON_3_9]);
-    // let layers: ILayerVersion[] = [];
-
-    // If the runtime is Python we use the common Lambda Layer with boto3
-    // if (runtimes.includes(functionProps.runtime as Runtime)) {
-    //   layers.push(PreBundledLayer.getOrCreate(scope, 'common/resources/lambdas/pre-bundled-layer'));
-    // }
-
     //delete props that were added to force user input
     delete functionProps.codePath;
     delete functionProps.name;
     delete functionProps.lambdaPolicyStatements;
-    delete functionProps.lambdaLayers;
 
     super(scope, id, { ...(functionProps as FunctionProps) });
   }

core/src/data-generator/batch-replayer.ts

@@ -145,7 +145,6 @@ export class BatchReplayer extends Construct {
      * Find all paths within the time range from the manifest file
      */
     const findFilePathsFn = new PreBundledFunction(this, 'FindFilePath', {
-      name: 'FindFilePathsFn',
       memorySize: 1024,
       codePath: 'data-generator/resources/lambdas/find-file-paths',
       runtime: Runtime.PYTHON_3_9,
@@ -187,8 +186,7 @@ export class BatchReplayer extends Construct {
      * Rewrite data
      */
     const writeInBatchFn = new PreBundledFunction(this, 'WriteInBatch', {
-      name: 'WriteInBatchFn',
-      memorySize: 1024 * 5,
+      memorySize: 1024 * 3,
       codePath: 'data-generator/resources/lambdas/write-in-batch',
       runtime: Runtime.PYTHON_3_9,
       handler: 'write-in-batch.handler',

core/src/db-schema-manager/flyway-runner.ts

@@ -160,9 +160,8 @@ export class FlywayRunner extends Construct {
       }),
     ];
 
-    const flywayLambda = new PreBundledFunction(this, 'runner', {
+    const flywayLambda = new PreBundledFunction(this, 'FlywayLambda', {
       codePath: path.join(__dirname.split('/').slice(-1)[0], './resources/flyway-lambda/flyway-all.jar'),
-      name: 'flywayLambda',
       lambdaPolicyStatements: flywayLambdaPolicy,
       handler: 'com.geekoosh.flyway.FlywayCustomResourceHandler::handleRequest',
       runtime: lambda.Runtime.JAVA_11,

core/src/emr-eks-platform/emr-eks-nodegroup-asg-tag.ts

@@ -69,11 +69,10 @@ export class EmrEksNodegroupAsgTagProvider extends Construct {
     ];
 
     // AWS Lambda function supporting the create, update, delete operations on Amazon EMR on EKS managed endpoints
-    const onEvent = new PreBundledFunction(this, 'OnEvent', {
+    const onEvent = new PreBundledFunction(this, 'Tag', {
       runtime: Runtime.PYTHON_3_9,
       codePath: 'emr-eks-platform/resources/lambdas/nodegroup-asg-tag',
       handler: 'lambda.on_event',
-      name: 'EmrEksNodegroupAsgTagOnEventFn',
       lambdaPolicyStatements: lambdaPolicy,
       logRetention: RetentionDays.ONE_WEEK,
       layers: [PreBundledLayer.getOrCreate(scope, 'common/resources/lambdas/pre-bundled-layer')],

core/src/emr-eks-platform/emr-managed-endpoint.ts

@@ -113,7 +113,6 @@ export class EmrManagedEndpointProvider extends Construct {
       codePath: 'emr-eks-platform/resources/lambdas/managed-endpoint',
       runtime: Runtime.PYTHON_3_9,
       handler: 'lambda.on_event',
-      name: 'EmrManagedEndpointProviderOnEvent',
       layers: [PreBundledLayer.getOrCreate(scope, 'common/resources/lambdas/pre-bundled-layer')],
       lambdaPolicyStatements: lambdaPolicy,
       logRetention: RetentionDays.ONE_WEEK,
@@ -124,7 +123,6 @@ export class EmrManagedEndpointProvider extends Construct {
     const isComplete = new PreBundledFunction(this, 'IsComplete', {
       codePath: 'emr-eks-platform/resources/lambdas/managed-endpoint',
       handler: 'lambda.is_complete',
-      name: 'EmrManagedEndpointProviderIsComplete',
       layers: [PreBundledLayer.getOrCreate(scope, 'common/resources/lambdas/pre-bundled-layer')],
       lambdaPolicyStatements: lambdaPolicy,
       runtime: Runtime.PYTHON_3_9,

core/src/synchronous-athena-query/synchronous-athena-query.ts

@@ -113,10 +113,9 @@ export class SynchronousAthenaQuery extends Construct {
 
 
     // AWS Lambda function for the AWS CDK Custom Resource responsible to start query
-    const athenaQueryStartFn = new PreBundledFunction(this, 'AthenaQueryStartFn', {
+    const athenaQueryStartFn = new PreBundledFunction(this, 'StartFn', {
       runtime: Runtime.PYTHON_3_9,
       codePath: 'synchronous-athena-query/resources/lambdas',
-      name: 'SynchronousAthenaCrStart',
       layers: [PreBundledLayer.getOrCreate(scope, 'common/resources/lambdas/pre-bundled-layer')],
       lambdaPolicyStatements: athenaQueryStartFnPolicy,
       handler: 'lambda.on_event',
@@ -144,10 +143,9 @@ export class SynchronousAthenaQuery extends Construct {
     }));
 
     // AWS Lambda function for the AWS CDK Custom Resource responsible to wait for query completion
-    const athenaQueryWaitFn = new PreBundledFunction(this, 'AthenaQueryWaitFn', {
+    const athenaQueryWaitFn = new PreBundledFunction(this, 'WaitFn', {
       runtime: Runtime.PYTHON_3_9,
       codePath: 'synchronous-athena-query/resources/lambdas',
-      name: 'SynchronousAthenaCrWait',
       layers: [PreBundledLayer.getOrCreate(scope, 'common/resources/lambdas/pre-bundled-layer')],
       lambdaPolicyStatements: athenaQueryWaitFnPolicy,
       handler: 'lambda.is_complete',

core/src/synchronous-crawler/synchronous-crawler.ts

@@ -64,10 +64,9 @@ export class SynchronousCrawler extends Construct {
     })];
 
     // AWS Lambda function for the AWS CDK Custom Resource responsible to start crawler
-    const crawlerStartFn = new PreBundledFunction(this, 'crawlerStartFn', {
+    const crawlerStartFn = new PreBundledFunction(this, 'StartFn', {
       runtime: Runtime.PYTHON_3_9,
       codePath: 'synchronous-crawler/resources/lambdas',
-      name: 'SynchronousCrawlerStartFn',
       layers: [PreBundledLayer.getOrCreate(scope, 'common/resources/lambdas/pre-bundled-layer')],
       lambdaPolicyStatements: lambdaCRPolicy,
       handler: 'lambda.on_event',
@@ -76,10 +75,9 @@ export class SynchronousCrawler extends Construct {
     });
 
     // AWS Lambda function for the AWS CDK Custom Resource responsible to wait for crawler completion
-    const crawlerWaitFn = new PreBundledFunction(this, 'crawlerWaitFn', {
+    const crawlerWaitFn = new PreBundledFunction(this, 'WaitFn', {
       runtime: Runtime.PYTHON_3_9,
       codePath: 'synchronous-crawler/resources/lambdas',
-      name: 'SynchronousCrawlerWaitFn',
       layers: [PreBundledLayer.getOrCreate(scope, 'common/resources/lambdas/pre-bundled-layer')],
       lambdaPolicyStatements: lambdaCRPolicy,
       handler: 'lambda.is_complete',

core/test/e2e/batch-replayer.test.ts

@@ -10,7 +10,7 @@
 import { Bucket } from 'aws-cdk-lib/aws-s3';
 import * as cdk from 'aws-cdk-lib';
 import { RemovalPolicy } from 'aws-cdk-lib';
-import { deployStack /*, destroyStack*/ } from './utils';
+import { deployStack , destroyStack } from './utils';
 
 import { BatchReplayer } from '../../src/data-generator/batch-replayer';
 import { PreparedDataset } from '../../src/data-generator/prepared-dataset';
@@ -30,6 +30,11 @@ const batchReplayer = new BatchReplayer(stack, 'BatchReplay', {
   sinkBucket: sinkBucket,
 });
 
+new BatchReplayer(stack, 'BatchReplay2', {
+  dataset: PreparedDataset.RETAIL_1_GB_CUSTOMER,
+  sinkBucket: sinkBucket,
+});
+
 new cdk.CfnOutput(stack, 'DatasetName', {
   value: batchReplayer.dataset.tableName,
   exportName: 'DatasetName',

core/test/unit/cdk-nag/nag-batch-replayer.test.ts

@@ -30,37 +30,37 @@ Aspects.of(batchReplayer).add(new AwsSolutionsChecks());
 
 NagSuppressions.addResourceSuppressionsByPath(
   batchReplayerStack,
-  'BatchReplayer/TestBatchReplayer/LambdaExecutionRolePolicyFindFilePathsFn/Resource',
+  'BatchReplayer/TestBatchReplayer/LambdaExecutionRolePolicyTestBatchReplayerFindFilePath/Resource',
   [{ id: 'AwsSolutions-IAM5', reason: 'The IAM policy needs access to all objects in the source dataset' }],
 );
 
 NagSuppressions.addResourceSuppressionsByPath(
   batchReplayerStack,
-  'BatchReplayer/TestBatchReplayer/LogRetentionLambdaExecutionRolePolicyFindFilePathsFn/Resource',
+  'BatchReplayer/TestBatchReplayer/LogRetentionLambdaExecutionRolePolicyTestBatchReplayerFindFilePath/Resource',
   [{ id: 'AwsSolutions-IAM5', reason: 'The Lambda execution role for log retention needs * access' }],
 );
 
 NagSuppressions.addResourceSuppressionsByPath(
   batchReplayerStack,
-  'BatchReplayer/TestBatchReplayer/LogRetentionLambdaExecutionRoleFindFilePathsFn/DefaultPolicy/Resource',
+  'BatchReplayer/TestBatchReplayer/LogRetentionLambdaExecutionRoleTestBatchReplayerFindFilePath/DefaultPolicy/Resource',
   [{ id: 'AwsSolutions-IAM5', reason: 'The Lambda execution role for log retention needs * access' }],
 );
 
 NagSuppressions.addResourceSuppressionsByPath(
   batchReplayerStack,
-  'BatchReplayer/TestBatchReplayer/LambdaExecutionRolePolicyWriteInBatchFn/Resource',
+  'BatchReplayer/TestBatchReplayer/LambdaExecutionRolePolicyTestBatchReplayerWriteInBatch/Resource',
   [{ id: 'AwsSolutions-IAM5', reason: 'The IAM policy needs access to all objects in the sink location' }],
 );
 
 NagSuppressions.addResourceSuppressionsByPath(
   batchReplayerStack,
-  'BatchReplayer/TestBatchReplayer/LambdaExecutionRoleWriteInBatchFn/DefaultPolicy/Resource',
+  'BatchReplayer/TestBatchReplayer/LambdaExecutionRoleTestBatchReplayerWriteInBatch/DefaultPolicy/Resource',
   [{ id: 'AwsSolutions-IAM5', reason: 'The IAM policy needs access to all objects in the sink location' }],
 );
 
 NagSuppressions.addResourceSuppressionsByPath(
   batchReplayerStack,
-  'BatchReplayer/TestBatchReplayer/LogRetentionLambdaExecutionRolePolicyWriteInBatchFn/Resource',
+  'BatchReplayer/TestBatchReplayer/LogRetentionLambdaExecutionRolePolicyTestBatchReplayerWriteInBatch/Resource',
   [{ id: 'AwsSolutions-IAM5', reason: 'The Lambda execution role for log retention needs * access' }],
 );