Improve Artifact Size and Decouple Lambda Code Updates by Leveraging OCI Layer for Lambda Binary

[markussiebert]

Problem:

Currently, we face challenges with storage limits across multiple package repositories (e.g., Maven, npm, PyPI, Go modules) due to inefficient packaging practices. Specifically, we include the same large Go binary within each artifact across different ecosystems. This results in significant redundancy and bloated packages, which quickly consume repository storage limits.

Additionally, the current approach tightly couples the Lambda binary with the individual artifacts. This makes it difficult to update the Lambda code without re-releasing all dependent packages, even for changes that strictly relate to the binary.

Proposal:

We can create the Lambda binary as an OCI Layer and publish it to a centralized, shared container registry (e.g., AWS Public ECR). Instead of embedding the binary directly in each artifact, these artifacts (Maven, npm, PyPI, Go, etc.) would simply reference the OCI Layer.

Benefits:

• Reduced Artifact Size: By referencing the shared OCI Layer, package sizes across all ecosystems would decrease significantly (e.g., by approximately 7 MB per package).
• Decoupled Lambda Code and Package Updates: Separating the Lambda binary as an independent OCI Layer allows us to update it without re-releasing each individual package. This significantly simplifies the maintenance and deployment process.
• Elimination of Redundancy: Avoids repeated embedding of the same binary across different packages.
  Optimized Storage Utilization: Helps avoid storage limit issues in repositories by minimizing redundant data.