Skip to content

Add CloudWatch EMF exporter foundation with basic Gauge support #382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jun 19, 2025

Conversation

mxiamxia
Copy link
Member

@mxiamxia mxiamxia commented Jun 5, 2025

This PR introduces the core structure for the CloudWatch EMF exporter for sending custom metrics. So for some specific customers they can send their own custom OTel metrics to CloudWatch metrics backend without requiring any Collector or Agent installed at this moment.

This PR includes:
Changes:

  • Basic EMF log structure creation
  • Gauge metric conversion and export
  • Unit mapping from OTel to CloudWatch units
  • Metric grouping by attributes and timestamps
  • Supports DELTA temporality for CloudWatch compatibility

Future PRs will add:

  • Support for Sum, Histogram, and ExponentialHistogram metrics
  • Advanced batching and CloudWatch Logs constraints
  • Enhanced error handling and validation

Testing:

  • Comprehensive unit tests for core functionality
  • Mock-based testing to avoid AWS dependencies
  • Tests for initialization, conversion, and basic export flow
cpu_usage_gauge = meter.create_gauge(
        name="system_cpu_usage_percent",
        description="Current CPU usage percentage",
        unit="percent"
    )
cpu_usage_gauge.set(0.2, {"host": "server-01", "region": "us-east-1"})
image
{
    "_aws": {
        "Timestamp": 1749774096539,
        "CloudWatchMetrics": [
            {
                "Namespace": "MyApplication1",
                "Dimensions": [
                    [
                        "host",
                        "region"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "system_cpu_usage_percent",
                        "Unit": "percent"
                    }
                ]
            }
        ]
    },
    "Version": "1",
    "otel.resource.telemetry.sdk.language": "python",
    "otel.resource.telemetry.sdk.name": "opentelemetry",
    "otel.resource.telemetry.sdk.version": "1.27.0",
    "otel.resource.service.name": "my-service",
    "otel.resource.service.version": "0.1.0",
    "otel.resource.deployment.environment": "production",
    "system_cpu_usage_percent": 0.2,
    "host": "server-01",
    "region": "us-east-1"
}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@mxiamxia mxiamxia requested a review from a team as a code owner June 5, 2025 04:12
@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch 2 times, most recently from bc2ce38 to 8491507 Compare June 5, 2025 04:19
@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch 12 times, most recently from 22ab59b to 5b6d407 Compare June 10, 2025 23:40
This PR introduces the core structure for the CloudWatch EMF (Embedded Metric Format)
exporter for OpenTelemetry Python. This foundation includes:

Core Features:
- CloudWatchEMFExporter class with full initialization
- Basic EMF log structure creation
- Gauge metric conversion and export
- Unit mapping from OTel to CloudWatch units
- CloudWatch Logs integration with log group/stream management
- Metric grouping by attributes and timestamps
- Basic error handling and logging

Architecture:
- Follows OTel MetricExporter interface
- Uses boto3 for CloudWatch Logs integration
- Implements proper resource attribute handling
- Supports DELTA temporality for CloudWatch compatibility

Testing:
- Comprehensive unit tests for core functionality
- Mock-based testing to avoid AWS dependencies
- Tests for initialization, conversion, and basic export flow

Future PRs will add:
- Support for Sum, Histogram, and ExponentialHistogram metrics
- Advanced batching and CloudWatch Logs constraints
- Enhanced error handling and validation
@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch from 5b6d407 to 1b34c6f Compare June 11, 2025 00:00
Copy link
Contributor

@thpierce thpierce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did not review _create_emf_log, _send_log_event, export, or unit tests. Will try to take another crack at this this week.

@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch 3 times, most recently from f44f3be to 34a406c Compare June 13, 2025 00:44
@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch from 34a406c to 7de65e4 Compare June 13, 2025 00:50
@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch from 195e95a to a0da19f Compare June 14, 2025 00:45
Copy link
Contributor

@srprash srprash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did not look at the unit tests. Will do so in the next pass.

@srprash
Copy link
Contributor

srprash commented Jun 19, 2025

LGTM. Thanks for addressing the comments. :)

@mxiamxia mxiamxia merged commit f7cd7b6 into aws-observability:main Jun 19, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants