logs: fix serialization of Extended attributes #4342

xrmx · 2024-12-09T11:45:14Z

Description

Logs attribute accepts AnyValue as AttributeValue add a type to describe that and start using it.

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Test A

Does This PR Require a Contrib Repo Change?

Yes. - Link to PR:
No.

Checklist:

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added
Documentation has been updated

opentelemetry-api/src/opentelemetry/_logs/_internal/__init__.py

opentelemetry-api/src/opentelemetry/util/types.py

opentelemetry-api/src/opentelemetry/_logs/_internal/__init__.py

opentelemetry-api/src/opentelemetry/util/types.py

CHANGELOG.md

xrmx · 2025-04-02T14:11:58Z

Updated PR with fixes for logs attributes serialization.

pmcollins

Thanks for this change -- left a couple of comments.

opentelemetry-api/src/opentelemetry/attributes/__init__.py

pmcollins

LGTM

xrmx · 2025-04-03T14:16:02Z

Ok tested this with an aiohttp server application with Pablo patch applied to not break logging, without that I got no logging out of the app.

Before:

2025-04-03 15:53:05,727 - INFO - 127.0.0.1 [03/Apr/2025:14:53:05 +0100] "GET / HTTP/1.1" 200 166 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:136.4) Gecko/20100101 Firefox/136.4"
2025-04-03 15:53:05,727 - WARNING - Invalid type dict for attribute 'request_header' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types
{
    "body": "Invalid type dict for attribute 'request_header' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types",
    "severity_number": 13,
    "severity_text": "WARN",
    "attributes": {
        "otelSpanID": "0",
        "otelTraceID": "0",
        "otelTraceSampled": false,
        "otelServiceName": "unknown_service",
        "code.filepath": "/home/rm/src/otel-python/venvaiohttp/lib/python3.10/site-packages/opentelemetry/attributes/__init__.py",
        "code.function": "_clean_attribute",
        "code.lineno": 100
    },
    "dropped_attributes": 0,
    "timestamp": "2025-04-03T13:53:05.727613Z",
    "observed_timestamp": "2025-04-03T13:53:05.727649Z",
    "trace_id": "0x00000000000000000000000000000000",
    "span_id": "0x0000000000000000",
    "trace_flags": 0,
    "resource": {
        "attributes": {
        },
        "schema_url": ""
    }
}
{
    "body": "127.0.0.1 [03/Apr/2025:14:53:05 +0100] \"GET / HTTP/1.1\" 200 166 \"-\" \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:136.4) Gecko/20100101 Firefox/136.4\"",
    "severity_number": 9,
    "severity_text": "INFO",
    "attributes": {
        "otelSpanID": "0",
        "otelTraceID": "0",
        "otelTraceSampled": false,
        "otelServiceName": "unknown_service",
        "remote_address": "127.0.0.1",
        "request_start_time": "[03/Apr/2025:14:53:05 +0100]",
        "first_request_line": "GET / HTTP/1.1",
        "response_status": 200,
        "response_size": 166,
        "code.filepath": "/home/rm/src/otel-python/venvaiohttp/lib/python3.10/site-packages/aiohttp/web_log.py",
        "code.function": "log",
        "code.lineno": 214
    },
    "dropped_attributes": 0,
    "timestamp": "2025-04-03T13:53:05.727498Z",
    "observed_timestamp": "2025-04-03T13:53:05.727549Z",
    "trace_id": "0x00000000000000000000000000000000",
    "span_id": "0x0000000000000000",
    "trace_flags": 0,
    "resource": {
        "attributes": {
        },
        "schema_url": ""
    }
}

After:

2025-04-03 16:07:59,652 - INFO - 127.0.0.1 [03/Apr/2025:15:07:59 +0100] "GET / HTTP/1.1" 200 166 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:136.4) Gecko/20100101 Firefox/136.4"
{
    "body": "127.0.0.1 [03/Apr/2025:15:07:59 +0100] \"GET / HTTP/1.1\" 200 166 \"-\" \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:136.4) Gecko/20100101 Firefox/136.4\"",
    "severity_number": 9,
    "severity_text": "INFO",
    "attributes": {
        "otelSpanID": "0",
        "otelTraceID": "0",
        "otelTraceSampled": false,
        "otelServiceName": "unknown_service",
        "remote_address": "127.0.0.1",
        "request_start_time": "[03/Apr/2025:15:07:59 +0100]",
        "first_request_line": "GET / HTTP/1.1",
        "response_status": 200,
        "response_size": 166,
        "request_header": {
            "Referer": "-",
            "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:136.4) Gecko/20100101 Firefox/136.4"
        },
        "code.filepath": "/home/rm/src/otel-python/venvextendedattrs/lib/python3.10/site-packages/aiohttp/web_log.py",
        "code.function": "log",
        "code.lineno": 214
    },
    "dropped_attributes": 0,
    "timestamp": "2025-04-03T14:07:59.652817Z",
    "observed_timestamp": "2025-04-03T14:07:59.652993Z",
    "trace_id": "0x00000000000000000000000000000000",
    "span_id": "0x0000000000000000",
    "trace_flags": 0,
    "resource": {
        "attributes": {
        },
        "schema_url": ""
    }
}

And as printed by collector debug exporter:

Attributes:
     -> otelSpanID: Str(0)
     -> otelTraceID: Str(0)
     -> otelTraceSampled: Bool(false)
     -> otelServiceName: Str(unknown_service)
     -> remote_address: Str(127.0.0.1)
     -> request_start_time: Str([03/Apr/2025:15:07:59 +0100])
     -> first_request_line: Str(GET / HTTP/1.1)
     -> response_status: Int(200)
     -> response_size: Int(166)
     -> request_header: Map({"Referer":"-","User-Agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:136.4) Gecko/20100101 Firefox/136.0"})
     -> code.filepath: Str(/home/rm/src/otel-python/venvextendedattrs/lib/python3.10/site-packages/aiohttp/web_log.py)
     -> code.function: Str(log)
     -> code.lineno: Int(214)

lmolkova

🚀

emdneto

Some typing issues. Just add "opentelemetry-sdk/src/opentelemetry/sdk/_logs", in the strict list in pyright and:

tox -e typecheck | grep ExtendedAttributes -A 5

  /workspaces/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/_logs/_internal/__init__.py:189:15 - error: Argument of type "AnyValue | TraceFlags | SeverityNumber | BoundedAttributes" cannot be assigned to parameter "attributes" of type "ExtendedAttributes | None" in function "__init__"
    Type "AnyValue | TraceFlags | SeverityNumber | BoundedAttributes" is not assignable to type "ExtendedAttributes | None"
      Type "SeverityNumber" is not assignable to type "ExtendedAttributes | None"
        "SeverityNumber" is not assignable to "Mapping[str, AnyValue]"
        "SeverityNumber" is not assignable to "None" (reportArgumentType)

  /workspaces/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/_logs/_internal/__init__.py:200:32 - error: Argument of type "ExtendedAttributes | None" cannot be assigned to parameter "attributes" of type "Attributes" in function "__init__"
    Type "ExtendedAttributes | None" is not assignable to type "Attributes"
      Type "ExtendedAttributes" is not assignable to type "Attributes"
        "Mapping[str, AnyValue]" is not assignable to "Mapping[str, AttributeValue]"
          Type parameter "_VT_co@Mapping" is covariant, but "AnyValue" is not a subtype of "AttributeValue"
            Type "AnyValue" is not assignable to type "AttributeValue"
        "Mapping[str, AnyValue]" is not assignable to "None" (reportArgumentType)

  /workspaces/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/_logs/_internal/__init__.py:255:36 - error: Cannot access attribute "dropped" for class "ExtendedAttributes"
    Attribute "dropped" is unknown (reportAttributeAccessIssue)
  
  /workspaces/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/_logs/_internal/__init__.py:646:17 - error: Argument of type "ExtendedAttributes | None" cannot be assigned to parameter "attributes" of type "Attributes" in function "__init__"
    Type "ExtendedAttributes | None" is not assignable to type "Attributes"
      Type "ExtendedAttributes" is not assignable to type "Attributes"
        "Mapping[str, AnyValue]" is not assignable to "Mapping[str, AttributeValue]"
          Type parameter "_VT_co@Mapping" is covariant, but "AnyValue" is not a subtype of "AttributeValue"
            Type "AnyValue" is not assignable to type "AttributeValue"
        "Mapping[str, AnyValue]" is not assignable to "None" (reportArgumentType)

opentelemetry-api/src/opentelemetry/_events/__init__.py

opentelemetry-api/src/opentelemetry/_logs/_internal/__init__.py

opentelemetry-api/src/opentelemetry/attributes/__init__.py

opentelemetry-sdk/src/opentelemetry/sdk/util/instrumentation.py

opentelemetry-api/src/opentelemetry/util/types.py

Logs attribute accepts AnyValue as AttributeValue add a type to describe that and start using it.

* logs: introduce LogAttributes type Logs attribute accepts AnyValue as AttributeValue add a type to describe that and start using it. * LogAttributes -> ExtendedAttributes * Handle ExtendedAttributes in BoundedAttributes * opentelemetry-sdk: serialize extended attributes * Add changelog * Fix typing * Fix handling of not attribute values inside sequences * Please mypy * Please lint * More typing * Even more typing fixes * Fix docs * Fix mypy * Update LogRecord attributes typing to match reality * More typing * Move changelog to unreleased * ExtendedAttributes -> _ExtendedAttributes * opentelemetry-sdk: keep instrumentation scope attributes as Attributes * exporter/otlp: allow export of none values in logs attributes

xrmx requested a review from a team as a code owner December 9, 2024 11:45

xrmx marked this pull request as draft December 9, 2024 11:46

xrmx added the Approve Public API check This label shows that the public symbols added or changed in a PR are strictly necessary label Dec 9, 2024

aabmass approved these changes Dec 10, 2024

View reviewed changes

xrmx marked this pull request as ready for review December 12, 2024 21:28

xrmx mentioned this pull request Dec 13, 2024

Logs API and SDK: logs attributes are allowed to have AnyValue values #4315

Closed

lmolkova reviewed Dec 14, 2024

View reviewed changes

opentelemetry-api/src/opentelemetry/_logs/_internal/__init__.py Outdated Show resolved Hide resolved

Kludex suggested changes Dec 24, 2024

View reviewed changes

opentelemetry-api/src/opentelemetry/util/types.py Outdated Show resolved Hide resolved

opentelemetry-api/src/opentelemetry/_logs/_internal/__init__.py Outdated Show resolved Hide resolved

xrmx force-pushed the logs-attributes-typing branch from 66ec409 to 57a6fb3 Compare December 24, 2024 08:15

Kludex approved these changes Dec 24, 2024

View reviewed changes

xrmx requested a review from lmolkova December 24, 2024 08:43

lmolkova reviewed Dec 29, 2024

View reviewed changes

opentelemetry-api/src/opentelemetry/util/types.py Outdated Show resolved Hide resolved

xrmx commented Mar 4, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

lmolkova mentioned this pull request Mar 27, 2025

Introduce "extended attributes" term open-telemetry/opentelemetry-specification#4462

Closed

5 tasks

xrmx force-pushed the logs-attributes-typing branch from 57a6fb3 to 65603ac Compare March 31, 2025 13:08

xrmx marked this pull request as draft March 31, 2025 13:09

xrmx force-pushed the logs-attributes-typing branch from 0ad43ac to 419334b Compare April 2, 2025 14:00

xrmx changed the title ~~logs: introduce LogAttributes type~~ logs: fix serialization of Extended attributes Apr 2, 2025

pmcollins reviewed Apr 2, 2025

View reviewed changes

opentelemetry-api/src/opentelemetry/attributes/__init__.py Show resolved Hide resolved

opentelemetry-api/src/opentelemetry/attributes/__init__.py Show resolved Hide resolved

pmcollins approved these changes Apr 3, 2025

View reviewed changes

xrmx marked this pull request as ready for review April 3, 2025 14:23

xrmx force-pushed the logs-attributes-typing branch from e215e93 to 9ee67b1 Compare April 3, 2025 15:12

xrmx requested a review from aabmass April 3, 2025 15:56

lmolkova approved these changes Apr 3, 2025

View reviewed changes

emdneto reviewed Apr 4, 2025

View reviewed changes

xrmx commented Apr 7, 2025

View reviewed changes

opentelemetry-api/src/opentelemetry/_events/__init__.py Outdated Show resolved Hide resolved

xrmx commented Apr 7, 2025

View reviewed changes

opentelemetry-api/src/opentelemetry/_logs/_internal/__init__.py Outdated Show resolved Hide resolved

xrmx force-pushed the logs-attributes-typing branch from d29c930 to 07c8b0c Compare April 8, 2025 15:54

xrmx force-pushed the logs-attributes-typing branch from 07c8b0c to 4aa2d48 Compare April 11, 2025 10:43

aabmass approved these changes Apr 14, 2025

View reviewed changes

aabmass reviewed Apr 14, 2025

View reviewed changes

opentelemetry-api/src/opentelemetry/util/types.py Outdated Show resolved Hide resolved

xrmx added 18 commits April 15, 2025 10:42

logs: introduce LogAttributes type

a254b65

Logs attribute accepts AnyValue as AttributeValue add a type to describe that and start using it.

LogAttributes -> ExtendedAttributes

84a03ed

Handle ExtendedAttributes in BoundedAttributes

52b9b8c

opentelemetry-sdk: serialize extended attributes

e2959e4

Add changelog

ab47594

Fix typing

b3d9ac9

Fix handling of not attribute values inside sequences

b0d72c5

Please mypy

ae53317

Please lint

2456614

More typing

007a75f

Even more typing fixes

879f20e

Fix docs

576027a

Fix mypy

cec6ef3

Update LogRecord attributes typing to match reality

f048a49

More typing

d048655

Move changelog to unreleased

f4f9914

ExtendedAttributes -> _ExtendedAttributes

ea6cf00

opentelemetry-sdk: keep instrumentation scope attributes as Attributes

879ab21

xrmx force-pushed the logs-attributes-typing branch from 4aa2d48 to 879ab21 Compare April 15, 2025 08:43

aabmass added Approve Public API check This label shows that the public symbols added or changed in a PR are strictly necessary and removed Approve Public API check This label shows that the public symbols added or changed in a PR are strictly necessary labels Apr 15, 2025

exporter/otlp: allow export of none values in logs attributes

1e61922

emdneto approved these changes Apr 17, 2025

View reviewed changes

xrmx merged commit 742171e into open-telemetry:main Apr 18, 2025
386 checks passed

github-project-automation bot moved this from In Progress to Done in Python Logging API/SDK RC Apr 18, 2025

lukaslihotzki-f mentioned this pull request Apr 22, 2025

multiple opentelemetry fixups famedly/synapse#27

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

logs: fix serialization of Extended attributes #4342

logs: fix serialization of Extended attributes #4342

Uh oh!

xrmx commented Dec 9, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xrmx commented Apr 2, 2025

Uh oh!

pmcollins left a comment

Uh oh!

Uh oh!

Uh oh!

pmcollins left a comment

Uh oh!

xrmx commented Apr 3, 2025 •

edited

Loading

Uh oh!

lmolkova left a comment

Uh oh!

emdneto left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

logs: fix serialization of Extended attributes #4342

logs: fix serialization of Extended attributes #4342

Uh oh!

Conversation

xrmx commented Dec 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How Has This Been Tested?

Does This PR Require a Contrib Repo Change?

Checklist:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xrmx commented Apr 2, 2025

Uh oh!

pmcollins left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pmcollins left a comment

Choose a reason for hiding this comment

Uh oh!

xrmx commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lmolkova left a comment

Choose a reason for hiding this comment

Uh oh!

emdneto left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xrmx commented Dec 9, 2024 •

edited

Loading

xrmx commented Apr 3, 2025 •

edited

Loading

emdneto left a comment •

edited

Loading