Unable to access metadata in log data received via opentelemetry input plugin

[trondhindenes]

Hi, I have an input that looks like this:

  - name: opentelemetry
    listen: 0.0.0.0
    port: 4321
    tag: logs

if I output to stdout, it looks like this (from a properly instrumented app):

[0] logs: [[1713517641.125841653, {"otelSpanID"=>"223c13433c580fcc", "otelTraceID"=>"ee0399b2524507ed56f459662d30cc2f", "otelTraceSampled"=>true, "otelServiceName"=>"stuff", "code.filepath"=>"/code/src/app/main.py", "code.function"=>"get_the_thing", "code.lineno"=>36}], {"message"=>"getting the thing: http://localhost:8000/thing2"}]

However, if I use the "json" format in the stdout, it looks like this:

{"date":1713517571.122547,"message":"getting the thing: http://localhost:8000/thing3"}

So, I'm guessing all the metadata for the log event gets truncated somehow, or not shown if outputting as json.

My problem is, I need that data to be sent to the actual logging system I want to use (loki). I've also tried accessing the metadata like so, using a filter:

filters:
- name: record_modifier
  match: logs
  record:
  - yam $otelServiceName

However, otelServiceName is not resolved even tho it exists in the metadata.

I'm struggling to get this work, and I cannot find any documentation around accessing event metadata.

Should what I'm trying to do work?

[psyhomb]

Exactly the same issue on my end.

[patrick-stephens]

Metadata access is not supported for everything I'm afraid at the moment, the legacy filters only ever worked with log data previously for example.

There are some plugins, e.g. OTEL output, that explicitly call out metadata access via config options now with xx_metadata_key vs xx_message_key: https://docs.fluentbit.io/manual/pipeline/outputs/opentelemetry

The https://docs.fluentbit.io/manual/pipeline/processors/opentelemetry-envelope may be useful though.