[receiver/filelog] Re: Add option to insert offset as attribute

[francois07]

Component(s)

pkg/stanza, receiver/filelog

Is your feature request related to a problem? Please describe.

Reopen #34053: "When sending logs to an intermediate service such as Kafka, there are times when unicity and order of messages are not 100% guaranteed."

Describe the solution you'd like

It would be useful to be able to insert the offset of each log record as an attribute, to be able to guarantee order and unicity of logs when sending them to an intermediate service before the backend.

A config parameter such as include_file_offset seems appropriate

Describe alternatives you've considered

Currently, there exists the include_file_record_number attribute but it is not exactly similar to a file offset. Log collection backends that want to move from filebeat to OpenTelemetry in particular could have issues if they use filebeat's file offset to detect duplicate, out of order and missing logs.

One solution is to recreate file offset using record number and length, but as of right now I don't think this is possible in the collector since you would also need the previous record's offset to get the current one.
This would obviously be doable on the backend side but including it directly into the record makes the transition from filebeat to OpenTelemetry almost transparent which is optimal

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/filelog: @djaglowski @andrzej-stencel
• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

Seems reasonable to me. To clarify, the offset would mark the beginning of the log? i.e. Using file.Seek with this value would allow someone to read exactly from the first character of the log.

[francois07]

Hello, the offset would mark the beginning of the log. This way, it's possible to know if you're at the very beginning of the file since the offset would be 0

[francois07]

Any news on this idea @djaglowski ? I've been reading through stanza's code for a little while now to understand how this could be implemented and I already have a few ideas. I would love to try to implement this myself and send a pull request if you haven't already started 👍

[andrzej-stencel]

Feel free to submit a proposal PR @francois07 👍 I'll assign this issue to you.