Enrichment based on a value look up

[olegmukhin]

Is your feature request related to a problem? Please describe

When forwarding logs to a log aggregator, mandatory fields or labels are often required to process and route records effectively. These fields, such as an environment label (e.g., dev, test, prod), may not always be present in the log record. In cases where determining the environment is complex and cannot be achieved through simple log transformations, Fluent Bit needs a mechanism to enrich logs by looking up predefined values specific to the environment.

For example, a log aggregator may reject records missing an environment label, and deriving this label may require mapping a set of conditions or identifiers (e.g., hostname, namespace, or custom metadata) to a predefined list of environment values. Without this capability, logs may be dropped or misrouted, complicating log processing and analysis.

Describe the solution you'd like

I propose adding a lookup-based enrichment feature to Fluent Bit, allowing users to define a static key-value mapping in a file to add mandatory fields like an environment label to log records. This feature would enable Fluent Bit to:

• Read a predefined lookup table (CSV file to start with) containing mappings, such as hostnames to environment values.
• Match a specified log field (e.g. hostname) against the lookup table.
• Enrich the log record with the corresponding value (e.g. environment: dev) if a match is found.

This solution has been implemented in pull request: PR #10620. The PR introduces a new filter plugin that supports lookup-based enrichment, allowing users to define mappings in a configuration file and apply them to log records dynamically. Documentation for the filter is submitted as part of PR #1953.

Describe alternatives you've considered

• Checklist filter: The proposed solution shares similarities with the existing checklist filter in Fluent Bit, which also loads data from a file to perform matching logic. However, the checklist filter is designed for filtering records based on matches rather than enriching them with additional fields, limiting its applicability for this use case.
• Log Transformation: Using existing Fluent Bit filters like modify or lua to hardcode environment labels. This approach is complicated if the look up involves several hundred or thousands of values and is less flexible that loading in a file.
• External Enrichment: Enriching logs downstream in the log aggregator (e.g. Loki). This shifts the burden to the aggregator, but may not be possible where the source is a platform hosting several environments.

Additional context

In the future the filter can be enhanced to support other formats (e.g. JSON) and support reload of the look up file on a schedule.