We use the official fluent-bit helm chart and deploy it to Kubernetes. So, the script itself can be found here. And the settings for passing it to fluent-bit are here. The settings are optimized for our load and now it specifies a large write-to-file cache, if you don't need that, remove those settings.

So, what does this script do? It takes the log field and checks if there is a JSON string there. If there isn't, it doesn't do anything. But if there is a JSON string, it starts parsing it and serializes all nested objects into a string with a dot. Thus the data type of all keys and values become strings, except for those strings where the content contains timestamp, and any strings with timestamp OpenSearch/ElasticSearch (OS/ES) assigns the date type. This behavior can be disabled at the index template level.

It is also possible to specify forced conversion of fields with dates to text via the ENABLE_DATE_REPLACEMENT variable. It replaces the colon with an underscore and adds the _date prefix. Thus, there is no need to specify disabling date type conversion in the index template of OpenSearch itself. However, you should note that you will not be able to search these fields by date and time, but this only applies to fields within the log. object.

Also note that in our setup, fluent-bit will replace any dots with underscores: Replace_Dots On. Also we do not enable the built-in parser: Merge_Log Off, but it is disabled by default and you can remove the setting. The built-in parser perfectly parses the log field and its subobjects, however, sometimes the field types of object values can be different and this is what causes OS/ES to refuse to accept logs.