fluentd buffer not pushing to opensearch after restart. Experiencing data loss because of fluentd restart

[Sameer0998]

Problem : We have two instances of opensearch one primary and one secondary with different host url's. During confugration phase lets say the secondary is not reachable the push is temporarily stopped to both the primary os and secondary os. Now during this time, the buffer data file size increases upto certain point and then the file size stops increasing even though the chunk_limit_size is much more higher.

what's expected : No data loss when restart of fluentd.

As you can see in the picture the log size seems to be somewhere around 2mb and stops increasing further , but based on the buffer settings of fluentd as shown below you can see the chunk_limit_size : 64M and we have queued chunks limit size : 100 , but still I dont see any logs getting queued either.

        <buffer>
          @type file
          path /var/log/fluentd-edge-moss-pm-buffers/small.moss.system.buffer.pm
          flush_mode interval
          flush_thread_count 4
          flush_interval 60s
          retry_type periodic
          retry_max_times 20
          retry_wait 20s
          chunk_limit_size 64M
          queued_chunks_limit_size 100
          overflow_action throw_exception
        </buffer>

Now lets say somehow we have got the secondary up and running , now when the fluentd restarts it is expected to push the data which is in buffer to the opensearch but somehow the log file is removed and data is not pushed to opensearch.

so, if the secondary is down for 1 hr duration of time...the logs had to be queued and data should try to push once the connection is up and running. but we are experiencing loss of data ? why is this happening or Am I missing something here?

Also nowadays, whenever fluentd pod restarts , we are experiencing certain data points missing or loss of data.
Also Is there anyway to improve the performance of fluentd to experience minimum data loss. ?

Also, let's say :
If the time has reached to 60 seconds and one of the instance is not reachable then the chunk is flushed hence data is loss right ?

[Sameer0998]

Can someone please take a look at this issue and tell me why this happens, the buffer should be stored and then on restart of fluentd it is supposed to push the entire logs to opensearch, but we are experiencing some data points loss during restart of fluentd. Can someone explain please ?

[Watson1978]

Please provide a reproducible config.
If you provide the part of configuration, we may not be possible to investigate.

[Sameer0998]

Please check the config below :

  output.conf: |-
    <match **>
      @type copy
      <store ignore_error>
        @type relabel
        @label @primary
      </store>
  {{- if .Values.opensearch.mossSecondary.enabled }}
      <store ignore_error>
        @type relabel
        @label @secondary
      </store>
  {{- end }}
    </match>
    
    <label @primary>
      <match **>
        @id elasticsearch_moss_pm
        @type elasticsearch
        @log_level trace
        scheme https
        ssl_verify false
        ssl_version TLSv1_2
        suppress_type_name true
        host xxxxxxxxxxx
        port 443
        user xxxxxxxxx
        password xxxxxxxxx
        templates xxxxxxxxxxxxx
        template_overwrite true
        write_operation upsert
        target_index_key @target_index
        index_name defaulmosstindex
        type_name _doc
        id_key request_id
        remove_keys request_id
        reconnect_on_error true
        reload_on_failure true
        reload_connections false
        request_timeout 45s
        bulk_message_request_threshold -1
        <buffer>
          @type file
          path /var/log/fluentd-edge-moss-pm-buffers/moss.system.buffer.pm
          flush_mode interval
          flush_thread_count 4
          flush_interval 30s
          retry_type periodic
          retry_max_times 20
          retry_wait 20s
          chunk_limit_size 64M
          queued_chunks_limit_size 100
          overflow_action throw_exception
        </buffer>
      </match>
    </label>
  {{- if .Values.opensearch.mossSecondary.enabled }}
    <label @secondary>
      <filter **>
        @type record_modifier
        <record>
          @target_index ${record["@target_index"]}_${(((Time.at(time).strftime("%j").to_i - 1) / 3) * 3 + 1)}
        </record>
      </filter>
      <match **>
        @id secondary_elasticsearch_moss_pm
        @type elasticsearch
        @log_level trace
        scheme https
        ssl_verify false
        ssl_version TLSv1_2
        suppress_type_name true
        host xxxxxxxxxxxx
        port 443
        user xxxxxxxxx
        password xxxxxxxxxxx
        templates xxxxxxxxxxxx
        template_overwrite true
        write_operation upsert
        target_index_key @target_index
        index_name duplicatedefaulmosstindex
        type_name _doc
        id_key request_id
        remove_keys request_id
        reconnect_on_error true
        reload_on_failure true
        reload_connections false
        request_timeout 45s
        bulk_message_request_threshold -1
        <buffer>
          @type file
          path /var/log/fluentd-edge-moss-pm-buffers/small.moss.system.buffer.pm
          flush_mode interval
          flush_thread_count 4
          flush_interval 30s
          retry_type periodic
          retry_max_times 20
          retry_wait 20s
          chunk_limit_size 64M
          queued_chunks_limit_size 100
          overflow_action throw_exception
        </buffer>
      </match>
    </label>
  {{- end }}
{{- end }}

problem :

1. As mentioned above the size of the file increases only upto 2 mb or so and then stops increasing further and the files are not queuing up either even though the set limit is way higher in buffer config. :

chunk_limit_size 64M
queued_chunks_limit_size 100

2. Analysis required : (please explain) : In configuration phase of the plugin lets say the secondary store is not able to connect to opensearch and then the data stored in buffer upto certain time with problem 1 , then when the secondary host is up and running ..the buffered data is just not pushed. whereas the retry is 20 times but still experiencing loss of data

[Watson1978]

The original issue was closed at uken/fluent-plugin-elasticsearch#1060
So, I will close this.