Fluentd to Elasticsearch error: Invalid media-type value on headers [Content-Type, Accept]

[Srijitha-code]

I am encountering an error when Fluentd tries to ship logs to an Elasticsearch cluster. The error is:
2025-04-26 14:04:47 +0000 [warn]: #0 fluent/log.rb:383:warn: failed to flush the buffer. retry_times=3 next_retry_time=2025-04-26 14:04:55 +0000 chunk="633aeee3e06bda3cf586d204d14f472c" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"quickstart-v2-es-http\", :port=>9200, :scheme=>\"https\", :user=>\"elastic\", :password=>\"obfuscated\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Content-Type, Accept]\"}],\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Content-Type, Accept]\",\"caused_by\":{\"type\":\"status_exception\",\"reason\":\"A compatible version is required on both Content-Type and Accept headers if either one has requested a compatible version. Accept=null, Content-Type=application/vnd.elasticsearch+x-ndjson; compatible-with=9\"}},\"status\":400}"

Environment Details:

Fluentd Version: 1.18.0

Fluentd Gems:

fluent-plugin-elasticsearch version 5.4.3

elastic-transport version 8.4.0

elasticsearch version 9.0.2

elasticsearch-api version 9.0.2

Fluentd Deployment: Kubernetes Deployment

Elasticsearch Deployment: ECK (Elastic Cloud on Kubernetes) Operator 3.0

Elasticsearch Version: 9.0

OS Details: Debian GNU/Linux 12 (Bookworm) [ARM64]

Fluentd Config for Elasticsearch Output Plugin

<match local.test>
  @type elasticsearch
  host ******
  port 9200
  scheme https
  user elastic
  password *****
  verify_es_version_at_startup true
  default_elasticsearch_version 9
  ca_file /fluentd/etc/ca.crt
  index_name test-logs
  content_type application/x-ndjson
  logstash_format false
  include_timestamp true
  time_key @timestamp
  <buffer>
    @type file
    path /fluentd/buffer/local_test
    flush_interval 5s
  </buffer>
</match>

Problem Details:

Fluentd logs were not getting pushed to Elasticsearch.

The error suggested a mismatch between the Content-Type and Accept headers.

Accept was null, and Content-Type was application/vnd.elasticsearch+x-ndjson; compatible-with=9.

Workaround Found:
After adding a custom_headers field in the Fluentd config, the issue was resolved.
Here’s what I added in fluent.conf
custom_headers {"Accept":"application/vnd.elasticsearch+json; compatible-with=9"}
After applying this workaround, logs started successfully shipping to Elasticsearch.

Request for Guidance:

Is adding custom_headers like this the correct or recommended approach for Elasticsearch 9.0 compatibility?

Should there be a built-in fix or an official update in the fluent-plugin-elasticsearch or elasticsearch-ruby libraries to handle this automatically?

Are there any better ways to fix this without using a manual custom header override?

Any suggestions would be greatly appreciated! Thanks in advance.

[lev-michael]

@Srijitha-code Hello, I have same issue. I tried your workaround with custom_headers {"Accept":"application/vnd.elasticsearch+json; compatible-with=9"}, but it didn't work. Can you please send me your fluent.conf? I wasn't sure where to place your workround, so just for sure :)

[400] {\"error\":{\"root_cause\":[{\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Content-Type, Accept]\"}],\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Content-Type, Accept]\",\"caused_by\":{\"type\":\"status_exception\",\"reason\":\"Content-Type version must be either version 8 or 7, but found 9. Content-Type=application/vnd.elasticsearch+x-ndjson; compatible-with=9\"}},\"status\":400}"

[DivyaYash]

Am also facing the incompatibility issue. Basically a new version of fluentd-elasticsearch plugin needs to be released which supports ES v9. When can we except this to be released?

[cosmo0920]

Do y'all have cycles to take a look on this, @kenhys @daipom?

[daipom]

Sorry, it is hard to make time right away.
I may not be able to make time in May.

[cosmo0920]

content_type application/x-ndjson

This should be:

content-type 'application/vnd.elasticsearch+json; compatible-with=9'

Elasticsearch 9 requests to add compatible-with=9 for content-type.

[cosmo0920]

I tried to handle the new Content-Type validation for ES9 here:
#1064

Could you confirm this type of work is effective for ES9, @daipom or @kenhys?

[DivyaYash]

@daipom @kenhys Please can you help here, my team has initiated a log mgmt project and have setup latest version of ES/Kibana and hence am in need of Fluentd to be able to write logs to ES

[JakeConnorsCTI]

I am experiencing a related error, that matches what @lev-michael reported.

2025-05-09 03:51:18 +0000 [warn]: #0 failed to flush the buffer. retry_times=2 next_retry_time=2025-05-09 03:51:22 +0000 chunk="634abe0bdc5bf4c28dc274916c9ef3bd" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch
", :port=>9200, :scheme=>"https", :user=>"fluentd", :password=>"obfuscated"}): [400] {"error":{"root_cause":[{"type":"media_type_header_exception","reason":"Invalid media-type value on headers [Content-Type, Accept]"}],"type":"media_type_header_exception","reason":"Invalid media-type valu
e on headers [Content-Type, Accept]","caused_by":{"type":"status_exception","reason":"A compatible version is required on both Content-Type and Accept headers if either one has requested a compatible version. Accept=null, Content-Type=application/vnd.elasticsearch+x-ndjson; compatible-with=9"}},"status":400}"
2025-05-09 03:51:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.4.4/lib/fluent/plugin/out_elasticsearch.rb:1171:in rescue in send_bulk' 2025-05-09 03:51:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.4.4/lib/fluent/plugin/out_elasticsearch.rb:1133:in send_bulk'
2025-05-09 03:51:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.4.4/lib/fluent/plugin/out_elasticsearch.rb:909:in block in write' 2025-05-09 03:51:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.4.4/lib/fluent/plugin/out_elasticsearch.rb:908:in each'
2025-05-09 03:51:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.4.4/lib/fluent/plugin/out_elasticsearch.rb:908:in write' 2025-05-09 03:51:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.8/lib/fluent/plugin/output.rb:1225:in try_flush'
2025-05-09 03:51:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.8/lib/fluent/plugin/output.rb:1538:in flush_thread_run' 2025-05-09 03:51:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.8/lib/fluent/plugin/output.rb:510:in block (2 levels) in start'
2025-05-09 03:51:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.8/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Environment Details:

Elasticsearch Version: 9.0.1
Fluentd Version: 1.16.8 (current edge-debian docker image)

Fluentd Gems:
fluent-plugin-elasticsearch version 5.4.4 (I updated to the very latest hoping this was fixed)
elastic-transport version 8.4.0
elasticsearch version 9.0.2
elasticsearch-api version 9.0.2

FluentD, ElasticSearch and Kibana are running in a Docker Compose Stack in Docker version 28.0.1.

Config for fluent-plugin-elasticsearch

<match sunrise**>
   @type copy
   <store>
        @type "elasticsearch"
        host "elasticsearch"
        port 9200
        user "fluentd"
        password xxxxxx
        scheme https
        ca_file "/fluentd/etc/certs/http_ca.crt"
        logstash_format true
        logstash_prefix "log-test"
        logstash_dateformat "%Y%m%d"
        include_tag_key true
        type_name "access_log"
        tag_key "@log_name"
        flush_interval 1s
        <buffer>
          flush_interval 1s
        </buffer>
   </store>
</match>

With the environment and configuration described above, the fluentd-elasticsearch plugin fails to flush the buffer to elasticsearch and reports to error above.

I was able to find a workaround however, by downgrading the elasticsearch and elasticsearch-api gems back to 9.0.0:

fluent-gem install elasticsearch --no-document --version 9.0.0
fluent-gem uninstall elasticsearch --version 9.0.2
fluent-gem uninstall elasticsearch-api --version 9.0.2

After doing this, the fluentd-elasticsearch plugin flushes the buffers to elasticsearch as expected with no errors. So this appears to maybe be a bug introduced in the elasticsearch or elasticsearch-api 9.0.2 gems.

@lev-michael @DivyaYash Try this workaround and see if it works for you.

[daipom]

@cosmo0920 fixes this issue in #1064, and it is already merged.
fluent-plugin-elasticsearch v6.0.0 will be released soon.

[ilbarone87]

Do I need to add anything to my fluentd config? After bumping the gem to v6 I still see "error 400"

[daipom]

Do I need to add anything to my fluentd config? After bumping the gem to v6 I still see "error 400"

You should not set content_type since it will be automatically set for ES9.

If you setverify_es_version_at_startup false, you must set default_elasticsearch_version 9.

[ilbarone87]

I haven't set any of those values but still getting that error:

My config is:

<match **>
          @type elasticsearch
          @id out_es
          @log_level info
          include_tag_key true
          host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
          port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
          scheme http
          user "#{ENV['FLUENT_ELASTICSEARCH_USER'] || use_default}"
          password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD'] || use_default}"

          # Reload and reconnect options
          reconnect_on_error true
          reload_on_failure true
          reload_connections false

          # HTTP request timeout
          request_timeout 15s
           
          # Log ES HTTP API errors
          log_es_400_reason true

          # avoid 7.x errors
          suppress_type_name true

          # setting sniffer class
          sniffer_class_name Fluent::Plugin::ElasticsearchSimpleSniffer
    
          # Do not use logstash format
          logstash_format false

          # Setting index_name
          index_name fluentd-${index_app_name}

          # specifying time key
          time_key time

          # including @timestamp field
          include_timestamp true

          # ILM Settings - WITH ROLLOVER support
          # https://github.com/uken/fluent-plugin-elasticsearch/blob/master/README.Troubleshooting.md#enable-index-lifecycle-management
          # application_name ${index_app_name}
          index_date_pattern ""
          enable_ilm true
          ilm_policy_id fluentd-policy
          ilm_policy {"policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"10gb","max_age":"7d"}}},"warm":{"min_age":"2d","actions":{"shrink":{"number_of_shards":1},"forcemerge":{"max_num_segments":1}}},"delete":{"min_age":"7d","actions":{"delete":{"delete_searchable_snapshot":true}}}}}}
          ilm_policy_overwrite true
          
          # index template
          use_legacy_template false
          template_overwrite true
          template_name fluentd-${index_app_name}
          template_file "/etc/fluent/template/fluentd-es-template.json"
          customize_template {"<<shard>>": "1","<<replica>>": "0", "<<TAG>>":"${index_app_name}"}
          
          remove_keys idex_app_name

          <buffer tag, index_app_name>
            flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
            flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"
            chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}"
            queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"
            retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
            retry_forever true
          </buffer>
        </match>
      </label>

The logs:

2025-05-09 12:48:50 +0000 [info]: starting fluentd-1.18.0 pid=7 ruby="3.2.8"
2025-05-09 12:48:50 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "--config", "/fluentd/etc/../../../etc/fluent/fluent.conf", "--plugin", "/fluentd/plugins", "-r", "/usr/local/lib/ruby/gems/3.2.0/gems/fluent-plugin-elasticsearch-6.0.0/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]
2025-05-09 12:48:51 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2025-05-09 12:48:51 +0000 [info]: adding match in @FLUENT_LOG pattern="**" type="null"
2025-05-09 12:48:51 +0000 [info]: adding match in @FORWARD pattern="kube.var.log.containers.fluentd**" type="relabel"
2025-05-09 12:48:51 +0000 [info]: adding filter in @FORWARD pattern="kube.**" type="record_modifier"
2025-05-09 12:48:51 +0000 [info]: adding match in @FORWARD pattern="**" type="relabel"
2025-05-09 12:48:51 +0000 [info]: adding filter in @DISPATCH pattern="**" type="prometheus"
2025-05-09 12:48:51 +0000 [info]: adding match in @DISPATCH pattern="**" type="copy"
2025-05-09 12:48:51 +0000 [info]: adding filter in @OUTPUT_ES pattern="kube.**" type="record_transformer"
2025-05-09 12:48:51 +0000 [info]: adding filter in @OUTPUT_ES pattern="host.**" type="record_transformer"
2025-05-09 12:48:51 +0000 [info]: adding match in @OUTPUT_ES pattern="**" type="elasticsearch"
2025-05-09 12:48:51 +0000 [info]: adding filter in @OUTPUT_LOKI pattern="kube.**" type="record_modifier"
2025-05-09 12:48:51 +0000 [info]: adding match in @OUTPUT_LOKI pattern="**" type="loki"
2025-05-09 12:48:51 +0000 [info]: adding source type="forward"
2025-05-09 12:48:51 +0000 [warn]: #0 For security reason, setting private_key_passphrase is recommended when cert_path is specified
2025-05-09 12:48:51 +0000 [info]: adding source type="prometheus"
2025-05-09 12:48:51 +0000 [info]: adding source type="prometheus_monitor"
2025-05-09 12:48:51 +0000 [info]: adding source type="prometheus_output_monitor"
2025-05-09 12:48:51 +0000 [info]: #0 starting fluentd worker pid=19 ppid=7 worker=0
2025-05-09 12:48:51 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2025-05-09 12:48:51 +0000 [warn]: #0 For security reason, setting private_key_passphrase is recommended when cert_path is specified
2025-05-09 12:48:51 +0000 [info]: #0 fluentd worker is now running worker=0
2025-05-09 12:48:57 +0000 [info]: #0 [out_es] Template 'fluentd-argocd' overwritten with /etc/fluent/template/fluentd-es-template.json.
2025-05-09 12:48:57 +0000 [info]: #0 [out_es] Installing ILM policy: {"policy"=>{"phases"=>{"hot"=>{"min_age"=>"0ms", "actions"=>{"rollover"=>{"max_size"=>"10gb", "max_age"=>"7d"}}}, "warm"=>{"min_age"=>"2d", "actions"=>{"shrink"=>{"number_of_shards"=>1}, "forcemerge"=>{"max_num_segments"=>1}}}, "delete"=>{"min_age"=>"7d", "actions"=>{"delete"=>{"delete_searchable_snapshot"=>true}}}}}}
2025-05-09 12:48:57 +0000 [warn]: #0 [out_es] failed to flush the buffer. retry_times=0 next_retry_time=2025-05-09 12:48:58 +0000 chunk="634b36377cfca609e36467159a304b95" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"efk-es-http.logging\", :port=>9200, :scheme=>\"http\", :user=>\"es-fluentd\", :password=>\"obfuscated\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Accept, Content-Type]\"}],\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Accept, Content-Type]\",\"caused_by\":{\"type\":\"status_exception\",\"reason\":\"A compatible version is required on both Content-Type and Accept headers if either one has requested a compatible version. Accept=null, Content-Type=application/vnd.elasticsearch+x-ndjson; compatible-with=9\"}},\"status\":400}"
  2025-05-09 12:54:02 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluent-plugin-elasticsearch-6.0.0/lib/fluent/plugin/out_elasticsearch.rb:1186:in `rescue in send_bulk'
  2025-05-09 12:54:02 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluent-plugin-elasticsearch-6.0.0/lib/fluent/plugin/out_elasticsearch.rb:1148:in `send_bulk'
  2025-05-09 12:54:02 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluent-plugin-elasticsearch-6.0.0/lib/fluent/plugin/out_elasticsearch.rb:924:in `block in write'
  2025-05-09 12:54:02 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluent-plugin-elasticsearch-6.0.0/lib/fluent/plugin/out_elasticsearch.rb:923:in `each'
  2025-05-09 12:54:02 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluent-plugin-elasticsearch-6.0.0/lib/fluent/plugin/out_elasticsearch.rb:923:in `write'
  2025-05-09 12:54:02 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:1225:in `try_flush'
  2025-05-09 12:54:02 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:1540:in `flush_thread_run'
  2025-05-09 12:54:02 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
  2025-05-09 12:54:02 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluentd-1.18.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2025-05-09 12:54:03 +0000 [info]: #0 [out_es] Template 'fluentd-longhorn-system' overwritten with /etc/fluent/template/fluentd-es-template.json.
The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.

[m-m-moradi]

I encountered the exact same issue while using Elasticsearch 8.x.

This started happening after rebuilding a custom Fluentd image. It was especially confusing because I had pinned every version I could — even going back to Fluentd image versions from two years ago — and the issue still persisted.

Eventually, I discovered that the .gemspec file does not pin the version of the elasticsearch Ruby gem. In my case, the image had installed elasticsearch gem version 9.0.3, which was incompatible with my setup.

To fix the issue, I updated my Dockerfile to explicitly install a compatible version of the elasticsearch gem:

RUN fluent-gem install elasticsearch -v 8.18.0

After this change, the issue was resolved.

Here are the working gem versions in my environment:

elastic-transport (8.4.0)
elasticsearch (8.18.0)
elasticsearch-api (8.18.0)
fluent-plugin-elasticsearch (5.4.3)

Hope this helps others facing the same problem!