Default index pattern not compatible with Elasticsearch 7.9 xpack metrics template

[danielbonelli]

Hi Team,

micrometer/implementations/micrometer-registry-elastic/src/main/java/io/micrometer/elastic/ElasticMeterRegistry.java

Line 148 in 9b6d7e1

indexLine = "{ \"index\" : {} }\n";

indexLine is always { "index" : {} } or with pipeline but according to elasticsearch (https://www.elastic.co/guide/en/elasticsearch/reference/7.9/use-a-data-stream.html#data-streams-bulk-indexing-requests) data stream documentation it only supports "create"

micrometer-registry-elastic-1.5.4

Thanks

[shakuzen]

Could we take a step back and describe what you are trying to achieve and the problem you are having before we get into the cause?
The ElasticMeterRegistry does not currently use data streams, which are a very recently added feature as far as I can tell. Is this an enhancement request to support shipping metrics to a data stream instead of as documents?

[lnorregaard]

Hi @shakuzen

I ran into this today that from Elasticsearch 7.9 on Elastic cloud they are using an xpack default feature with a managed index called metrics--. This is a data stream index.

So what I needed to do was to call the management.metrics.export.elastic.index=metrics_something so it is not hitting the managed template.

Maybe an idea is to have this as an option if it is a data stream index or a normal index?

[danielbonelli]

Hi @shakuzen

I tried to use out of the box micrometer-registry-elastic-1.5.4 in a spring boot application (2.1.3.RELEASE)
With this yaml configuration

        export:
            elastic:
                enabled: true
                host: http://localhost:9200
                auto-create-index: true
                step: 10s
        distribution:
            percentiles-histogram:
                http:
                    server:
                        requests: true
            sla:
                http:
                    server:
                        requests: 100ms, 400ms, 500ms, 2000ms
            percentiles:
                http:
                    server:
                        requests: 0.5, 0.75, 0.95, 0.99

And was getting this error: "only write ops with an op_type of create are allowed in data streams"

What is happening is that calling POST http://localhost:9200/metrics-2020-09/_bulk which gets called from ElasticMeterRegistry publish method is creating a Data Stream and it does not support "index" action only "create"

As Inorregaard above mentions changing to metrics_something is a possible workaround.. but with default options it does not work.

Hope this clears the issue a bit

[izeye]

I created #2288 to try to resolve this.

[shakuzen]

This is resolved by the change in #2288 which changes the default index name to something that doesn't conflict.