Error running against elasticsearch: error creating index template jaeger-span

[TsengSR]

Component(s)

No response

What happened?

Description

I spent the whole day trying to get the operator running for jaeger to consume. After spending many hours just trying to get a working configuration, figuring out how to bypass TLS validation and passing credentials (docs lack here a bit, to put it mildly) I got the below manifest which authenticated against elasticsearch

Right now I am completely at loss. Any idea whats being wrong, how to fix it or at least a workaround? The whole reason I spent 3 days installing ELK is for jaeger to use it as a source and jaeger v2 suggests installing via OpenTelementry operator (well also for search capabilities and metrics, as it gave me an excuse to uninstall prometheus/grafana).

Steps to Reproduce

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: jaeger-telemetry
spec:
  image: jaegertracing/jaeger:latest
  ports:
  - name: jaeger
    port: 16686
  env:
  - name: ES_USERNAME
    valueFrom:
      secretKeyRef:
        name: jaeger-es-credentials
        key: username
  - name: ES_PASSWORD
    valueFrom:
      secretKeyRef:
        name: jaeger-es-credentials
        key: password
  config:
    service:
      extensions: [jaeger_storage, jaeger_query]
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [jaeger_storage_exporter]
    extensions:
      jaeger_query:
        storage:
          traces: elastic_storage
      jaeger_storage:
        backends:
          elastic_storage:
            elasticsearch:
              server_urls:
                - https://elk-default-es-internal-http.monitoring.svc:9200
              auth:
                basic:
                  username: "${ES_USERNAME}"
                  password: "${ES_PASSWORD}"
              # Fix this later, once ELK certs are replaced with ag-eck-ca or ag-elk-ca
              tls:
                insecure_skip_verify: true
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    exporters:
      jaeger_storage_exporter:
        trace_storage: elastic_storage

Expected Result

Create index properly and start gathering telemetry

Actual Result

failed to initialize storage 'elastic_storage': failed to create template "jaeger-span": error creating index template jaeger-span: [400 Bad Request]

Kubernetes Version

1.33.2

Operator version

ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.129.1

Collector version

jaegertracing/jaeger:2.8.0

Environment information

Environment

OS: Talso 1.10.5
Other:

• Elastic Cluster on k8s (ECK): 3.0.0
• ELK 9.0.3

Log output

error    extensions/extensions.go:58    Failed to start extension    {"resource": {"service.instance.id": "89cef11f-3cb9-4048-ba51-a302e10d0fe2", "service.name": "jaeger", "service.version": "v2.8.0"}, "otelcol.component.id": "jaeger_storage", "otelcol.component.kind": "extension", "error": "failed to initialize storage 'elastic_storage': failed to create template \"jaeger-span\": error creating index template jaeger-span: [400 Bad Request] {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"request body is required\"}],\"type\":\"parse_exception\",\"reason\":\"request body is required\"},\"status\":400}"}

Additional context

No response