Getting uncompaction(actual) data

[sentoz]

Hello.

When trying to request data from Mimir to Grafana or Promlens, there is no data for the last 2.5-3 hours.

Maybe I missed something in the Mimir configuration.
Mimir deploy in microservice configuration, using s3.

My config:

---
# Source: mimir-distributed/templates/mimir-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: mimir-config
  labels:
    app.kubernetes.io/name: mimir
    app.kubernetes.io/version: "2.6.0"
  namespace: "monitoring"
data:
  mimir.yaml: |
    
    common:
      storage:
        backend: s3
        s3:
          endpoint: URL

    server:
      grpc_server_max_concurrent_streams: 1000
      grpc_server_max_connection_age: 2m
      grpc_server_max_connection_age_grace: 5m
      grpc_server_max_connection_idle: 1m
      http_listen_port: 8080
      log_level: debug
      grpc_server_max_recv_msg_size: 67108864


    limits:
      max_cache_freshness: 10m
      max_query_parallelism: 240
      max_total_query_length: 12000h
      ingestion_rate: 165000
      max_global_series_per_user: 0
      max_global_series_per_metric: 100000
      compactor_blocks_retention_period: 90d
      max_queriers_per_tenant: 1
      ingestion_tenant_shard_size: 2
      compactor_tenant_shard_size: 3
      query_sharding_total_shards: 16
      store_gateway_tenant_shard_size: 3


    activity_tracker:
      filepath: /active-query-tracker/activity.log

    alertmanager:
      data_dir: /data
      enable_api: true
      external_url: /alertmanager
      fallback_config_file: /configs/alertmanager_fallback_config.yaml
    alertmanager_storage:
      s3:
        access_key_id: SECRET
        bucket_name: mimir-alertmanager
        secret_access_key: SECRET

    blocks_storage:
      backend: s3
      storage_prefix: blocks
      bucket_store:
        max_chunk_pool_bytes: 12884901888
        sync_dir: /data/tsdb-sync
        ignore_blocks_within: 0s
        sync_interval: 5m
      s3:
        access_key_id: SECRET
        bucket_name: mimir-blocks
        secret_access_key: SECRET
      tsdb:
        dir: /data/tsdb
        block_ranges_period:
          - 2h0m0s
        retention_period: 24h0m0s


    compactor:
      cleanup_interval: 5m
      compaction_interval: 30m
      compaction_retries: 3
      data_dir: /data
      deletion_delay: 10h
      max_closing_blocks_concurrency: 2
      max_opening_blocks_concurrency: 4
      sharding_ring:
        kvstore:
          store: memberlist
        wait_stability_min_duration: 1m
      symbols_flushers_concurrency: 4
      block_ranges:
        - 2h0m0s
        - 12h0m0s
        - 24h0m0s

    distributor:
      ring:
        kvstore:
          store: memberlist


    frontend:
      parallelize_shardable_queries: true
      querier_forget_delay: 1m
      scheduler_address: mimir-query-scheduler-headless.monitoring.svc:9095
    frontend_worker:
      grpc_client_config:
        max_send_msg_size: 419430400
      scheduler_address: mimir-query-scheduler-headless.monitoring.svc:9095

    ingester:
      ring:
        final_sleep: 0s
        num_tokens: 512
        tokens_file_path: /data/tokens
        unregister_on_shutdown: false
        kvstore:
          store: memberlist
        replication_factor: 3
    ingester_client:
      grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600

    memberlist:
      abort_if_cluster_join_fails: false
      compression_enabled: false
      cluster_label: k8s-infra-cluster
      message_history_buffer_bytes: 1024
      join_members:
      - mimir-gossip-ring.monitoring.svc.cluster.local:7946

    querier:
      max_concurrent: 16
      query_store_after: 15s
      timeout: 1m
      shuffle_sharding_ingesters_enabled: false
      query_ingesters_within: 10s
    query_scheduler:
      max_outstanding_requests_per_tenant: 800

    ruler:
      alertmanager_url: http://_http-metrics._tcp.mimir-alertmanager-headless.monitoring.svc.cluster.local/alertmanager
      enable_api: true
      rule_path: /data
      ring:
        heartbeat_period: 2s
        heartbeat_timeout: 10s

    ruler_storage:
      s3:
        access_key_id: SECRET
        bucket_name: mimir-ruler
        secret_access_key: SECRET

    runtime_config:
      file: /var/mimir/runtime.yaml

    store_gateway:
      sharding_ring:
        tokens_file_path: /data/tokens
        wait_stability_min_duration: 1m
        replication_factor: 1
        kvstore:
          store: memberlist
    usage_stats:
      enabled: false

[pstibrany]

Hi. Your query_ingesters_within: 10s option confuses queriers -- it tells them to ONLY query ingesters for LATEST 10s of data. Given that ingesters build 2h blocks, and store up to 3h of data in memory only, 10s for query_ingesters_within is very low. I'd suggest to use default value of 13h. (Your ingesters keep data for 24h!)

Also query_store_after: 15s is very suspicious... if you set query_ingesters_within: 13h, then there is no need to query store before this time. That's why default value for query_store_after is 12h.

I'd recommend to stick with default values unless you need to alter specific behaviour.

[sentoz]

@pstibrany
I misinterpreted the parameter description.
Made adjustments, now ok.

Thank you!