Feature request: ClickHouse query settings

[typedarray]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

The ClickHouse sink currently exposes three query settings as explicit sink options:

• input_format_skip_unknown_fields
• date_time_input_format
• insert_distributed_one_random_shard

This allowlist seems somewhat arbitrary. I'd like to configure other settings, like async_insert.

Attempted Solutions

No response

Proposal

Add a query_settings option in the sink configuration of type object containing a key/value map of query settings. These would just need to be appended to the URL as query params, just like the 3 that are currently supported. As far as I can tell, all clickhouse setting values are uint, float, or string, so they should work just fine in the various Vector config file formats.

sinks:
  my_clickhouse_sink:
    type: clickhouse
    inputs:
      - my_source
    endpoint: http://localhost:8123
    database: mydatabase
    table: mytable
    query_settings:
      input_format_skip_unknown_fields: 1
      date_time_input_format: best_effort
      async_insert: 1
      async_insert_busy_timeout_ms: 3_000

References

No response

Version

No response