Buffered logs for a lower latency #63650
Description
Summary
Introduce buffered audit log support in TiDB audit log plugin, with configurable flush interval and buffer size.
Motivation
Currently, audit log entries are written immediately to the audit log plugin. While this ensures low latency, it can create performance overhead and excessive I/O under high-throughput workloads. Many users prefer a more efficient logging mechanism that allows batching without sacrificing reliability.
Proposal
Add two new configuration options under the instance section in the TiDB config file:
PluginAuditLogFlushInterval: Interval (time duration) for flushing buffered audit logs.PluginAuditLogMaxBufferSize: Maximum number of audit log entries to buffer before a flush is triggered.
How it works
- Audit log entries are temporarily buffered in memory.
- A flush is triggered when either the interval expires or the buffer reaches its maximum size.
- Buffered logs are then written to the audit log plugin in batch.
Benefits
- Reduced I/O pressure on the audit log sink.
- Improved performance under heavy workloads.
- More flexibility for users to balance latency vs throughput.
Use Cases
- Large-scale deployments where audit logging is enabled but immediate flushing introduces overhead.
- Workloads that generate frequent audit events and benefit from batching.
Alternatives Considered
- Continue immediate log flushing (status quo).
- Make buffering mandatory without user control (less flexible).
Additional Notes
- Default values will preserve backward compatibility by mimicking immediate flushing (e.g., small buffer size or very short interval).
- Documentation updates required for the new configuration options.