Skip to content

clickhouse: WARNING for large mmap_log_buffer_size #2464

@RomanBelozerov

Description

@RomanBelozerov

I received some problems with large value for buffer:

  • mmap_log_buffer_size > 4 Mb - ERROR in dmesg - ERROR: Unable to start module 'access_log': -22;
  • mmap_log_buffer_size > 128 Mb - WARNING with trace in dmesg;
  • not info about the buffer size in the log file;

To Reproduce
I cannot to reproduce it.

Configuration file

listen 80;
server ${server_ip}:8000;

mmap_log_buffer_size 400M;
access_log mmap logger_config=${tfw_logger_logger_config};

Version or commit hash
kernel - 5.10.35.tfw-16a4e99
tempesta - a72e107

Stacktrace or debug log

[ 8016.590023] ------------[ cut here ]------------
[ 8016.590410] WARNING: CPU: 5 PID: 30738 at mm/page_alloc.c:4935 __alloc_pages_nodemask+0x2c7/0x340
[ 8016.590756] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 nvme_tcp nvme_fabrics nvme_core vhost_vsock vmw_vsock_virtio_transport_common vhost vhost_iotlb vsock xt_conntrack xt_MASQUERADE xt_set ip_set xt_addrtype nft_compat xfrm_user xfrm_algo nft_masq nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bridge stp llc overlay binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common kvm_intel kvm snd_hda_codec_generic ledtrig_audio crct10dif_pclmul ghash_clmulni_intel snd_hda_intel aesni_intel snd_intel_dspcfg snd_hda_codec crypto_simd cryptd snd_hda_core glue_helper snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event iTCO_wdt snd_rawmidi iTCO_vendor_support snd_seq qxl drm_ttm_helper ttm snd_seq_device snd_timer i2c_i801 drm_kms_helper snd i2c_smbus cec fb_sys_fops soundcore syscopyarea sysfillrect sysimgblt lpc_ich joydev input_leds serio_raw mac_hid sch_fq_codel nf_tables libcrc32c msr parport_pc
[ 8016.590802]  ppdev lp parport drm efi_pstore nfnetlink dmi_sysfs qemu_fw_cfg ip_tables x_tables autofs4 hid_generic usbhid hid crc32_pclmul virtio_net psmouse net_failover ahci libahci virtio_rng failover virtio_blk [last unloaded: tempesta_lib]
[ 8016.595342] CPU: 5 PID: 30738 Comm: tempesta.sh Kdump: loaded Tainted: G           OE     5.10.35.tfw-16a4e99 #1
[ 8016.595917] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 8016.596495] RIP: 0010:__alloc_pages_nodemask+0x2c7/0x340
[ 8016.597060] Code: 48 8b 04 25 c0 7b 01 00 48 05 58 0c 00 00 41 be 01 00 00 00 48 89 45 b0 e9 14 fe ff ff 41 81 e4 00 20 00 00 0f 85 60 ff ff ff <0f> 0b e9 59 ff ff ff 31 c0 e9 e8 fd ff ff e8 f6 14 fc ff 48 89 c7
[ 8016.598212] RSP: 0018:ffffc128c1783bd8 EFLAGS: 00010246
[ 8016.598769] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 8016.599367] RDX: 0000000000000000 RSI: 000000000000000f RDI: 0000000000000cc0
[ 8016.600029] RBP: ffffc128c1783c30 R08: fffffb7dc5ad6200 R09: ffff9daa33583480
[ 8016.600623] R10: ffff9dab33fd4100 R11: fffffb7dc65b0008 R12: 0000000000000000
[ 8016.601211] R13: 0000000000000000 R14: 0000000000008000 R15: ffff9daa33583480
[ 8016.601777] FS:  00007fdb89fdc740(0000) GS:ffff9dab2bf40000(0000) knlGS:0000000000000000
[ 8016.602362] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 8016.602906] CR2: 000056041a837090 CR3: 000000016b94c004 CR4: 0000000000770ee0
[ 8016.603465] PKRU: 55555554
[ 8016.604100] Call Trace:
[ 8016.604717]  tfw_mmap_buffer_create+0x1a6/0x350 [tempesta_fw]
[ 8016.605272]  tfw_access_log_start+0x39/0x90 [tempesta_fw]
[ 8016.605821]  tfw_ctlfn_state_io+0x31b/0x460 [tempesta_fw]
[ 8016.606349]  ? tfw_cleanup+0x30/0x30 [tempesta_fw]
[ 8016.606897]  ? __cgroup_bpf_run_filter_sysctl+0x16d/0x2e0
[ 8016.607450]  proc_sys_call_handler+0x13d/0x250
[ 8016.608015]  proc_sys_write+0x13/0x20
[ 8016.608622]  new_sync_write+0x114/0x1a0
[ 8016.609173]  vfs_write+0x1bd/0x250
[ 8016.609737]  ksys_write+0x67/0xe0
[ 8016.610254]  __x64_sys_write+0x19/0x20
[ 8016.610758]  do_syscall_64+0x38/0x90
[ 8016.611307]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 8016.611848] RIP: 0033:0x7fdb8a0fb574
[ 8016.612397] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 ea 0e 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
[ 8016.613475] RSP: 002b:00007fff1b824318 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 8016.614005] RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00007fdb8a0fb574
[ 8016.614527] RDX: 0000000000000006 RSI: 000056041a837090 RDI: 0000000000000001
[ 8016.615051] RBP: 00007fff1b824340 R08: 0000000000000073 R09: 0000000000000000
[ 8016.615525] R10: 00000000ffffffff R11: 0000000000000202 R12: 0000000000000006
[ 8016.616065] R13: 000056041a837090 R14: 00007fdb8a1e35c0 R15: 00007fdb8a1e0ee0
[ 8016.616589] ---[ end trace 1ce0e9e86f695ff5 ]---
[ 8016.617076] [tempesta fw] ERROR: Unable to start module 'access_log': -22
[ 8016.617546] [tempesta fw] modules are stopped
[ 8016.618063] [tempesta fw] Warning: Configuration parsing has failed. Clean up...

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions