[CRASH] KeyDB Server crashes while running with configured Flash storage. #571
Description
Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
=== KEYDB BUG REPORT START: Cut & paste starting from here ===
39:80:M 16 Feb 2023 22:21:45.648 # ------------------------------------------------
39:80:M 16 Feb 2023 22:21:45.648 # !!! Software Failure. Press left mouse button to continue
39:80:M 16 Feb 2023 22:21:45.648 # Guru Meditation: Wrong obj->encoding in addReply() #networking.cpp:459
------ STACK TRACE ------
Backtrace:
keydb-server 0.0.0.0:17379 [cluster](addReply(client*, robj_roptr)+0x1b7) [0x561a2de78037]
keydb-server 0.0.0.0:17379 [cluster](addReplyBulk(client*, robj_roptr)+0x34) [0x561a2de784e4]
keydb-server 0.0.0.0:17379 [cluster](redisDbPersistentData::prefetchKeysAsync(client*, parsed_command&, bool)+0x112d) [0x561a2de59dcd]
keydb-server 0.0.0.0:17379 [cluster](parseClientCommandBuffer(client*)+0x3a3) [0x561a2de76d73]
keydb-server 0.0.0.0:17379 [cluster](readQueryFromClient(connection*)+0x39a) [0x561a2de7757a]
keydb-server 0.0.0.0:17379 [cluster](tlsHandleEvent(tls_connection*, int)+0x22d) [0x561a2dd9ce5d]
keydb-server 0.0.0.0:17379 [cluster](ProcessEventCore+0xf9) [0x561a2de9aa09]
keydb-server 0.0.0.0:17379 [cluster](aeProcessEvents+0x17d) [0x561a2de9bddd]
keydb-server 0.0.0.0:17379 [cluster](aeMain+0x3a) [0x561a2de9f7da]
keydb-server 0.0.0.0:17379 [cluster](workerThreadMain(void*)+0x88) [0x561a2de87d98]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f77e47b8609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f77e46db133]
------ INFO OUTPUT ------
# Server
redis_version:255.255.255
redis_git_sha1:687850a6
redis_git_dirty:1
redis_build_id:b535e138518f077a
redis_mode:cluster
os:Linux 5.10.147+ x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:9.4.0
process_id:39
process_supervised:no
run_id:b10fb7d1277abe9b2503ff0a1e579fe8f0c15a63
tcp_port:17379
server_time_usec:1676586105649270
uptime_in_seconds:72165
uptime_in_days:0
hz:40
configured_hz:40
lru_clock:15641721
executable:/data/keydb-server
config_file:/redis-conf/redis.conf
# Clients
connected_clients:360
cluster_connections:14
maxclients:60000
client_recent_max_input_buffer:1597
client_recent_max_output_buffer:1720
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:1
thread_0_clients:59
thread_1_clients:58
thread_2_clients:63
thread_3_clients:55
thread_4_clients:63
thread_5_clients:63
# Memory
used_memory:19164365112
used_memory_human:17.85G
used_memory_rss:22100135936
used_memory_rss_human:20.58G
used_memory_peak:19876436104
used_memory_peak_human:18.51G
used_memory_peak_perc:96.42%
used_memory_overhead:3066340932
used_memory_startup:25837864
used_memory_dataset:16098024180
used_memory_dataset_perc:84.11%
allocator_allocated:19167535392
allocator_active:21145632768
allocator_resident:21448155136
total_system_memory:33671958528
total_system_memory_human:31.36G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:17875228262
maxmemory_human:16.65G
maxmemory_policy:allkeys-lfu
allocator_frag_ratio:1.10
allocator_frag_bytes:1978097376
allocator_rss_ratio:1.01
allocator_rss_bytes:302522368
rss_overhead_ratio:1.03
rss_overhead_bytes:651980800
mem_fragmentation_ratio:1.15
mem_fragmentation_bytes:2935881208
mem_not_counted_for_evict:2000000000
mem_replication_backlog:2147483648
mem_clients_slaves:22232
mem_clients_normal:7392828
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:flash
flash_memory:18028387680
# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:99600586
rdb_bgsave_in_progress:0
rdb_last_save_time:1676513940
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:82360
total_commands_processed:132620892
instantaneous_ops_per_sec:12173
total_net_input_bytes:153163043855
total_net_output_bytes:319055282267
instantaneous_input_kbps:12346.78
instantaneous_output_kbps:16820.98
rejected_connections:0
sync_full:15
sync_partial_ok:0
sync_partial_err:4
expired_keys:1565384
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:28628027
keyspace_misses:4064063
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:132469469
total_writes_processed:282972390
instantaneous_lock_contention:1
avg_lock_contention:3.984375
storage_provider_read_hits:48869
storage_provider_read_misses:2528653
# Replication
role:master
connected_slaves:1
slave0:ip=10.110.44.235,port=17379,state=online,offset=151310739562,lag=1
master_failover_state:no-failover
master_replid:c1d0b72f2aad95d3dba2dacec31e27532f35ac9b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:151322374452
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:2000000000
repl_backlog_first_byte_offset:149322374453
repl_backlog_histlen:2000000000
# CPU
used_cpu_sys:9189.125099
used_cpu_user:18190.392180
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
server_threads:6
long_lock_waits:96374
used_cpu_sys_main_thread:2065.450301
used_cpu_user_main_thread:3610.725977
# Modules
module:name=statsd,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
# Commandstats
cmdstat_replping:calls=15,usec=12,usec_per_call=0.80,rejected_calls=0,failed_calls=0
cmdstat_info:calls=6696,usec=436376,usec_per_call=65.17,rejected_calls=0,failed_calls=0
cmdstat_command:calls=28,usec=14359,usec_per_call=512.82,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=6561,usec=3574,usec_per_call=0.54,rejected_calls=0,failed_calls=0
cmdstat_replconf:calls=140987,usec=179727,usec_per_call=1.27,rejected_calls=0,failed_calls=0
cmdstat_cluster:calls=173489,usec=7456756,usec_per_call=42.98,rejected_calls=0,failed_calls=0
cmdstat_client:calls=5,usec=10,usec_per_call=2.00,rejected_calls=0,failed_calls=0
cmdstat_psync:calls=15,usec=386,usec_per_call=25.73,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_setex:calls=99600586,usec=772713403,usec_per_call=7.76,rejected_calls=0,failed_calls=0
cmdstat_readonly:calls=408,usec=220,usec_per_call=0.54,rejected_calls=0,failed_calls=0
cmdstat_get:calls=32692090,usec=137604247,usec_per_call=4.21,rejected_calls=0,failed_calls=0
cmdstat_config:calls=11,usec=786,usec_per_call=71.45,rejected_calls=0,failed_calls=0
# Errorstats
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=11216128,expires=11216128,avg_ttl=32099182,cached_keys=9487185
# KeyDB
mvcc_depth:0
------ CLIENT LIST OUTPUT ------
...
------ MODULES INFO OUTPUT ------
------ FAST MEMORY TEST ------
39:80:M 16 Feb 2023 22:21:45.651 # main thread terminated
39:80:M 16 Feb 2023 22:21:45.651 # Bio thread for job type #0 terminated
39:80:M 16 Feb 2023 22:21:45.651 # Bio thread for job type #1 terminated
39:80:M 16 Feb 2023 22:21:45.651 # Bio thread for job type #2 terminated
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
=== KEYDB BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
https://github.com/JohnSully/KeyDB/issues
Suspect RAM error? Use keydb-server --test-memory to verify it.
Aditional information
- OS distribution and version
root@gke-caching-infra--a-nodepool-ad-plog-1cd6c9b1-glns:/data# uname -a
Linux gke-caching-infra--a-nodepool-ad-plog-1cd6c9b1-glns 5.10.147+ #1 SMP Sat Dec 10 09:54:06 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
- Steps to reproduce (if any)