Uninformative `Memory(Wait(RecvError()))` thrown from Foyer cache under high load

[matthewbayer]

I occasionally receive a cryptic error: Memory(Wait(RecvError())) from my foyer cache when under high read load (with plenty of memory available), coming from HybridCache::obtain.

Is there any way to diagnose this, or throw a more informative error if we are indeed in an error state?

Here's my cache setup:

let working_dir = env::var(WORKING_DIR_ENV_VAR)?;
let dir = tempfile::tempdir_in(working_dir)?;

let device = FsDeviceBuilder::new(dir.path())
    .with_capacity(100 * 1024 * 1024 * 1024)
    .build()?;

let file_cache: HybridCache<String, FileData> = HybridCacheBuilder::new()
    .memory(128 * 1024 * 1024)
    .with_weighter(|_key: &String, value: &FileData| value.data.len())
    .storage()
    .with_engine_config(BlockEngineBuilder::new(device))
    .build()
    .await?;

This is on version 0.19.1, but confirmed to occur in 0.17.x and 0.18.x as well.

[MrCroxx]

Hi, Matthew. Thank you for reporting. Was there any other error/warning level logs? Have you run your application with env RUST_BACKTRACE=1 enabled?

[matthewbayer]

Caused by:
0: foyer memory error: wait for concurrent fetch result error: channel closed
1: wait for concurrent fetch result error: channel closed

I'm running a foyer cache within a local s3 proxy to store file data, so standard caveats around multithreaded access will apply here.

Notably, this appears to only happen on cache misses - particularly for access patterns with a high ratio of cache misses, and a small (or maybe even empty) cache.

[MrCroxx]

Caused by:
0: foyer memory error: wait for concurrent fetch result error: channel closed
1: wait for concurrent fetch result error: channel closed

I'm running a foyer cache within a local s3 proxy to store file data, so standard caveats around multithreaded access will apply here.

Notably, this appears to only happen on cache misses - particularly for access patterns with a high ratio of cache misses, and a small (or maybe even empty) cache.

According to the message, it may be that an error caused the channel in the wait queue to be dropped prematurely. However, this message is too concise and there is not much information. If possible, add debug="full" to the build release binary profile, which may yield more detailed logs.