Can we have synchronous API for log.ReadAsync

[dkantyadev]

Thank you so much for this amazing project!

I’m currently working on building an in-memory database that can handle datasets larger than the available RAM.
For persistence, I’m using Faster Log, which seems very promising. The concurrency control relies on the ReaderWriterLockSlim, which keeps the implementation simple, particularly for synchronous APIs (without using async/await).

The challenge appears when considering how to fetch data from the log once the data record has been flushed to disk (to maintain a small memory footprint). I noticed that the log only provides an asynchronous method for this:

(result, length) = await log.ReadAsync(iter.CurrentAddress);

However, this creates an issue for my implementation, which relies on ReaderWriterLockSlim. It breaks serializable execution because I can’t use async/await within a transaction.

Having a synchronous version of log.ReadAsync would be incredibly helpful. That way, during a transaction, I could verify that a record is missing from RAM but still fetch it from disk using its address.

Maybe I can use log.ReadAsync(iter.CurrentAddress).GetAwaiter().GetResult(), but I am not sure about deadlocks that may happen with such an approach.

I would appreciate any suggestions.

[AntyaDev]

@badrishc Any ideas?

[cschuchardt88]

Use AsyncPool. If you dont use AsyncPool you could have deadlocks. Otherwise just use ClientSession. But keep in mind that if you have more than one ClientSession open, it will deadlock one of them.

Example

AsyncPool<
            ClientSession<
                SpanByte,
                SpanByte,
                SpanByte,
                SpanByte,
                SpanByte,
                SpanByteFunctions<SpanByte, SpanByte, SpanByte>>> sessionPool;

sessionPool = new(
                _logSettings.LogDevice.ThrottleLimit,
                () => _store.For(new SpanByteFunctions<SpanByte, SpanByte, SpanByte>())
                    .NewSession<SpanByteFunctions<SpanByte, SpanByte, SpanByte>>());

How to Use

if (_sessionPool.TryGet(out var session) == false)
    session = _sessionPool.Get();

var (status, output) = session.Read(SpanByte.FromFixedSpan(key));
byte[]? value = null;

if (status.Found)
    value = output.ToByteArray();
if (status.IsPending && session.CompletePendingWithOutputs(out var iter, true, true))
{
    using (iter)
    {
        while (iter.Next())
        {
            if (iter.Current.Key.AsSpan().SequenceEqual(key))
            {
                value = iter.Current.Output.ToByteArray();
                break;
            }
        }
    }
}

_sessionPool.Return(session);

[AntyaDev]

Hi @cschuchardt88 and thank you!
We don't use session
We use just log

I asked for specific log API
log.ReadAsync(iter.CurrentAddress)