Incorrect or corrupted CSV log lines when multiple cluster nodes write to the same log file with prudent mode enabled

[Dushyant-GitHub]

We have a cluster-based application running on multiple nodes. Each node uses Logback to write logs in CSV-based format to the same shared log file.
Under high load (when multiple requests are processed across nodes at the same time), we observed that some log lines become corrupted — e.g., lines get mixed up or partially written, resulting in broken or half-written CSV entries.

It seems that Logback is not correctly handling concurrent writes across multiple nodes when using this configuration.

Configuration Details

We are programmatically configuring Logback as follows:

context = new LoggerContext();
context.stop();
context.setMDCAdapter(new LogbackMDCAdapter());

encoder = new PatternLayoutEncoder();
encoder.setContext(context);
encoder.setPattern("%msg%n");
encoder.start();

RollingFileAppender<ILoggingEvent> rollingFileAppender = new RollingFileAppender<>();
rollingFileAppender.setContext(context);
rollingFileAppender.setName(appName);
rollingFileAppender.setAppend(true);
rollingFileAppender.setImmediateFlush(true);
rollingFileAppender.setBufferSize(new FileSize(bufferSize));  // 8192 bytes
rollingFileAppender.setPrudent(true);

rollingPolicy = new CustomTimeBasedRollingPolicy<>();
rollingPolicy.setContext(context);
rollingPolicy.setParent(rollingFileAppender);
rollingPolicy.setFileNamePattern(logFilePath);
rollingPolicy.setMaxHistory(retentionPeriod);

String previousLogfilePath = resolvePreviousLogFilePath(logFilePath, interval); // interval = Monthly
rollingPolicy.setTimeBasedFileNamingAndTriggeringPolicy(
    new CustomTimeBasedFileNamingAndTriggeringPolicy<>(previousLogfilePath)
);
rollingPolicy.start();

rollingFileAppender.setRollingPolicy(rollingPolicy);
rollingFileAppender.setEncoder(encoder);
rollingFileAppender.start();

AsyncAppender asyncAppender = new AsyncAppender();
asyncAppender.setName("async_" + appName);
asyncAppender.setContext(context);
asyncAppender.setQueueSize(5000);
asyncAppender.addAppender(rollingFileAppender);
asyncAppender.start();

logger = context.getLogger("ROOT");
logger.addAppender(asyncAppender);
logger.setLevel(Level.INFO);
context.start();

Observed Behavior

• Some log lines are partially written (only part of the CSV line appears).

• Some lines are merged or mixed between log entries written from different nodes.

• This happens only when multiple cluster nodes write to the same file around the same time.

Expected Behavior

Each log line should be written atomically and remain intact (no mixing or truncation), even when multiple nodes are writing to the same file with prudent=true.

Environment

• Logback version: 1.5.17

• Java version: Java 21

• Application server: Clustered setup (multiple JVM nodes writing to shared log file)

• File system: NFS

Can you please confirm:

1. Whether prudent=true fully supports concurrent writes across multiple JVMs/nodes?

2. If not, is there a recommended approach for cluster-safe logging?

Example Screenshot of Corrupted CSV Output

[VidhPhadke]

@ceki We used the program here to determine if the locking used by FileAppender is working in our kubernetes deployment on Azure. It looks like we can re-produce the corruption with that. Since that is the code used by logback logback also has the same problem. Has this code been tested on various files mounting mechanism like NFS , SMB from multiple hosts ?
Could you comment on whether atomic writes of individual log lines is supported ?

[ceki]

@Dushyant-GitHub @VidhPhadke Regarding your various questions, please refer to the relevant documentation of prudent mode. Note the applicable restrictions.

Your environment makes use of CustomTimeBasedFileNamingAndTriggeringPolicy which, as the name suggests, specific to your environment. If I had to wager, I would guess that the issue stems from the custom policy.

I also think that having a large queue size of 5000 for AsyncAppender is not good practice. I would recommend the default size of 256 and not much more.

For in depth answers with fast response times, consider championing a logback release.

[VidhPhadke]

Thanks @ceki for your response. However, i don't think our problems are related to the queue size or the CustomTimeBasedFileNamingAndTriggeringPolicy .
To get to the root of the issue we used the test application here https://gist.github.com/stefanrufer/bc66547bf6d35ccf2992 which does not use logback. The problem exists at the OS / JVM /SMB level. looks like the Network sharing mechanism in use in our Azure kubernetes cluster is SMB which seems to have a problem. We can reproduce the corruption with just that.

So, IMHO it would be useful to make a enhancement to Networked file locks section. It should be stated that the prudent mode may not work correctly at all. It would be best to strongly recommends testing with the https://gist.github.com/2794241 not just for performance, but also for correctness. So, errors like Permission errors when reading from the file which is locked or corruption i the file seems to suggest SMB is being used.