Skip to content

Conversation

@QianyongY
Copy link
Contributor

@QianyongY QianyongY commented Jul 23, 2025

What changes are proposed in this pull request?

This PR addresses the NoSuchFieldError related to org.apache.log4j.MDC class when integrating Alluxio with Spark 3. The issue occurs due to version conflicts between reload4j (used by Alluxio) and log4j (used by Spark).

Problem: The reload4j dependency introduced in #18435 has compatibility issues with Spark 3, causing NoSuchFieldError when using Spark.

Related discussion: qos-ch/reload4j#41

Why are the changes needed?

The current reload4j causes compatibility issues when Alluxio is integrated with Spark:

  • Alluxio: Uses slf4j-reload4j 1.7.34 → reload4j 1.2.18.3 → MDC.class tlm type: ThreadLocalMap
  • Spark: Uses log4j 1.2.17 → MDC.class tlm type: Object

When using Spark , causing:

java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
	at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$setMDCForTask(Executor.scala:762)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:448)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchFieldError: Class org.apache.log4j.MDC does not have member field 'java.lang.Object tlm' [in thread "Executor task launch worker-0"]
	at org.apache.log4j.MDCFriend.fixForJava9(MDCFriend.java:11)
	at org.slf4j.impl.Log4jMDCAdapter.<clinit>(Log4jMDCAdapter.java:38)
	at org.slf4j.impl.StaticMDCBinder.getMDCA(StaticMDCBinder.java:59)
	at org.slf4j.MDC.bwCompatibleGetMDCAdapterFromBinder(MDC.java:99)
	at org.slf4j.MDC.<clinit>(MDC.java:108)

Therefore, we should exclude reload4j from the Alluxio client

Does this PR introduce any user facing changes?

No

Copy link

@cchung100m cchung100m left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks to @QianyongY 😄

@QianyongY
Copy link
Contributor Author

alluxio-bot, merge this please

@alluxio-bot
Copy link
Contributor

merge failed:
The user is not in the maintainer list, skip it for merge

@QianyongY
Copy link
Contributor Author

LGTM, thanks to @QianyongY 😄

I can't use alluxio-bot. Could you help me merge it? Thanks. @cchung100m

@cchung100m
Copy link

Hi @apc999

Can you help to review this PR and trigger the alluxio-bot?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants