ClosedFileSystemException or NullPointerException thrown when SharedArchivePathTree is opened and closed concurrently #48221

ppalaga · 2025-06-04T10:19:26Z

Fixes: ClosedFileSystemException or NullPointerException thrown when SharedArchivePathTree is opened and closed concurrently #48220

ppalaga · 2025-06-04T10:19:39Z

cc @holly-cummins @aloubyansky

aloubyansky

Thanks a lot @ppalaga

SharedArchivePathTree is opened and closed concurrently, fix quarkusio#48220

ppalaga · 2025-06-05T07:23:17Z

@geoand did you do anything else than rebase? I wanted to have a closer look at the failed tests today.

geoand · 2025-06-05T07:29:06Z

Only rebase onto main

aloubyansky · 2025-06-05T11:03:45Z

I'm gonna re-run it again.

ppalaga · 2025-06-05T11:44:13Z

...ndent-projects/bootstrap/app-model/src/main/java/io/quarkus/paths/SharedArchivePathTree.java

        }
        try {
-            this.lastOpen = new SharedOpenArchivePathTree(openFs());
+            lastOpen = this.lastOpen = new SharedOpenArchivePathTree(openFs());


I am still not 100% sure what the intention behind the split into SharedArchivePathTree, SharedOpenArchivePathTree and CallerOpenPathTree is. A sentence of JavaDoc on each of those classes would help.

My hypothesis is that the author wanted to allow multiple instances of SharedOpenArchivePathTree per SharedArchivePathTree where:

The one stored in lastOpen is the preferred and longer living one

All others are closed immediately when they are disposed via close().

Calling openFs() multiple times on a single file (per new SharedOpenArchivePathTree(openFs())) does not make the underlying ZipFilesystemProvider throw a FileSystemAlreadyExistsException, because perhaps the SharedArchivePathTree.CACHE somehow prevents it? Or what else is expected to happen in such situation?

When I saw SharedArchivePathTree.lastOpen writes and reads being protected by OpenArchivePathTree.lock in SharedOpenArchivePathTree, I thought it was a mistake not to protect also writing to lastOpen in SharedArchivePathTree.open():

try { lastOpen = this.lastOpen = new SharedOpenArchivePathTree(openFs()); } catch (IOException e) { throw new UncheckedIOException(e); }

But then, I figured out that the reads/writes from SharedOpenArchivePathTree are protected by per SharedOpenArchivePathTree lock anyway, so multiple instances of SharedOpenArchivePathTree can still race on accessing it. So perhaps it was not really intended not to protect SharedArchivePathTree.lastOpen by those locks at all but rather, those concurrent reads/writes to SharedArchivePathTree.lastOpen are by design and not considered harmful. Could please anybody confirm that? Git blame shows @aloubyansky as an author but hard to say if he wrote it originally.

SharedArchivePathTree has a bit of a Javadoc explaining the idea behind it.
SharedOpenArchivePathTree is the OpenArchivePathTree impl of it.
CallerOpenPathTree is basically shielding the SharedOpenArchivePathTree from the same caller calling close() multiple times on it and so from decrementing the user counter.

You're right about lastOpen not being well protected though. I'll have another look into it.

SharedArchivePathTree has a bit of a Javadoc explaining the idea behind it.

I was very thankful to see it :)

SharedOpenArchivePathTree is the OpenArchivePathTree impl of it.
CallerOpenPathTree is basically shielding the SharedOpenArchivePathTree from the same caller calling close() multiple times on it and so from decrementing the user counter.

Do we really need both Open* and Caller* flavors (and all abstraction layers above)?
Does OpenPathTree have to inherit from PathTree?
All those inheritance, inner-class and delegation links make it really hard to understand what is going on.

Does OpenPathTree have to inherit from PathTree?

Yeah, OpenPathTree is something that needs to be closed but it otherwise a PathTree.

The whole complication is around caching and concurrent access to the same open archive.

Unfortunately, the test rarely fails for me w/o the fix.

quarkus-bot · 2025-06-06T02:07:30Z

Status for workflow `Quarkus CI`

This is the status report for running Quarkus CI on commit 7c4df1c.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

You can consult the Develocity build scans.

Flaky tests - Develocity

⚙️ JVM Tests - JDK 21

📦 extensions/smallrye-reactive-messaging/deployment

✖ io.quarkus.smallrye.reactivemessaging.hotreload.ConnectorChangeTest.testUpdatingConnector - History

Expecting actual: ["-6","-7","-9","-10","-11","-12","-13","-14"] to start with: ["-6", "-7", "-8", "-9"] - java.lang.AssertionError

java.lang.AssertionError: 

Expecting actual:
  ["-6","-7","-9","-10","-11","-12","-13","-14"]
to start with:
  ["-6", "-7", "-8", "-9"]

	at io.quarkus.smallrye.reactivemessaging.hotreload.ConnectorChangeTest.testUpdatingConnector(ConnectorChangeTest.java:41)

⚙️ JVM Tests - JDK 17 Windows

📦 extensions/micrometer-opentelemetry/deployment

✖ io.quarkus.micrometer.opentelemetry.deployment.compatibility.MicrometerTimedInterceptorTest.testTimeMethod_AsyncFailed - History

Stream has no elements - java.lang.IllegalArgumentException

java.lang.IllegalArgumentException: Stream has no elements
	at io.quarkus.micrometer.opentelemetry.deployment.common.MetricDataFilter.lambda$lastReading$2(MetricDataFilter.java:213)
	at java.base/java.util.Optional.orElseThrow(Optional.java:403)
	at io.quarkus.micrometer.opentelemetry.deployment.common.MetricDataFilter.lastReading(MetricDataFilter.java:213)
	at io.quarkus.micrometer.opentelemetry.deployment.common.MetricDataFilter.lastReadingDataPoint(MetricDataFilter.java:231)
	at io.quarkus.micrometer.opentelemetry.deployment.compatibility.MicrometerTimedInterceptorTest.testTimeMethod_AsyncFailed(MicrometerTimedInterceptorTest.java:150)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.test.QuarkusUnitTest.runExtensionMethod(QuarkusUnitTest.java:521)

⚙️ Gradle Tests - JDK 17 Windows

📦 integration-tests/gradle

✖ io.quarkus.gradle.devmode.IncludedKotlinBuildDevModeTest.main - History

Condition with Lambda expression in io.quarkus.test.devmode.util.DevModeClient was not fulfilled within 1 minutes 30 seconds. - org.awaitility.core.ConditionTimeoutException

org.awaitility.core.ConditionTimeoutException: Condition with Lambda expression in io.quarkus.test.devmode.util.DevModeClient was not fulfilled within 1 minutes  30 seconds.
	at app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1160)
	at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1129)
	at app//io.quarkus.test.devmode.util.DevModeClient.getHttpResponse(DevModeClient.java:164)
	at app//io.quarkus.gradle.devmode.QuarkusDevGradleTestBase.getHttpResponse(QuarkusDevGradleTestBase.java:165)

Condition with Lambda expression in io.quarkus.test.devmode.util.DevModeClient was not fulfilled within 1 minutes 30 seconds. - org.awaitility.core.ConditionTimeoutException

org.awaitility.core.ConditionTimeoutException: Condition with Lambda expression in io.quarkus.test.devmode.util.DevModeClient was not fulfilled within 1 minutes  30 seconds.
	at app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1160)
	at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1129)
	at app//io.quarkus.test.devmode.util.DevModeClient.getHttpResponse(DevModeClient.java:164)
	at app//io.quarkus.gradle.devmode.QuarkusDevGradleTestBase.getHttpResponse(QuarkusDevGradleTestBase.java:165)

✖ io.quarkus.gradle.devmode.MavenExclusionInExtensionDependencyDevModeTest.main - History

Condition with Lambda expression in io.quarkus.test.devmode.util.DevModeClient was not fulfilled within 1 minutes 30 seconds. - org.awaitility.core.ConditionTimeoutException

org.awaitility.core.ConditionTimeoutException: Condition with Lambda expression in io.quarkus.test.devmode.util.DevModeClient was not fulfilled within 1 minutes  30 seconds.
	at app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1160)
	at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1129)
	at app//io.quarkus.test.devmode.util.DevModeClient.getHttpResponse(DevModeClient.java:164)
	at app//io.quarkus.gradle.devmode.QuarkusDevGradleTestBase.getHttpResponse(QuarkusDevGradleTestBase.java:165)

⚙️ JVM Integration Tests - JDK 17

📦 integration-tests/opentelemetry

✖ io.quarkus.it.opentelemetry.LoggingResourceTest.testException - History

Condition with Lambda expression in io.quarkus.it.opentelemetry.LoggingResourceTest was not fulfilled within 2 minutes. - org.awaitility.core.ConditionTimeoutException

org.awaitility.core.ConditionTimeoutException: Condition with Lambda expression in io.quarkus.it.opentelemetry.LoggingResourceTest was not fulfilled within 2 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1160)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1129)
	at io.quarkus.it.opentelemetry.LoggingResourceTest.testException(LoggingResourceTest.java:113)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)

📦 integration-tests/opentelemetry-grpc-only

✖ io.quarkus.it.opentelemetry.grpc.HelloGrpcClientTest.testHello - History

java.lang.RuntimeException: Failed to start quarkus - java.lang.RuntimeException

java.lang.RuntimeException: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.test.junit.QuarkusTestExtension.throwBootFailureException(QuarkusTestExtension.java:668)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestClassConstructor(QuarkusTestExtension.java:763)
	at java.base/java.util.Optional.orElseGet(Optional.java:364)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)

⚙️ JVM Integration Tests - JDK 21

📦 integration-tests/opentelemetry

✖ io.quarkus.it.opentelemetry.LoggingResourceTest.testException - History

Condition with Lambda expression in io.quarkus.it.opentelemetry.LoggingResourceTest was not fulfilled within 2 minutes. - org.awaitility.core.ConditionTimeoutException

org.awaitility.core.ConditionTimeoutException: Condition with Lambda expression in io.quarkus.it.opentelemetry.LoggingResourceTest was not fulfilled within 2 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1160)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1129)
	at io.quarkus.it.opentelemetry.LoggingResourceTest.testException(LoggingResourceTest.java:113)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)

gsmet · 2025-06-10T08:27:59Z

@aloubyansky do we merge this one or do you want to have another look before we merge?

aloubyansky · 2025-06-10T08:31:11Z

I'd merge it and also have another look. I have a few things to look into, so I wouldn't delay it since it appears to fix an issue. I didn't expect the CI failures though. They might be unrelated but I don't think we had a clean CI for this PR.

aloubyansky · 2025-06-10T08:32:47Z

I also struggled to make the included test fail w/o the fix. But it's kinda expected for it to be tricky to fail consistently.

quarkus-bot bot added the area/devtools Issues/PR related to maven, gradle, platform and cli tooling/plugins label Jun 4, 2025

ppalaga mentioned this pull request Jun 4, 2025

Failed to access [... .m2/repository/.../httpclient-4.5.14.jar] because the FileSystem has been closed quarkiverse/quarkus-cxf#1803

Closed

aloubyansky approved these changes Jun 4, 2025

View reviewed changes

aloubyansky added triage/backport-3.20 triage/backport labels Jun 4, 2025

This comment has been minimized.

Sign in to view

quarkus-bot bot added the triage/flaky-test label Jun 4, 2025

ClosedFileSystemException or NullPointerException thrown when

7c4df1c

SharedArchivePathTree is opened and closed concurrently, fix quarkusio#48220

geoand force-pushed the cxf1803 branch from 67fc1d8 to 7c4df1c Compare June 5, 2025 07:07

This comment has been minimized.

Sign in to view

ppalaga commented Jun 5, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

gsmet merged commit d4bc475 into quarkusio:main Jun 11, 2025
164 of 166 checks passed

quarkus-bot bot added this to the 3.24 - main milestone Jun 11, 2025

gsmet modified the milestones: 3.24.0.CR1, 3.23.4 Jun 14, 2025

gsmet removed the triage/backport-3.23 label Jun 14, 2025

jmartisk modified the milestones: 3.23.4, 3.20.2 Jun 25, 2025

jmartisk removed the triage/backport-3.20 label Jun 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ClosedFileSystemException or NullPointerException thrown when SharedArchivePathTree is opened and closed concurrently #48221

ClosedFileSystemException or NullPointerException thrown when SharedArchivePathTree is opened and closed concurrently #48221

Uh oh!

ppalaga commented Jun 4, 2025 •

edited by geoand

Loading

Uh oh!

ppalaga commented Jun 4, 2025

Uh oh!

aloubyansky left a comment

Uh oh!

This comment has been minimized.

ppalaga commented Jun 5, 2025

Uh oh!

geoand commented Jun 5, 2025

Uh oh!

This comment has been minimized.

aloubyansky commented Jun 5, 2025

Uh oh!

ppalaga Jun 5, 2025

Uh oh!

aloubyansky Jun 5, 2025

Uh oh!

ppalaga Jun 5, 2025

Uh oh!

aloubyansky Jun 5, 2025

Uh oh!

This comment has been minimized.

quarkus-bot bot commented Jun 6, 2025 •

edited by github-actions bot

Loading

Uh oh!

gsmet commented Jun 10, 2025

Uh oh!

aloubyansky commented Jun 10, 2025

Uh oh!

aloubyansky commented Jun 10, 2025

Uh oh!

Uh oh!

Uh oh!

ClosedFileSystemException or NullPointerException thrown when SharedArchivePathTree is opened and closed concurrently #48221

ClosedFileSystemException or NullPointerException thrown when SharedArchivePathTree is opened and closed concurrently #48221

Uh oh!

Conversation

ppalaga commented Jun 4, 2025 • edited by geoand Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ppalaga commented Jun 4, 2025

Uh oh!

aloubyansky left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

ppalaga commented Jun 5, 2025

Uh oh!

geoand commented Jun 5, 2025

Uh oh!

This comment has been minimized.

aloubyansky commented Jun 5, 2025

Uh oh!

ppalaga Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

aloubyansky Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

ppalaga Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

aloubyansky Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

quarkus-bot bot commented Jun 6, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Status for workflow Quarkus CI

Flaky tests - Develocity

⚙️ JVM Tests - JDK 21

📦 extensions/smallrye-reactive-messaging/deployment

⚙️ JVM Tests - JDK 17 Windows

📦 extensions/micrometer-opentelemetry/deployment

⚙️ Gradle Tests - JDK 17 Windows

📦 integration-tests/gradle

⚙️ JVM Integration Tests - JDK 17

📦 integration-tests/opentelemetry

📦 integration-tests/opentelemetry-grpc-only

⚙️ JVM Integration Tests - JDK 21

📦 integration-tests/opentelemetry

Uh oh!

gsmet commented Jun 10, 2025

Uh oh!

aloubyansky commented Jun 10, 2025

Uh oh!

aloubyansky commented Jun 10, 2025

Uh oh!

Uh oh!

Uh oh!

ppalaga commented Jun 4, 2025 •

edited by geoand

Loading

quarkus-bot bot commented Jun 6, 2025 •

edited by github-actions bot

Loading

Status for workflow `Quarkus CI`