Ceph stress testing runs into "icechunk.IcechunkError: × session error: error writing object to object store service error"

[petermkr]

What happened?

When trying to run highly parallel code against a Ceph S3-compatible object store with Icechunk, the program exited with the following error message:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib64/python3.13/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
~~~~^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/username/icechunk-demo/demo/demo06_icechunk_parallel_partial_write.py", line 100, in worker
session.commit(message)
~~~~~~~~~~~~~~^^^^^^^^^
File "/home/username/.cache/pypoetry/virtualenvs/icechunk-demo-ZY7yPLKK-py3.13/lib64/python3.13/site-packages/icechunk/session.py", line 271, in commit
return self._session.commit(
~~~~~~~~~~~~~~~~~~~~^
message, metadata, rebase_with=rebase_with, rebase_tries=rebase_tries
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
icechunk.IcechunkError: × session error: error writing object to object store service error
│
│ context:
│ 0: icechunk::storage::s3::write_ref
│ with ref_key="branch.main/ref.json" previous_version=VersionInfo { etag: Some(ETag(""7a9369dcfbc377183fef8e9d8f0bf40e"")), generation: None }
│ at icechunk/src/storage/s3.rs:819
│ 1: icechunk::refs::update_branch
│ with name="main" new_snapshot=6C5N1DRCE5SP0CV9MJCG current_snapshot=Some(0GYPX3ZH15N48YG7V8K0)
│ at icechunk/src/refs.rs:173
│ 2: icechunk::session::_commit
│ with Worker (init_time=18, lead_time_index=03): Wrote chunk data. rewrite_manifests=false
│ at icechunk/src/session.rs:985
│ 3: icechunk::session::commit
│ with Worker (init_time=18, lead_time_index=03): Wrote chunk data.
│ at icechunk/src/session.rs:948
│

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/username/icechunk-demo/demo/demo06_icechunk_parallel_partial_write.py", line 199, in
main()
~~~~^^
File "/home/username/icechunk-demo/demo/demo06_icechunk_parallel_partial_write.py", line 168, in main
stats: list[tuple[int, int, int, int, float, float, float]] = pool.starmap(worker, tasks)
~~~~~~~~~~~~^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib64/python3.13/multiprocessing/pool.py", line 774, in get
raise self._value
icechunk.IcechunkError: × session error: error writing object to object store service error
│
│ context:
│ 0: icechunk::storage::s3::write_ref
│ with ref_key="branch.main/ref.json" previous_version=VersionInfo { etag: Some(ETag(""7a9369dcfbc377183fef8e9d8f0bf40e"")), generation: None }
│ at icechunk/src/storage/s3.rs:819
│ 1: icechunk::refs::update_branch
│ with name="main" new_snapshot=6C5N1DRCE5SP0CV9MJCG current_snapshot=Some(0GYPX3ZH15N48YG7V8K0)
│ at icechunk/src/refs.rs:173
│ 2: icechunk::session::_commit
│ with Worker (init_time=18, lead_time_index=03): Wrote chunk data. rewrite_manifests=false
│ at icechunk/src/session.rs:985
│ 3: icechunk::session::commit
│ with Worker (init_time=18, lead_time_index=03): Wrote chunk data.
│ at icechunk/src/session.rs:948
│

What did you expect to happen?
The commit should be successful with a Ceph Object Gateway backend.

Minimal Complete Verifiable Example
Given that I have a concrete idea of where things might not be working together, I have omitted the example. If you require it, just tell me.

Anything else we need to know?
I have analyzed the problem and found the following:

In this code (part of write_ref in s3.rs),

        match res {
            Ok(_) => Ok(WriteRefResult::Written),
            Err(err) => {
                let code = err.as_service_error().and_then(|e| e.code()).unwrap_or("");
                if code.contains("PreconditionFailed")
                    || code.contains("ConditionalRequestConflict")
                {
                    Ok(WriteRefResult::WontOverwrite)
                } else {
                    Err(Box::new(err).into())
                }
            }
        }

there are two error codes which cause the operation to retry.
While Ceph correctly returns PreconditionFailed for HTTP 412, it uses ConcurrentModification for HTTP 409 as verified with Wireshark:

This means that the code follows the "real error path" and aborts.

If you agree with this analysis, would it then be possible to add ConcurrentModification to the or-condition in the mentioned code, or check more generally for generic 409 HTTP responses?

Environment
platform: Fedora Linux 42 (Workstation Edition) (6.15.10-200.fc42.x86_64)
python: 3.13.7
icechunk: 1.1.4
zarr: 3.1.0
numcodecs: 0.16.2
xarray: 2025.8.0

Thanks for the great tool!

[paraseba]

Thank you for the deep analysis @petermkr ! This sounds like the right addition. Just to make sure, was your code actually doing concurrent commits? I wouldn't expect a condition failure if you weren't.

Your proposed change seems to be the right approach, different object stores differ in behavior on these (more modern) conditional writes. Could you submit a PR with the fix?

Thank you!

[petermkr]

Thanks for your fast reply, @paraseba! Yes, we were driving it to run into conflicts regularly.

I can definitely contribute a PR with the fix.

Could you give me some pointers to tools you use for development?
In https://icechunk.io/en/latest/contributing/, the section for Rust is not ready yet, so it would be great to profit from your experiences (which IDE/tools to use) when entering a new programming language.
Mostly the features integrating Python/Rust for development would be of interest.

[paraseba]

Fantastic @petermkr !

To setup your Rust environment I recommend rustup. This is what most people use, and it's a single command that gives you the compiler, package manager, LSP, etc. For editor I use Neovim, but probably most people use vscode. They all work very similarly because they use the official LSP provider, rust-analyzer, that gets installed via rustup.

If you happen to be a Nix user, we have a flake in the project that you can use to get everything installed, but I don't recommend it if you are not already a Nix user.

Once you have Rust setup, you'll also need docker to be able to run tests.

Finally, you can go to the root directory in the project and run:

docker compose up -d
cargo test --all-targets

and that should run (and pass) all Rust tests.

With that out of the way, it's time for Python. You install maturin with the package manager of choice, go to the icechunk-python directory and run maturin dev, for example, in my case I do

uvx maturin dev

This builds the Python module that provides the bindings. Still in the same directory, you should be able to run pytest to get all the python tests.

Hope this helps, but don't hesitate to ask here on in the icechunk channel of our community slack

Thank you!

[petermkr]

Hi @paraseba, I have just tested the freshly released Icechunk 1.1.5, and I can confirm that this resolves the problem.

Thanks for the quick release!