How can I build a test environment with different emscripten settings?

[martenrichter]

I am currently trying to test if squashfs support may reduce xeus-python start time.

However, I need for the current experiment the SHARED_MEMORY feature.
Therefore, I created a fork of the recipes and build the packages by hand with pixi:
https://github.com/martenrichter/recipes/tree/testsharedmemory

(And does not only contain fixes due to shared memory, but also some build fixes due to newer version, they may be also of help in your work).
However, I do not seem to have gotten a build with consistent versions as I get (with and without squashfs support):

    at xpython.wasm.PyDict_New (http://127.0.0.1:8000/xeus/xeus-python-kernel/bin/xpython.wasm:wasm-function[20889]:0x7e8fd8)
    at xpython.wasm._PyUnicode_InitGlobalObjects (http://127.0.0.1:8000/xeus/xeus-python-kernel/bin/xpython.wasm:wasm-function[22208]:0x85b230)
    at xpython.wasm.pycore_interp_init (http://127.0.0.1:8000/xeus/xeus-python-kernel/bin/xpython.wasm:wasm-function[23758]:0x91e352)
    at xpython.wasm.pyinit_config (http://127.0.0.1:8000/xeus/xeus-python-kernel/bin/xpython.wasm:wasm-function[23739]:0x91c045)
    at xpython.wasm.Py_InitializeFromConfig (http://127.0.0.1:8000/xeus/xeus-python-kernel/bin/xpython.wasm:wasm-function[23738]:0x91bbd5)
    at xpython.wasm.pybind11::initialize_interpreter(PyConfig*, int, char const* const*, bool) (http://127.0.0.1:8000/xeus/xeus-python-kernel/bin/xpython.wasm:wasm-function[18887]:0x72a143)
    at invoke_viiii (http://127.0.0.1:8000/xeus/xeus-python-kernel/bin/xpython.js:29061:29)
    at xpython.wasm.pybind11::scoped_interpreter* emscripten::internal::operator_new<pybind11::scoped_interpreter>() (http://127.0.0.1:8000/xeus/xeus-python-kernel/bin/xpython.wasm:wasm-function[18875]:0x72893a)
    at xpython.wasm.emscripten::internal::Invoker<emscripten::internal::rvp::take_ownership, pybind11::scoped_interpreter*>::invoke(pybind11::scoped_interpreter* (*)()) (http://127.0.0.1:8000/xeus/xeus-python-kernel/bin/xpython.wasm:wasm-function[18874]:0x7288c0)
    at constructor _Interpreter (eval at newFunc (http://127.0.0.1:8000/xeus/xeus-python-kernel/bin/xpython.js:4498:23), <anonymous>:4:10)

I assume that it is caused by versions that do not seem to be compatible.
So my question is, how can I get a development environment with a working set of versions?

[DerThorsten]

Hi, currently its impossible to get a develpment environment with a working set of versions.
With each emscripten update we need to update a lot of code because usualy a lot of things break. Also we need to patch emscripten itself.
We are currently working on a update to use the emscripten 4.x family but I have not yet uploaded many packages.

[martenrichter]

Thanks, that is sad and there isn't an older tag or branch I can use.
I just want to test squashfs support. (and if it is beneficial...)
Otherwise, I have to wait... (and of course I am looking forward for emscripten 4.0, some feature will make the squashfs test easier).

[DerThorsten]

@martenrichter what is the minimal version of emscripten you need? You can always just create a a custom version of the compiler pkg/recipe and change the emscripten version in the variant file

[martenrichter]

Well, I need support for SHARED_MEMORY, that is already in the 3.1 branch as it was added more than 2 years ago and also atomics are necessary but these are also there for some time. So I am not picky with this respect.
But I do not need a different emscripten version the old branch is fine, I need to recompile the other packages to get a minimal xeus-python package with different build flags.
The emscripten 4. branch will come in handy, if memory grows, as this can be handled easier with the SharedArrayBuffer, but for the first test the 3 branch is fine.

So far I have just checked out the cloned branch and modified some build file and build them manually with pixi.

[DerThorsten]

So my question is, how can I get a development environment with a working set of versions?

I am not sure what you mean? You need a development version of what?
If emscripten 3.1.73 (the current version) is fine, you probably just need to compile the host, ie xeus-python, and likely the dependencyies of xeus-python (python, pyjs) with the respective SHARED_MEMORY / atomics flags.

The emscripten 4. branch will come in handy, if memory grows, as this can be handled easier with the SharedArrayBuffer, but for the first test the 3 branch is fine.

I am a bit sceptical of this. SHARED_MEMORY / atmics and SharedArrayBuffer all require special server settings (https://web.dev/articles/coop-coep), therefore we cannot realy rely on solution which require SharedArrayBuffer / threads and similar constructs.

[martenrichter]

So my question is, how can I get a development environment with a working set of versions?

I am not sure what you mean? You need a development version of what? If emscripten 3.1.73 (the current version) is fine, you probably just need to compile the host, ie xeus-python, and likely the dependencyies of xeus-python (python, pyjs) with the respective SHARED_MEMORY / atomics flags.

Sure, exactly I need just need to recompile it, and but I need to know a consistent set of versions.

The emscripten 4. branch will come in handy, if memory grows, as this can be handled easier with the SharedArrayBuffer, but for the first test the 3 branch is fine.

I am a bit sceptical of this. SHARED_MEMORY / atmics and SharedArrayBuffer all require special server settings (https://web.dev/articles/coop-coep), therefore we cannot realy rely on solution which require SharedArrayBuffer / threads and similar constructs.

Yes, and no. SharedArrayBuffer is only needed to get the full benefits (that means downloading on demand in the background), it is possible to realize squashfs without it, but then you have to download the whole file beforehand. (I have also tried asyndify but it does not work well with python just in time compilation ....).
But I want to start with the complete solution, and only if this gives a substantial boost in startup time, it is worth to pursue the idea further and to code the fallbacks.

[DerThorsten]

Sure, exactly I need just need to recompile it, and but I need to know a consistent set of versions.

I think whatever version is currently in the recipes should be fine (python 3.13, pyjs 3.1.0)

[DerThorsten]

Yes, and no. SharedArrayBuffer is only needed to get the full benefits (that means downloading on demand in the background),

So SHARED_MEMORY is not dragging in SharedArrayBuffers?

[martenrichter]

Yes, and no. SharedArrayBuffer is only needed to get the full benefits (that means downloading on demand in the background),

So SHARED_MEMORY is not dragging in SharedArrayBuffers?

Yes on default it is. But you can overwrite this in the module initialization, and this what one would do, if the fallback is needed. (I have checked the emscripten source code to see what the flag does.)