Seeded RNGs only work reproducibly within a cell

[penelopeysm]

Description

Calling Random.seed! at the top of a notebook doesn't ensure reproducibility of random number generation in other cells; it only works within the same cell that seed! is called.

I'm not sure if this is intentional, but it's slightly counterintuitive behaviour so if it's intentional it might be good to document it somewhere.

A related issue (and perhaps an example of how this behaviour is confusing!) is that the Pumas tutorial on reproducibility says that the second batch of samples in Random.seed!(123); rand(3); rand(3) should always result in "the same behavior [...] every time we run the code". This is true if done in a REPL, but the Quarto source has the second rand(3) in a different cell and consequently the second rand(3) will not always return the same numbers.

MWE

The following notebook works to demonstrate the behaviour. The first call to randn() should always give the same result no matter how many times the notebook is rendered. The second call however will not.

---
engine: julia
---

```{julia}
versioninfo()
```

```{julia}
using Random
Random.seed!(42)
randn()
```

```{julia}
randn()
```

My versioninfo from the notebook above

Julia Version 1.10.5
Commit 6f3fdf7b362 (2024-08-27 14:19 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 10 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)
Environment:
  JULIA_LOAD_PATH = @:@stdlib

[devmotion]

I think this might be an upstream issue but I'm not too familiar with Malt so maybe it's intended and/or could be avoided:

julia> using Malt

julia> w = Malt.Worker();

julia> Malt.remote_eval_fetch(w, quote
           using Random
           Random.seed!(42)
           randn()
       end)
0.7883556016042917

julia> Malt.remote_eval_fetch(w, quote
           randn()
       end)
0.8653602653617717

julia> Malt.remote_eval_fetch(w, quote
           using Random
           Random.seed!(42)
           randn()
       end)
0.7883556016042917

julia> Malt.remote_eval_fetch(w, quote
           randn()
       end)
-0.6715764280647957

[MichaelHatherly]

Will investigate, thanks for the thorough issue description @penelopeysm.

[jkrumbiegel]

Maybe this has something to do with task-local RNG state? Could be that Malt evals in different tasks per fetch and that this has an effect on the RNG. If they were doing some random number generation themselves that changed the RNG state, then it should still lead to the same numbers if it was the same n operations between two remote_eval_fetches so that can't be it.

[devmotion]

Hmm, the RNG object id seems to be stable, so it seems to be the same RNG?

julia> Malt.remote_eval_fetch(w, quote
           objectid(Random.default_rng())
       end)
0xffffffff0968f0b4

julia> Malt.remote_eval_fetch(w, quote
           objectid(Random.default_rng())
       end)
0xffffffff0968f0b4

julia> Malt.remote_eval_fetch(w, quote
           objectid(Random.default_rng())
       end)
0xffffffff0968f0b4

julia> Malt.remote_eval_fetch(w, quote
           objectid(Random.default_rng())
       end)
0xffffffff0968f0b4

[jkrumbiegel]

I think that's because it's TaskLocalRNG and that's a singleton type:

julia> TaskLocalRNG() === TaskLocalRNG()
true

The RNG state for that is stored in the task itself.

[MichaelHatherly]

cc @fonsp in case you happen to have any insights.

[jkrumbiegel]

This shows that we get different task-local RNG state in a new remote_eval_fetch:

julia> Malt.remote_eval_fetch(w, quote
           using Random
           Random.seed!(123)

           rngfields = filter(fieldnames(Task)) do field
               startswith(string(field), "rng")
           end
           state = NamedTuple(map(rngfields) do field
               field => getfield(Base.current_task(), field)
           end)
       end)
(rngState0 = 0xfefa8d41b8f5dca5, rngState1 = 0xf80cc98e147960c1, rngState2 = 0x20e2ccc17662fc1d, rngState3 = 0xea7a7dcb2e787c01, rngState4 = 0xf4e85a418b9c4f80)

julia> Malt.remote_eval_fetch(w, quote
           rngfields = filter(fieldnames(Task)) do field
               startswith(string(field), "rng")
           end
           state = NamedTuple(map(rngfields) do field
               field => getfield(Base.current_task(), field)
           end)
       end)
(rngState0 = 0x4e291591396795b9, rngState1 = 0xc9a680b08b4c9006, rngState2 = 0x3bb5088d1c600f58, rngState3 = 0xe2bb063e7e800a67, rngState4 = 0xe264ff418d928577)

Probably because each invocation is a different task?

julia> Malt.remote_eval_fetch(w, quote
           objectid(Base.current_task())
       end)
0xb68e01fb526e2337

julia> Malt.remote_eval_fetch(w, quote
           objectid(Base.current_task())
       end)
0x49246d797797f2ad

[jkrumbiegel]

If this is hard to fix in Malt, I guess we could store the rngState fields after a task runs a cell, and then first thing in the next cell, mutate that one's rngState fields so they take the same values. Not sure if Malt can be configured to use only a single task.

[devmotion]

Maybe one could take some inspiration from how Test resets the task-local RNG: https://github.com/JuliaLang/julia/blob/1438b1578941d0f1cc8f8c958cf3bd2927fd482c/stdlib/Test/src/Test.jl#L1694-L1720

[jkrumbiegel]

Interesting, it seems this looked quite different on 1.10 where Random.get_tls_seed() did not exist

https://github.com/JuliaLang/julia/blob/6f3fdf7b36250fb95f512a2b927ad2518c07d2b5/stdlib/Test/src/Test.jl#L1567-L1578

[fonsp]

@Pangoraw knows best about RNGs and threading in Pluto :)

[Pangoraw]

remote_eval_fetch indeed spawns a new task at each call. One solution would be for QuartoNotebookWorker to manage its own persistent task for evaluation and then communicate with this task using channels.