How to deal with stored states using Python spikequeue

[mstimberg]

In this discussion on our discourse forum, I realized that we have a problem for a not very common corner case: if someone used the Python spike queue fallback and stored the state of the network to disk with Network.store(filename=...), this file can no longer be loaded, since the Cython spike queue uses a slightly different format to store its internal state (I don't quite remember why, though…). Of course, using a Python spike queue state with the Cython spike queue was not working before, either, but now there is no way to run Brian without the Cython spike queue, so this can be considered a regression. Here's a simple example that stores a network with a spike queue to disk:

from brian2 import *

G = NeuronGroup(2, 'x: 1', threshold="True", reset="")
S = Synapses(G, G, on_pre = "x+=1")
S.connect(j="i")
S.delay = [2*defaultclock.dt, 4*defaultclock.dt]
run(defaultclock.dt)
store(filename="test.pkl")

We can load the stored state with pickle:

>>> import pickle
>>> with open("test.pkl", "rb") as f:
...     d = pickle.load(f)
... 
>>> d["default"]["synapses_pre"]["_spikequeue"]

With the Python spike queue, this gives:

(0.0001, array([[1, 0],
       [3, 1]]), (5, 4))

And with the Cython spike queue:

(1, [[], [], [0], [], [1]])

Not surprisingly, loading one format with the other version fails. If you try to load the Python spike queue state with the Cython version, the error looks like this:

Traceback (most recent call last):
  File "/home/mstimberg/scratch/pickle_test.py", line 6, in <module>
    restore(filename="test.pkl")
  File "/home/mstimberg/git/brian2/brian2/core/magic.py", line 467, in restore
    magic_network.restore(
  File "/home/mstimberg/git/brian2/brian2/core/magic.py", line 273, in restore
    super().restore(
  File "/home/mstimberg/git/brian2/brian2/core/base.py", line 346, in device_override_decorated_function
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/mstimberg/git/brian2/brian2/core/network.py", line 706, in restore
    obj._restore_from_full_state(state[obj.name])
  File "/home/mstimberg/git/brian2/brian2/synapses/synapses.py", line 452, in _restore_from_full_state
    self.queue._restore_from_full_state(queue_state)
  File "brian2/synapses/cythonspikequeue.pyx", line 72, in brian2.synapses.cythonspikequeue.SpikeQueue._restore_from_full_state
    if real_delays.dtype == np.float32:
  File "<stringsource>", line 189, in pair.from_py.__pyx_convert_pair_from_py_int__and_std_3a__3a_vector_3c_std_3a__3a_vector_3c_int32_t_3e____3e___
ValueError: too many values to unpack (expected 2)

Changing the format of Cython spike queue to use the same internal format as the Python spike queue would break backwards compatibility in an even worse way, since most users have been using the Cython spike queue. Therefore, I think the cleanest way is probably to special case this situation here:

brian2/brian2/synapses/synapses.py

Line 452 in 31f945d

self.queue._restore_from_full_state(queue_state)

and manually convert the state from the "Python format" to the "Cython format" (which is luckily quite straightforward to detect, since it consists of a 3-tuple instead of a 2-tuple).

Ping @Legend101Zz 😊 This is not a high priority right away, but it would be could to deal with this before the release that removes the Python fallback.

[Legend101Zz]

@mstimberg Great thanks noted ... I'll keep working on this issue and also if possible can you please create the random generator issue too that we dicussed in last week's meeting , I'll start looking into that too :)