Issue838 enforece limited queue

[HansVRP]

No description provided.

[soxofaan]

FYI: I pushed a commit to simplify the run/queuing logic a bit (and merge master)

[soxofaan]

I also noticed that some tests are getting more flaky with these changes,
not sure yet what is going on

[HansVRP]

checking if it can be related to omitting this 'created' state

[soxofaan]

checking if it can be related to omitting this 'created' state

I was also looking at this, and it is indeed a subtle interaction between

• not counting "created" as "running"
• mixing openeo job status (queued, running) and internal status (e.g. queued_for_start) in the same "status" column"
• unpredictability of threading

Apparently, it can happen that the "status" of a job goes from "queued_for_start" to "created". While this looks wrong, it's actually possible now that status polling and job starting happens in different threads.
If this happens on backend "bar", the job manager (wrongly) thinks that "bar" has capacity, and launches the last job there, while it used to launch it on "foo" (which was set up to actually finish faster than "bar")

[soxofaan]

I tried to put "created" back (also on master),
Let's see what this gives

[soxofaan]

flakiness seems to be solved again by adding "created" to "running" count, oh well

I'll add a changelog entry and merge then I think