improvement(tests): Use pytest-xdist to run tests

[pehala]

• Shortens time for unittests from 13 to 7 minutes (CI run for this PR) with 2 workers
  • Enabled by recent unit tests improvements
• Shortens time for integration tests from 40minutes to 14minutes with 4 workers

Testing

• locally
• CI

PR pre-checks (self review)

• I added the relevant backport labels
• I didn't leave commented-out/debugging code

Reminders

• Add New configuration option and document them (in sdcm/sct_config.py)
• Add unit tests to cover my changes (under unit-test/ folder)
• Update the Readme/doc folder relevant to this change (if needed)

[pehala]

Reliably fails on CI due to:
Cannot contact i-0ec1cf98cc5aaf828: java.lang.InterruptedException

Probably overload?

[fruch]

keep in mind the machine we are using for CI isn't not like you laptop.
it's only 4 cpu

and as you can see it's making thing slower... and not faster.

[pehala]

keep in mind the machine we are using for CI isn't not like you laptop. it's only 4 cpu

and as you can see it's making thing slower... and not faster.

Its making them not work at all (for now). Can I see what is the problem, I see it is interrupted but I would like to know why. (Is it memory? is it CPU? is it some other limit being broken?)

Also I do not think 4 workers should overload 4 CPU system

[fruch]

keep in mind the machine we are using for CI isn't not like you laptop. it's only 4 cpu

and as you can see it's making thing slower... and not faster.

Its making them not work at all (for now). Can I see what is the problem, I see it is interrupted but I would like to know why. (Is it memory? is it CPU? is it some other limit being broken?)

We don't have any machinery like those for those tests

You can connect to a running builder, or spin your own SCT runner with the same size, and experiment with it.

Also I do not think 4 workers should overload 4 CPU system

I don't know if it overload, bit having 12 cores running in 3 min, I would expect them to be slower when using 1/3 of the cores

[pehala]

v3:

• Lower default factor to 2 for unit tests
• Add option to hydra to customize factor
• Add paralelization to integration tests
  • With 4 workers the time is now 16 minutes

It seems that since integration tests provision their own machine, paralelization is not a problem. I will do last few tests with unittests and if they do not still reliably work I will open another PR just for integration tests

EDIT: It seems to work with two workers, which roughly halves the execution time (13 -> 7min), still valuable improvement in my opinion.
Integration tests are a huge improvement from 40 to 14 min