Test Automation Performance Improvements + Streamlining

[phillip-stephens]

Closes #112

Revamp of testing framework to leverage docker compose to improve speed as well as allow tests to even be run on ARM MacBooks. Consolidation of test build environment into docker-compose.ymlgreatly reduced the files required for our build system.

Notable Changes

• Tests run consistently in 4 min -> 4.5 mins on GH runners, . Even faster on a well-provisioned VM.
• Tests will fail faster. Previously, each module's test.sh ran, generating a scan output. After all modules had run, these schemas were validated. This meant a failure in the first would wait to fail until all had run. Now we validate as we scan, letting us fail fast.
• Remove all setup.sh and cleanup.sh scripts, these are all consolidated in a single docker-compose.yml file
• Create a base zgrab2_ipp_service_base since both ipp images were installing cups and cups-pdf to improve build efficiency
• Port main integration_tests/test.sh to python3 for easier maintainability
• Update README.md to remove out-dated guidance on using Python2
• Update README.md to show how you can run only a single module's integration tests

Performance Improvements

Github Actions Runner
10 mins master -> 4min branch
MacBook Air
Could not run master -> 2 mins 50 sec. branch (taking advantage of docker caching)

Testing

Positive Test Case

https://github.com/phillip-stephens/zgrab2/actions/runs/13206866505 -> 3 min 57 sec.

Negative Test Case

Here, I've added a typo in the amqp module so that schema validation should fail. It fails immediately rather than waiting for all tests to run and then performing validation on the results.

Schema validation failed for amqp091/zgrab-output/amqp091/3.12.14.json:
ERROR:root:data.amqp091.result: result: versio_major is not a valid subkey
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/__main__.py", line 136, in <module>
    main()
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/__main__.py", line 121, in main
    record.validate(
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/compounds.py", line 582, in validate
    self._handle_validation_exception(calculated_policy, e)
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/keys.py", line 213, in _handle_validation_exception
    raise e
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/compounds.py", line 574, in validate
    self.definition[subkey].validate(
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/compounds.py", line 404, in validate
    self._handle_validation_exception(calculated_policy, e)
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/keys.py", line 213, in _handle_validation_exception
    raise e
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/compounds.py", line 396, in validate
    self.definition[subkey].validate(
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/compounds.py", line 404, in validate
    self._handle_validation_exception(calculated_policy, e)
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/keys.py", line 213, in _handle_validation_exception
    raise e
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/compounds.py", line 396, in validate
    self.definition[subkey].validate(
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/compounds.py", line 404, in validate
    self._handle_validation_exception(calculated_policy, e)
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/keys.py", line 217, in _handle_validation_exception
    raise e
  File "/Users/phillip/zgrab2-pstephens/venv/lib/python3.12/site-packages/zschema/compounds.py", line 392, in validate
    raise DataValidationException(
zschema.keys.DataValidationException: result: versio_major is not a valid subkey

Potential Areas of Concern

By using docker compose, we bring up all containers at once. This takes around 3 GB of RAM but CPU usage quickly drops back to baseline after the containers come up since nothing is actively computing anything. IMO, the increased RAM usage by bringing up all containers at once is worth the efficiency gains of building/starting all docker images in parallel.