test: refactor support bundle and checks to improve reliability #504
Conversation
| ignore_extras=True, | ||
| ignore_missing=True, | ||
| pre_expected_names=ser_files, | ||
| post_expected_names=[] |
There was a problem hiding this comment.
note that I removed ignore extras and missing. If the auto test is still flaky (cause 2 bundles) - I can re-add them but I would want to make sure the warnings show up in the pytest results
azext_edge/tests/helpers.py
Outdated
| expected_names = [name.split(".")[0] for name in expected_names] | ||
| result_names = [name.split(".")[0] for name in result_names] | ||
| extra_names = [name for name in result_names if name not in expected_names] | ||
| # vilit double check if ^ is still needed |
There was a problem hiding this comment.
currently with int testing - this is no longer needed, might not be true with containerized tests
| assert pod_evals[0]["name"] == f"pod/{pod_name}" | ||
|
|
||
| # check phase and conditions | ||
| phase_conditions_eval = [pod for pod in pod_evals if "status.phase" in pod["value"]].pop() |
There was a problem hiding this comment.
nit, more like [eval for eval in pod_evals if "status.phase" in eval["value"]].pop() ?
Also usage of 1 .pop() implies a single eval should be asserted for status.phase, but then why list comprehension?
There was a problem hiding this comment.
have not looked at this code in a while - prob need to use next instead
|
new example of error text: (note that this is the mq test so extra name aio-broker-backend-1-0 is ok) the rest is not ok: |
well... I had to fix checks too...
The goal here is to make the tests more tolerant of state changes. Now there is:
For the comparison, first the before (1) and the support bundle (2) are compared. If there are extra or missing resources, the after (3) is also compared.
NOTE: this is ONLY for workload resources since the custom resources seem to be more "stable".
This will not get rid of the possibility of a resource showing up after 1, being caught in the support bundle, and being deleted before 3 (and the reverse missing resource scenario).
Some other improvements:
Extra resources do not have this (as in the kubectl mapping is missing).
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.
Thank you for contributing to Azure IoT Operations tooling!
This checklist is used to make sure that common guidelines for a pull request are followed.
General Guidelines
Intent for Production
devormainare of production grade. Corollary to this, any merged contributions to these branches may be deployed in a public release at any given time. By checking this box, you agree and commit to the expected production quality of code.Basic expectations
pytest <project root> -vv. Please provide evidence in the form of a screenshot showing a succesful run of tests locally OR a link to a test pipeline that has been run against the change-set..pylintrcand.flake8rules? Look at the CI scripts for example usage.Azure IoT Operations CLI maintainers reserve the right to enforce any of the outlined expectations.
A PR is considered ready for review when all basic expectations have been met (or do not apply).