`test_protocols.test_output_compliance`: `case` should be copied

[jakkdl]

Describe the bug

The test_output_compliance test does copy.deepcopy on json_description, but not on case. This is despite both being shared among test function invocations and both being modified. This means that if the test gets parametrized, or is rerun due to pytest-rerunfailures, or similar, the test fails.

botocore/tests/unit/test_protocols.py

Lines 197 to 206 in a4bd6ae

	@pytest.mark.parametrize(
	"json_description, case, basename", _compliance_tests(TestType.OUTPUT)
	)
	def test_output_compliance(json_description, case, basename):
	service_description = copy.deepcopy(json_description)
	operation_name = case.get('given', {}).get('name', 'OperationName')
	service_description['operations'] = {
	operation_name: case,
	}
	case['response']['context'] = {'operation_name': operation_name}

Regression Issue

• Select this option if this issue appears to be a regression.

Possible Solution

Add

case = copy.deepcopy(case)

Additional Information/Context

This was noticed in aio-libs/aiobotocore#1085

[adev-code]

Hello @jakkdl, thanks for reaching out. I have tested the code and confirmed it does re-use the same case object between test runs, causing failures when tests are parametrized or rerun. I will bring this up to the team if case = copy.deepcopy(case) can be added. Thank you.

[adev-code]

Hi @jakkdl, thanks for the patience. Could you please provide your use case and also how you are implementing or reproducing this? Also, please add boto3.set_stream_logger('') for logs too. Thanks.

[jakkdl]

Hi @jakkdl, thanks for the patience. Could you please provide your use case and also how you are implementing or reproducing this? Also, please add boto3.set_stream_logger('') for logs too. Thanks.

https://github.com/aio-libs/aiobotocore/ runs a copy of the test suite from botocore with minimal changes, maintaining a diff to make sure they match the behavior of upstream. But aio-libs/aiobotocore#1085 wanted to introduce parametrization of the test suite, which made this test fail.
https://github.com/jakkdl/aiobotocore/blob/25351b7064451a15df957fea66486d94a841b69e/tests/conftest.py#L703-L718

It's not really a problem currently for botocore, but I could see it biting somebody in the future. Either if somebody adds parametrization, adds another test that makes use of _compliance_tests, uses pytest-rerunfailures and it fails for a diff reason the first time, etc. And it saves aiobotocore having to maintain a diff relative to upstream :)

I don't think logs are terribly useful here, and idk where to add boto3.set_stream_logger to do anything useful - the relevant test file isn't importing boto3 in the first case.