APEx tests intermittently fail due to HEAD /openeo-data-staging-waw4-1/batch_jobs/{job_id}/{asset} failing with a 502

[JeroenVerstraelen]

The APEx tests/test_benchmarks.py::test_run_benchmark[LCFM_LSF_MONTHLY_S2_MEDIAN] fails intermittently because of a 502 when performing a HEAD request to /openeo-data-staging-waw4-1/batch_jobs/{job_id}. Doing the same HEAD requests manually later worked just fine.

Should a single 502 really break the entire download_files? Note that it has to download a lot of files.
The issue is likely somewhere in download_files in the openeo-python-client but I have not been able to check yet.

Debug logs

DEBUG    openeo.rest._connection:_connection.py:107 Request `HEAD https://s3.stag.waw3-1.openeo.v1.dataspace.copernicus.eu/openeo-data-staging-waw4-1/batch_jobs/j-2512101909374e0fb2599653c4c584a1/LCFM_LSF-MONTHLY_V008_2020_29TNE_2020-12-01Z_B07.tif_gdalinfo.json?X-Proxy-Head-As-Get=true&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=197503f65f0b4ac184a66a2a592eaf33%2F20251210%2Fwaw4-1%2Fs3%2Faws4_request&X-Amz-Date=20251210T191741Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Security-Token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlX2FybiI6ImFybjpvcGVuZW93czppYW06Ojpyb2xlL29wZW5lby1kYXRhLXN0YWdpbmctd2F3NC0xLXdvcmtzcGFjZSIsImluaXRpYWxfaXNzdWVyIjoib3BlbmVvLnN0YWcud2F3My0xLm9wZW5lby1pbnQudjEuZGF0YXNwYWNlLmNvcGVybmljdXMuZXUiLCJodHRwczovL2F3cy5hbWF6b24uY29tL3RhZ3MiOnsicHJpbmNpcGFsX3RhZ3MiOnsiam9iX2lkIjpbImotMjUxMjEwMTkwOTM3NGUwZmIyNTk5NjUzYzRjNTg0YTEiXSwidXNlcl9pZCI6WyIyNWM2OTA3NS0zOTI5LTRmMDctYjU4Ni1iM2E4NGY2Nzk3YTIiXX0sInRyYW5zaXRpdmVfdGFnX2tleXMiOlsidXNlcl9pZCIsImpvYl9pZCJdfSwiaXNzIjoic3RzLnN0YWcud2F3My0xLm9wZW5lby52MS5kYXRhc3BhY2UuY29wZXJuaWN1cy5ldSIsInN1YiI6Im9wZW5lby1kcml2ZXIiLCJleHAiOjE3NjU0Mzc0NjEsIm5iZiI6MTc2NTM5NDI2MSwiaWF0IjoxNzY1Mzk0MjYxLCJqdGkiOiJkMTUyNDIyNi1jYWY5LTQ1Y2MtOGI2Yy0wOTI1ODNiYzkxMGEiLCJhY2Nlc3Nfa2V5X2lkIjoiMTk3NTAzZjY1ZjBiNGFjMTg0YTY2YTJhNTkyZWFmMzMifQ.mTyiVMV6VJzthxd5tC5Us5euSvCtlrJw3o1L7mKpkLZIpZkRn6FVR3Otq36-Yc6UsiU4oV4x25S9CXYLa5GqpZpgvix84SPb6ZgOZdrRTRp-drF082i90VxCBBW8Ou9sCqbWp-Gz6S0NemfDHXjH6UmdDO24gQLg_3PpyNk_hJhXzxk8y6NSSpdMowxS-qmu9ZITaqZ-btXg7cL7UpJGp8IgT9UsG3Oi1pvG7ZDFV-fvsC_Hbfafihha7IY8wPDlen9hqBB7lCdr7dKIDAGj78bwG6YVfkP1GDsH6Lrx_w35tbcXghikMwEJpqhWDAcBYu0Yr4ES2tnBXYpYabdkEg&X-Amz-Signature=26dc4964add9268d1d47a54b944890ce89bb14a07b11e51334305e8b5fdc5479` with params None, headers None, auth NoneType, kwargs ['stream']
DEBUG    urllib3.connectionpool:connectionpool.py:544 https://s3.stag.waw3-1.openeo.v1.dataspace.copernicus.eu:443 "HEAD /openeo-data-staging-waw4-1/batch_jobs/j-2512101909374e0fb2599653c4c584a1/LCFM_LSF-MONTHLY_V008_2020_29TNE_2020-12-01Z_B07.tif_gdalinfo.json?X-Proxy-Head-As-Get=true&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=197503f65f0b4ac184a66a2a592eaf33%2F20251210%2Fwaw4-1%2Fs3%2Faws4_request&X-Amz-Date=20251210T191741Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Security-Token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlX2FybiI6ImFybjpvcGVuZW93czppYW06Ojpyb2xlL29wZW5lby1kYXRhLXN0YWdpbmctd2F3NC0xLXdvcmtzcGFjZSIsImluaXRpYWxfaXNzdWVyIjoib3BlbmVvLnN0YWcud2F3My0xLm9wZW5lby1pbnQudjEuZGF0YXNwYWNlLmNvcGVybmljdXMuZXUiLCJodHRwczovL2F3cy5hbWF6b24uY29tL3RhZ3MiOnsicHJpbmNpcGFsX3RhZ3MiOnsiam9iX2lkIjpbImotMjUxMjEwMTkwOTM3NGUwZmIyNTk5NjUzYzRjNTg0YTEiXSwidXNlcl9pZCI6WyIyNWM2OTA3NS0zOTI5LTRmMDctYjU4Ni1iM2E4NGY2Nzk3YTIiXX0sInRyYW5zaXRpdmVfdGFnX2tleXMiOlsidXNlcl9pZCIsImpvYl9pZCJdfSwiaXNzIjoic3RzLnN0YWcud2F3My0xLm9wZW5lby52MS5kYXRhc3BhY2UuY29wZXJuaWN1cy5ldSIsInN1YiI6Im9wZW5lby1kcml2ZXIiLCJleHAiOjE3NjU0Mzc0NjEsIm5iZiI6MTc2NTM5NDI2MSwiaWF0IjoxNzY1Mzk0MjYxLCJqdGkiOiJkMTUyNDIyNi1jYWY5LTQ1Y2MtOGI2Yy0wOTI1ODNiYzkxMGEiLCJhY2Nlc3Nfa2V5X2lkIjoiMTk3NTAzZjY1ZjBiNGFjMTg0YTY2YTJhNTkyZWFmMzMifQ.mTyiVMV6VJzthxd5tC5Us5euSvCtlrJw3o1L7mKpkLZIpZkRn6FVR3Otq36-Yc6UsiU4oV4x25S9CXYLa5GqpZpgvix84SPb6ZgOZdrRTRp-drF082i90VxCBBW8Ou9sCqbWp-Gz6S0NemfDHXjH6UmdDO24gQLg_3PpyNk_hJhXzxk8y6NSSpdMowxS-qmu9ZITaqZ-btXg7cL7UpJGp8IgT9UsG3Oi1pvG7ZDFV-fvsC_Hbfafihha7IY8wPDlen9hqBB7lCdr7dKIDAGj78bwG6YVfkP1GDsH6Lrx_w35tbcXghikMwEJpqhWDAcBYu0Yr4ES2tnBXYpYabdkEg&X-Amz-Signature=26dc4964add9268d1d47a54b944890ce89bb14a07b11e51334305e8b5fdc5479&_origin=apex-algorithms%2Fbenchmarks%2F%2Ftest_run_benchmark%5BLCFM_LSF_MONTHLY_S2_MEDIAN%5D HTTP/1.1" 502 0
DEBUG    openeo.rest._connection:_connection.py:134 openEO request `HEAD /openeo-data-staging-waw4-1/batch_jobs/j-2512101909374e0fb2599653c4c584a1/LCFM_LSF-MONTHLY_V008_2020_29TNE_2020-12-01Z_B07.tif_gdalinfo.json?X-Proxy-Head-As-Get=true&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=197503f65f0b4ac184a66a2a592eaf33%2F20251210%2Fwaw4-1%2Fs3%2Faws4_request&X-Amz-Date=20251210T191741Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Security-Token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlX2FybiI6ImFybjpvcGVuZW93czppYW06Ojpyb2xlL29wZW5lby1kYXRhLXN0YWdpbmctd2F3NC0xLXdvcmtzcGFjZSIsImluaXRpYWxfaXNzdWVyIjoib3BlbmVvLnN0YWcud2F3My0xLm9wZW5lby1pbnQudjEuZGF0YXNwYWNlLmNvcGVybmljdXMuZXUiLCJodHRwczovL2F3cy5hbWF6b24uY29tL3RhZ3MiOnsicHJpbmNpcGFsX3RhZ3MiOnsiam9iX2lkIjpbImotMjUxMjEwMTkwOTM3NGUwZmIyNTk5NjUzYzRjNTg0YTEiXSwidXNlcl9pZCI6WyIyNWM2OTA3NS0zOTI5LTRmMDctYjU4Ni1iM2E4NGY2Nzk3YTIiXX0sInRyYW5zaXRpdmVfdGFnX2tleXMiOlsidXNlcl9pZCIsImpvYl9pZCJdfSwiaXNzIjoic3RzLnN0YWcud2F3My0xLm9wZW5lby52MS5kYXRhc3BhY2UuY29wZXJuaWN1cy5ldSIsInN1YiI6Im9wZW5lby1kcml2ZXIiLCJleHAiOjE3NjU0Mzc0NjEsIm5iZiI6MTc2NTM5NDI2MSwiaWF0IjoxNzY1Mzk0MjYxLCJqdGkiOiJkMTUyNDIyNi1jYWY5LTQ1Y2MtOGI2Yy0wOTI1ODNiYzkxMGEiLCJhY2Nlc3Nfa2V5X2lkIjoiMTk3NTAzZjY1ZjBiNGFjMTg0YTY2YTJhNTkyZWFmMzMifQ.mTyiVMV6VJzthxd5tC5Us5euSvCtlrJw3o1L7mKpkLZIpZkRn6FVR3Otq36-Yc6UsiU4oV4x25S9CXYLa5GqpZpgvix84SPb6ZgOZdrRTRp-drF082i90VxCBBW8Ou9sCqbWp-Gz6S0NemfDHXjH6UmdDO24gQLg_3PpyNk_hJhXzxk8y6NSSpdMowxS-qmu9ZITaqZ-btXg7cL7UpJGp8IgT9UsG3Oi1pvG7ZDFV-fvsC_Hbfafihha7IY8wPDlen9hqBB7lCdr7dKIDAGj78bwG6YVfkP1GDsH6Lrx_w35tbcXghikMwEJpqhWDAcBYu0Yr4ES2tnBXYpYabdkEg&X-Amz-Signature=26dc4964add9268d1d47a54b944890ce89bb14a07b11e51334305e8b5fdc5479&_origin=apex-algorithms%2Fbenchmarks%2F%2Ftest_run_benchmark%5BLCFM_LSF_MONTHLY_S2_MEDIAN%5D` -> response 502 headers {'Date': 'Wed, 10 Dec 2025 19:36:06 GMT', 'Content-Length': '11'}
WARNING  openeo.rest._connection:_connection.py:174 Failed to parse API error response: [502] ''

Request that failed:
https://s3.stag.waw3-1.openeo.v1.dataspace.copernicus.eu:443

HEAD /openeo-data-staging-waw4-1/batch_jobs/j-2512101909374e0fb2599653c4c584a1/LCFM_LSF-MONTHLY_V008_2020_29TNE_2020-12-01Z_B07.tif_gdalinfo.json?X-Proxy-Head-As-Get=true&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=197503f65f0b4ac184a66a2a592eaf33%2F20251210%2Fwaw4-1%2Fs3%2Faws4_request&X-Amz-Date=20251210T191741Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Security-Token={token}&X-Amz-Signature={signature}&_origin=apex-algorithms%2Fbenchmarks%2F%2Ftest_run_benchmark%5BLCFM_LSF_MONTHLY_S2_MEDIAN%5D

[soxofaan]

This is the only place where client does a HEAD request (to detect ranged download):

openeo-python-client/openeo/rest/job.py

Lines 456 to 458 in de6f91b

	def _download_to_file(self, url: str, target: Path, *, chunk_size: int=DEFAULT_DOWNLOAD_CHUNK_SIZE, range_size: int=DEFAULT_DOWNLOAD_RANGE_SIZE):
	head = self.job.connection.head(url, stream=True)
	if head.ok and head.headers.get("Accept-Ranges") == "bytes" and 'Content-Length' in head.headers:

Note that this doesn't necessarily has to be fixed in the client itself, as a user (e.g. the apex tests) you can already set/override retry settings when creating the connection:

connection = openeo.connect(..., retry=...)

see https://open-eo.github.io/openeo-python-client/api.html#openeo.connect for more info

[jdries]

@soxofaan the default here is:
(default) to use default openEO-oriented retry settings
do you think the request was actually retried, or is the default for this particular head request to only try once?

[soxofaan]

No I think there was no retry. Current default does not include retry on 502.

background story: #770 initially had 502 (and some other 5xx) included as default, but it turned out that's not ideal as that conflicts with the poll loop of the mutlibackend job manager, where you want skip a failure and do other work in between before retrying. By adding too much 5xx codes to the default retry list of the generic connection session, you could introduce blocking sleeps that would ruin the responsiveness/throughput of the job manager. Also note that overriding defaults in the context of the job manager is also not trivial because the user (not the job manager) is in charge of creating the connection object.
Also see #770 (comment) and #770 (comment)

Long story short: we have to be careful with setting client defaults too aggressively because improving one use case might ruin another.

For this particular benchmark suite use case, I think it makes sense anyway to explicitly set retry settings that are finetuned to a benchmark context, instead of hoping generic defaults are good enough.

[soxofaan]

Added 502, 503 and 504 to the default retry list with 084097b