Increase timeout in MixedClusterClientYamlTestSuiteIT #100585
Conversation
This suite now has a couple of thousand tests, some of which take a couple of seconds, so it times out occasionally. Relaxing the timeout further.
DaveCTurner
added
>test
|
Pinging @elastic/es-delivery (Team:Delivery) |
|
This is nuts. Here is a "normal" run where this takes less than 15 minutes: https://gradle-enterprise.elastic.co/s/kyht6stktieh6/tests/overview?class=org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT @brianseeders and I have been talking about this and our theory is that these BWC tests are very sensitive to resource contention on the build agent because many (most) of them spin up multi-cluster tests. In this case, it's a 4 node cluster and depending on what else is running on the machine it can cause the test execution time to explode. Out intended solution there is to strip these tests out and run them more isolated (or not at all in our platform matrix). I'm good with bumping the timeout for now though just to help with these failures. |
|
Yeah these tests alone run 4 nodes (each apparently believing it has 32 CPUs to play with) and Gradle counts this as a single job and runs lots of other multi-node tests at the same time too: 
It'd be nice to point a Universal Profiler at the test workers at some point to see what's really going on, but I would expect running these things all in parallel is probably not optimal. |
DaveCTurner
deleted the
2023/10/10/longer-MixedClusterClientYamlTestSuiteIT
branch
Should we change this behavior by setting |
💚 All backports created successfully
Questions ?Please refer to the Backport tool documentation |
This suite now has a couple of thousand tests, some of which take a couple of seconds, so it times out occasionally. Relaxing the timeout further. (cherry picked from commit 63b4ee1)
) This suite now has a couple of thousand tests, some of which take a couple of seconds, so it times out occasionally. Relaxing the timeout further. (cherry picked from commit 63b4ee1) Co-authored-by: David Turner <[email protected]>
DaveCTurner
restored the
2023/10/10/longer-MixedClusterClientYamlTestSuiteIT
branch
4 participants
This suite now has a couple of thousand tests, some of which take a
couple of seconds, so it times out occasionally. Relaxing the timeout
further.