ci.jenkins.io jobs on Windows agents are much slower than 21 days ago #3904
Description
Service(s)
ci.jenkins.io
Summary
Jobs run on ci.jenkins.io Windows agents are significantly slower today than they were 21 days ago
Reproduction steps
Compare job execution times for the Windows portions of several plugins that I maintain. Dec 23 compared to today shows a large enough increase to cause most of the plugin BOM upgrade pull requests to those repositories to fail with a timeout error. The jobs that failed with a timeout error include:
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/apache-httpcomponents-client-4-api-plugin#197
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/authorize-project-plugin#210
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/basic-branch-build-strategies-plugin#61
- Bump nl.jqno.equalsverifier:equalsverifier from 3.15.5 to 3.15.6 jenkinsci/elastic-axis-plugin#343
- Bump nl.jqno.equalsverifier:equalsverifier from 3.15.5 to 3.15.6 jenkinsci/git-client-plugin#1093
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/git-plugin#1547
- Bump nl.jqno.equalsverifier:equalsverifier from 3.15.5 to 3.15.6 jenkinsci/git-plugin#1546
- Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.2.3 to 3.2.5 jenkinsci/gitlab-plugin#1610
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/gitlab-plugin#1611
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/implied-labels-plugin#180
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/markdown-formatter-plugin#130
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/nodelabelparameter-plugin#289
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/platformlabeler-plugin#1249
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/priority-sorter-plugin#328
- chore(deps): bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/schedule-build-plugin#295
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/testng-plugin-plugin#270
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/versioncolumn-plugin#212
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/xshell-plugin#140
There were a few pull requests at that same time that passed, but their build was dramatically longer than the typical build for their repository. The passing builds pull requests included:
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/elastic-axis-plugin#342 - 31 minutes when Dec 23 and Dec 20 builds took less than 7 minutes
- Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/git-client-plugin#1094 - 62 minutes when prior builds range from 21 minutes to 44 minutes
Usually the update of the plugin bill of materials in those plugins is simple and fast. Current agents seem to be running much slower than agents were running previously.