ci/cpu: split | common & distributed #3188

Borda · 2025-07-25T12:05:33Z

What does this PR do?

This shall boost each iteration but preserve all test configurations

Before submitting

Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
Did you read the contributor guideline, Pull Request section?
Did you make sure to update the docs?
Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

📚 Documentation preview 📚: https://torchmetrics--3188.org.readthedocs.build/en/3188/

for more information, see https://pre-commit.ci

codecov · 2025-07-25T12:19:16Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69%. Comparing base (0f78932) to head (4f5aedc).

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #3188   +/-   ##
======================================
  Coverage      69%     69%           
======================================
  Files         364     364           
  Lines       20096   20095    -1     
======================================
+ Hits        13790   13816   +26     
+ Misses       6306    6279   -27

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Borda · 2025-07-25T17:54:44Z

@SkafteNicki, any idea why most of the DDP test, when we run them without the common test before, suddenly hangs...

SkafteNicki · 2025-07-28T08:59:56Z

@SkafteNicki, any idea why most of the DDP test, when we run them without the common test before, suddenly hangs...

@Borda this looks super weird to me. When I look into the failed tests, it is not even the same ones that are failing in same settings so it does not even seem to be systematic failure.

Borda · 2025-08-03T10:22:50Z

any idea why most of the DDP test, when we run them without the common test before, suddenly hangs...

this looks super weird to me. When I look into the failed tests, it is not even the same ones that are failing in same settings so it does not even seem to be systematic failure.

yeah, so do you agree that the change shall be completely fine

SkafteNicki · 2025-08-04T06:20:41Z

any idea why most of the DDP test, when we run them without the common test before, suddenly hangs...

this looks super weird to me. When I look into the failed tests, it is not even the same ones that are failing in same settings so it does not even seem to be systematic failure.

yeah, so do you agree that the change shall be completely fine

Yes, change should be fine. Right now I am thinking that it may be due to a port coalition, testing...

… into ci/split

for more information, see https://pre-commit.ci

Borda · 2025-08-06T20:43:02Z

so many cases are resolved now, but still seeing:

unittests/audio/test_nisqa.py::TestNISQA::test_nisqa[True-preds0-16000-reference0] +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~ Stack of Thread-4 (_handle_results) (140416607311424) ~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/pool.py", line 579, in _handle_results
    task = get()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
~~~~~~~~~~~~~ Stack of Thread-3 (_handle_tasks) (140416624092736) ~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/pool.py", line 531, in _handle_tasks
    for taskseq, set_length in iter(taskqueue.get, None):
~~~~~~~~~~~~ Stack of Thread-2 (_handle_workers) (140416640874048) ~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/pool.py", line 522, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/pool.py", line 502, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/selectors.py", line 416, in select
    fd_event_list = self._selector.poll(timeout)
~~~~~~~~~~~~~~~ Stack of Thread-1 (run_server) (140416729474624) ~~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/pytest_rerunfailures.py", line 438, in run_server
    conn, _ = self.sock.accept()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/socket.py", line 293, in accept
    fd, addr = self._accept()

Borda · 2025-08-11T17:35:00Z

by looking at the following, it seems it uses the same port for all?

FAILED unittests/text/test_rouge.py::TestROUGEScore::test_rouge_score_class[True-avg-preds0-targets0-rouge2_fmeasure-True] - RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [10.1.0.203]:21380
FAILED unittests/text/test_rouge.py::TestROUGEScore::test_rouge_score_class[True-avg-preds0-targets0-rougeL_precision-False] - RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [10.1.0.203]:21380
FAILED unittests/text/test_rouge.py::TestROUGEScore::test_rouge_score_class[True-avg-preds0-targets0-rougeL_recall-False] - RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [10.1.0.203]:21380

for more information, see https://pre-commit.ci

ci: split common & distributed

d7dc9b2

Borda requested review from justusschock and ethanwharris as code owners July 25, 2025 12:05

[pre-commit.ci] auto fixes from pre-commit.com hooks

bf756d3

for more information, see https://pre-commit.ci

Merge branch 'master' into ci/split

324a610

try random port instead

dfeee3d

SkafteNicki self-requested a review as a code owner August 4, 2025 06:19

SkafteNicki and others added 5 commits August 4, 2025 09:13

try instead a free socket approach

c17511d

Merge branch 'master' into ci/split

608310f

fix ports not syncing

e2af420

Merge branch 'ci/split' of https://github.com/Lightning-AI/torchmetrics…

060939b

… into ci/split

try limit number of parallel jobs for debugging

6c92ed9

mergify bot added the has conflicts label Aug 6, 2025

Borda and others added 2 commits August 6, 2025 18:06

Merge branch 'master' into ci/split

61b7ada

[pre-commit.ci] auto fixes from pre-commit.com hooks

86f33bd

for more information, see https://pre-commit.ci

mergify bot removed the has conflicts label Aug 6, 2025

Borda and others added 5 commits August 6, 2025 18:58

Merge branch 'master' into ci/split

ce90ca4

pytest

0e502a6

gpu

913cea6

Merge branch 'master' into ci/split

9f2d1cb

doctest

5637d3b

Merge branch 'master' into ci/split

030c6b6

Borda mentioned this pull request Aug 7, 2025

ci/gpu: split | doctest & common & distributed #3215

Open

4 tasks

mergify bot added the has conflicts label Aug 7, 2025

Borda changed the title ~~ci: split common & distributed~~ ci/cpu: split common & distributed Aug 7, 2025

Borda added 2 commits August 7, 2025 19:40

rev GPU

a0ad488

Merge branch 'master' into ci/split

40e7840

mergify bot removed the has conflicts label Aug 7, 2025

Borda and others added 13 commits August 11, 2025 19:39

get_free_port

d10a688

get_free_port

1c93193

get_free_port

6f9bec7

[pre-commit.ci] auto fixes from pre-commit.com hooks

5f81e75

for more information, see https://pre-commit.ci

get_free_port()

4b3c37e

settimeout

b5f9ae3

setup_ddp

666cb7f

timeout-minutes: 70

12253e6

enumerate

e73e1d3

get_free_port

b8eae2c

get_free_port

2021ca5

master port

f7fd132

no rerun

154a4c6

Borda changed the title ~~ci/cpu: split common & distributed~~ ci/cpu: split | common & distributed Aug 13, 2025

Merge branch 'master' into ci/split

4f5aedc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci/cpu: split | common & distributed #3188

ci/cpu: split | common & distributed #3188

Uh oh!

Borda commented Jul 25, 2025 •

edited by github-actions bot

Loading

Uh oh!

codecov bot commented Jul 25, 2025 •

edited

Loading

Uh oh!

Borda commented Jul 25, 2025

Uh oh!

SkafteNicki commented Jul 28, 2025

Uh oh!

Borda commented Aug 3, 2025

Uh oh!

SkafteNicki commented Aug 4, 2025

Uh oh!

Borda commented Aug 6, 2025

Uh oh!

Borda commented Aug 11, 2025

Uh oh!

Uh oh!

ci/cpu: split | common & distributed #3188

Are you sure you want to change the base?

ci/cpu: split | common & distributed #3188

Uh oh!

Conversation

Borda commented Jul 25, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Did you have fun?

Uh oh!

codecov bot commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Borda commented Jul 25, 2025

Uh oh!

SkafteNicki commented Jul 28, 2025

Uh oh!

Borda commented Aug 3, 2025

Uh oh!

SkafteNicki commented Aug 4, 2025

Uh oh!

Borda commented Aug 6, 2025

Uh oh!

Borda commented Aug 11, 2025

Uh oh!

Uh oh!

Borda commented Jul 25, 2025 •

edited by github-actions bot

Loading

codecov bot commented Jul 25, 2025 •

edited

Loading