Skip to content

Conversation

rpoyner-tri
Copy link
Contributor

@rpoyner-tri rpoyner-tri commented Apr 30, 2025

Closes: #21836.


This change is Reviewable

Copy link
Contributor Author

@rpoyner-tri rpoyner-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tons of builds will fail without revised provisioning. The goal at this point is for the correct set of builds to fail.

Reviewable status: needs platform reviewer assigned, needs at least two assigned reviewers, labeled "do not merge", missing label for release notes (waiting on @rpoyner-tri)

@rpoyner-tri
Copy link
Contributor Author

@drake-jenkins-bot linux-jammy-unprovisioned-clang-bazel-experimental-debug please.

@rpoyner-tri
Copy link
Contributor Author

Previously, rpoyner-tri (Rick Poyner (rico)) wrote…

@drake-jenkins-bot linux-jammy-unprovisioned-clang-bazel-experimental-debug please.

@drake-jenkins-bot linux-jammy-unprovisioned-clang-bazel-experimental-debug please.

@rpoyner-tri
Copy link
Contributor Author

Previously, rpoyner-tri (Rick Poyner (rico)) wrote…

@drake-jenkins-bot linux-jammy-unprovisioned-clang-bazel-experimental-debug please.

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.

Copy link
Contributor Author

@rpoyner-tri rpoyner-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are now at an interesting spot. Pretty much the correct sets of builds pass and fail, and for the right reasons (provisioned debugs fail for lack of requested mold linker). The outlier is unprovisioned debug on jammy; the old version of mold in Jammy is not good enough to pass the install test. I will claim this maybe doesn't matter, and perhaps the plan should be to disable that test when config is (Jammy AND -compilation-mode=dbg).

The version of mold in noble is (recently?) considerably newer, and handles install test just fine.

Reviewable status: 1 unresolved discussion, needs platform reviewer assigned, needs at least two assigned reviewers, labeled "do not merge", missing label for release notes (waiting on @rpoyner-tri)

@rpoyner-tri
Copy link
Contributor Author

Previously, rpoyner-tri (Rick Poyner (rico)) wrote…

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.

@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

@rpoyner-tri rpoyner-tri force-pushed the mold-integration branch 3 times, most recently from c688b02 to b8d155a Compare May 28, 2025 23:11
@rpoyner-tri
Copy link
Contributor Author

Previously, rpoyner-tri (Rick Poyner (rico)) wrote…

@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.
@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

@rpoyner-tri rpoyner-tri force-pushed the mold-integration branch 2 times, most recently from dd016ff to 999792f Compare May 29, 2025 19:10
@rpoyner-tri
Copy link
Contributor Author

Previously, rpoyner-tri (Rick Poyner (rico)) wrote…

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.
@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.
@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

@rpoyner-tri
Copy link
Contributor Author

Previously, rpoyner-tri (Rick Poyner (rico)) wrote…

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.
@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.
@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

1 similar comment
@rpoyner-tri
Copy link
Contributor Author

Previously, rpoyner-tri (Rick Poyner (rico)) wrote…

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.
@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.
@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

@rpoyner-tri
Copy link
Contributor Author

Previously, rpoyner-tri (Rick Poyner (rico)) wrote…

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.
@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.
@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

Copy link
Contributor Author

@rpoyner-tri rpoyner-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r2 disabled some tests for jammy and dbg. Now the only build failing that should not is
https://drake-jenkins.csail.mit.edu/job/linux-jammy-unprovisioned-gcc-bazel-experimental-debug/27/
which is reported "no space left on device".

Reviewable status: 1 unresolved discussion, needs platform reviewer assigned, needs at least two assigned reviewers, labeled "do not merge", missing label for release notes (waiting on @rpoyner-tri)

Copy link
Contributor Author

@rpoyner-tri rpoyner-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest (still r2 according to reviewable) abandons jammy mold.

Reviewable status: 1 unresolved discussion, needs platform reviewer assigned, needs at least two assigned reviewers, labeled "do not merge", missing label for release notes (waiting on @rpoyner-tri)

Copy link
Contributor Author

@rpoyner-tri rpoyner-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-(status: do not review) +@jwnimmer-tri for a first read.
Still not-merge, since this will need Noble CI provisioning updates.

Reviewable status: 1 unresolved discussion, LGTM missing from assignee jwnimmer-tri(platform), needs at least two assigned reviewers, labeled "do not merge", missing label for release notes (waiting on @rpoyner-tri)

@rpoyner-tri
Copy link
Contributor Author

Previously, rpoyner-tri (Rick Poyner (rico)) wrote…

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.
@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

@drake-jenkins-bot linux-noble-unprovisioned-clang-bazel-experimental-debug please.
@drake-jenkins-bot linux-jammy-unprovisioned-gcc-bazel-experimental-debug please.

Copy link
Collaborator

@jwnimmer-tri jwnimmer-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 4 files at r2.
Reviewable status: 2 unresolved discussions, LGTM missing from assignee jwnimmer-tri(platform), needs at least two assigned reviewers, labeled "do not merge", missing label for release notes (waiting on @rpoyner-tri)


setup/ubuntu/source_distribution/packages-noble-test-only.txt line 3 at r2 (raw file):

curl
libstdc++6-10-dbg
mold

Users who build Drake from source via CMake still end up using -c dbg because we plumb the CMake "Debug" option through to -c dbg in our CMake-Bazel wrapper. Thus, those users will now get an error message about not having mold, since they don't enable "test only" dependencies.

Do we want downstream project consuming Drake as an external to use mold, or should it only be used for Drake Developers? That will decide how we go about re-aligning things.

Copy link
Contributor Author

@rpoyner-tri rpoyner-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 2 unresolved discussions, LGTM missing from assignee jwnimmer-tri(platform), needs at least two assigned reviewers, labeled "do not merge", missing label for release notes (waiting on @rpoyner-tri)


setup/ubuntu/source_distribution/packages-noble-test-only.txt line 3 at r2 (raw file):

Previously, jwnimmer-tri (Jeremy Nimmer) wrote…

Users who build Drake from source via CMake still end up using -c dbg because we plumb the CMake "Debug" option through to -c dbg in our CMake-Bazel wrapper. Thus, those users will now get an error message about not having mold, since they don't enable "test only" dependencies.

Do we want downstream project consuming Drake as an external to use mold, or should it only be used for Drake Developers? That will decide how we go about re-aligning things.

It's a good question. Right now we don't have an alternate story for getting gdb line numbers to work, but maybe we don't care about that w.r.t. downstream projects.

Copy link
Collaborator

@jwnimmer-tri jwnimmer-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 2 unresolved discussions, LGTM missing from assignee jwnimmer-tri(platform), needs at least two assigned reviewers, labeled "do not merge", missing label for release notes (waiting on @rpoyner-tri)


setup/ubuntu/source_distribution/packages-noble-test-only.txt line 3 at r2 (raw file):

Previously, rpoyner-tri (Rick Poyner (rico)) wrote…

It's a good question. Right now we don't have an alternate story for getting gdb line numbers to work, but maybe we don't care about that w.r.t. downstream projects.

Having a broken build is worse that missing line numbers. Since it's probably somewhat difficult to engineer this to work correctly when Drake is used as a CMake external, at the very least mold should be Bazel-only, and probably even Developer-only.

@jwnimmer-tri
Copy link
Collaborator

@drake-jenkins-bot linux-noble-clang-bazel-experimental-everything-release please

@jwnimmer-tri
Copy link
Collaborator

@drake-jenkins-bot linux-noble-clang-bazel-experimental-address-sanitizer please
@drake-jenkins-bot linux-noble-clang-bazel-experimental-debug please
@drake-jenkins-bot linux-noble-clang-bazel-experimental-thread-sanitizer please
@drake-jenkins-bot linux-noble-clang-bazel-experimental-undefined-behavior-sanitizer please
@drake-jenkins-bot linux-noble-clang-bazel-experimental-valgrind-memcheck please

Copy link
Collaborator

@jwnimmer-tri jwnimmer-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dismissed @rpoyner-tri from a discussion.
Reviewable status: labeled "do not merge" (waiting on @rpoyner-tri)

@jwnimmer-tri
Copy link
Collaborator

@drake-jenkins-bot linux-noble-gcc-bazel-experimental-debug please

The install_test in GCC Debug failed, early on in the test (while still installing, not running any tests?). Let's try it again.


The valgrind failure was just a timeout while running the test, we can ignore that for now.

Copy link
Collaborator

@jwnimmer-tri jwnimmer-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 1 unresolved discussion, labeled "do not merge" (waiting on @rpoyner-tri)


a discussion (no related file):
Per f2f, the linux-noble-gcc-bazel-experimental-debug failure is probably real, and needs further attention.

Copy link
Contributor Author

@rpoyner-tri rpoyner-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all commit messages.
Reviewable status: 1 unresolved discussion, labeled "do not merge"

@rpoyner-tri rpoyner-tri force-pushed the mold-integration branch 3 times, most recently from 3d04b14 to 97b6aaa Compare July 25, 2025 19:26
Copy link
Contributor Author

@rpoyner-tri rpoyner-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all commit messages.
Reviewable status: 1 unresolved discussion, labeled "do not merge", commits need curation (https://drake.mit.edu/reviewable.html#curated-commits)

@rpoyner-tri rpoyner-tri force-pushed the mold-integration branch 2 times, most recently from d50cf1e to c143691 Compare July 29, 2025 17:30
Copy link
Contributor Author

@rpoyner-tri rpoyner-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-(status: do not merge)

Reviewable status: labeled "do not merge", commits need curation (https://drake.mit.edu/reviewable.html#curated-commits)

@rpoyner-tri
Copy link
Contributor Author

@drake-jenkins-bot linux-noble-gcc-bazel-experimental-debug please

Copy link
Contributor Author

@rpoyner-tri rpoyner-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 1 unresolved discussion, commits need curation (https://drake.mit.edu/reviewable.html#curated-commits)


a discussion (no related file):
Hand-tuning test timeouts sorta worked, but what I now suspect is going on is that mold and bazel are blindly spamming out threads and processes in an unhealthy way. Let's try thread-limiting mold instead.

Copy link
Contributor Author

@rpoyner-tri rpoyner-tri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! all discussions resolved, LGTM from assignees sammy-tri(platform),jwnimmer-tri(platform)


a discussion (no related file):

Previously, rpoyner-tri (Rick Poyner (rico)) wrote…

Hand-tuning test timeouts sorta worked, but what I now suspect is going on is that mold and bazel are blindly spamming out threads and processes in an unhealthy way. Let's try thread-limiting mold instead.

All good that time. If the mold debug build causes further trouble, we can dail the --thread-count=2 down to 1.

@rpoyner-tri rpoyner-tri merged commit 22cbc53 into RobotLocomotion:master Jul 29, 2025
10 of 11 checks passed
BetsyMcPhail added a commit that referenced this pull request Jul 30, 2025
sherm1 pushed a commit that referenced this pull request Jul 30, 2025
rpoyner-tri added a commit to rpoyner-tri/drake that referenced this pull request Aug 5, 2025
rpoyner-tri added a commit to rpoyner-tri/drake that referenced this pull request Aug 6, 2025
@jwnimmer-tri
Copy link
Collaborator

-(release notes: fix) +(release notes: none) because this was reverted.

@jwnimmer-tri jwnimmer-tri added release notes: none This pull request should not be mentioned in the release notes and removed release notes: fix This pull request contains fixes (no new features) labels Aug 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release notes: none This pull request should not be mentioned in the release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

rust dependencies spoil debug symbol loading in gdb
4 participants