Skip to content

fix(nemesis): changed the order of repair and remove node invocation #12044

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

fix(nemesis): changed the order of repair and remove node invocation #12044

wants to merge 1 commit into from

Conversation

timtimb0t
Copy link
Contributor

@timtimb0t timtimb0t commented Sep 24, 2025 •
edited
Loading

Remove the dead node first and only then run repair. This avoids "Error: create repair target... "
fixes:#11603

Testing

https://jenkins.scylladb.com/job/scylla-staging/job/eugene_test_folder/job/repair_before_stop/2/

PR pre-checks (self review)

  • I added the relevant backport labels
  • I didn't leave commented-out/debugging code

Reminders

  • Add New configuration option and document them (in sdcm/sct_config.py)
  • Add unit tests to cover my changes (under unit-test/ folder)
  • Update the Readme/doc folder relevant to this change (if needed)

@timtimb0t timtimb0t added the backport/none Backport is not required label Sep 24, 2025
@timtimb0t timtimb0t marked this pull request as ready for review September 24, 2025 12:57
@timtimb0t
Copy link
Contributor Author

More refs: scylladb/scylla-manager#4529

@aleksbykov
Copy link
Contributor

IMHO, it should fixed on scylla manager side, because scylla remove node procedure require:
Run a full cluster repair before nodetool removenode, so all existing replicas have the most up-to-date data.

See doc: https://docs.scylladb.com/manual/stable/operating-scylla/procedures/cluster-management/remove-node.html#removing-an-unavailable-node

@timtimb0t
Copy link
Contributor Author

timtimb0t commented Sep 25, 2025 •
edited
Loading

IMHO, it should fixed on scylla manager side, because scylla remove node procedure require: Run a full cluster repair before nodetool removenode, so all existing replicas have the most up-to-date data.

See doc: https://docs.scylladb.com/manual/stable/operating-scylla/procedures/cluster-management/remove-node.html#removing-an-unavailable-node

Here is request for documentation change - scylladb/scylla-manager#4529 (comment)

And - scylladb/scylladb#25673 (comment)

@scylladbbot
Copy link

@timtimb0t new branch branch-2025.4 was added, please add backport label if needed

@timtimb0t timtimb0t added backport/2025.3 and removed backport/none Backport is not required labels Oct 1, 2025
aleksbykov
aleksbykov reviewed Oct 22, 2025
View reviewed changes
@timtimb0t
fix(nemesis): changed the order of repair and remove node invocation
4c8dcc4 
Remove the dead node first and only then run repair. This avoids
"Error: create repair target... "
fixes:scylladb#11603
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aleksbykov aleksbykov aleksbykov left review comments

@fruch fruch Awaiting requested review from fruch

@vponomaryov vponomaryov Awaiting requested review from vponomaryov

At least 2 approving reviews are required to merge this pull request.

Assignees

@timtimb0t timtimb0t

Labels

backport/2025.3 backport/2025.4

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@timtimb0t @aleksbykov @scylladbbot