Skip to content

Conversation

Wheest
Copy link
Contributor

@Wheest Wheest commented Jun 19, 2025

If we are running multiple instances of DVC in parallel in the same repo, commands can crash because they can't acquire a lock.

This is reasonable in most use-cases, but I've been using DVC in my bazel pipelines, which runs jobs in parallel. Therefore, I've added a --wait-for-lock flag that allows a process to be more patient.

Note this is my first PR, so I may not be following all the conventions of the project. Feedback appreciated!

Fixes #2520

@github-project-automation github-project-automation bot moved this to Backlog in DVC Jun 19, 2025
Copy link

codecov bot commented Jun 19, 2025

Codecov Report

Attention: Patch coverage is 89.04110% with 8 lines in your changes missing coverage. Please review.

Project coverage is 91.08%. Comparing base (2431ec6) to head (711e5ee).
Report is 72 commits behind head on main.

Files with missing lines Patch % Lines
dvc/lock.py 78.57% 6 Missing ⚠️
tests/func/test_lock.py 95.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10784      +/-   ##
==========================================
+ Coverage   90.68%   91.08%   +0.40%     
==========================================
  Files         504      504              
  Lines       39795    40227     +432     
  Branches     3141     3179      +38     
==========================================
+ Hits        36087    36640     +553     
+ Misses       3042     2958      -84     
+ Partials      666      629      -37     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@skshetry skshetry moved this from Backlog to Review In Progress in DVC Jun 19, 2025
@skshetry skshetry self-requested a review June 20, 2025 02:00
Wheest and others added 3 commits June 23, 2025 11:47
- unify lock in dvc/lock.py
- improve dvc/cli/command.py
- timeout takes precedence in HardlinkLock
@Wheest Wheest requested a review from skshetry June 26, 2025 09:59
@skshetry skshetry merged commit f3836ab into iterative:main Jun 26, 2025
39 checks passed
@github-project-automation github-project-automation bot moved this from Review In Progress to Done in DVC Jun 26, 2025
@skshetry
Copy link
Collaborator

skshetry commented Jun 26, 2025

Thank you for contributing.

Just so you are aware, there are two kinds of lock in dvc: 1) .dvc/lock that prevents another dvc process from running that could change repository, and 2) there's rwlock that is a more specific lock for specific dependencies and outputs (see Internal Directories and Files.

Although this adds wait for the first kind of lock, you may still run into rwlock if you try to change the same dependencies/outputs from different DVC processes.

@skshetry skshetry mentioned this pull request Jun 26, 2025
@skshetry skshetry added the feature is a feature label Jun 26, 2025
@Wheest
Copy link
Contributor Author

Wheest commented Jun 26, 2025

Many thanks, cheers! I'll need to see if this influences my use-cases when running bazel

@Wheest Wheest deleted the pez/wait4lock branch June 30, 2025 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature is a feature
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

lock: improve error message when .dvc/lock is taken
2 participants