Skip to content

More resilient droplet upload #195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Apr 15, 2025
Merged

More resilient droplet upload #195

merged 7 commits into from
Apr 15, 2025

Conversation

kathap
Copy link
Contributor

@kathap kathap commented Mar 21, 2025

Previously, the cc-uploader process did not have a draining mechanism. When the process was stopped, any ongoing droplet uploads were aborted immediately, leading to potential failures and incomplete uploads.

This PR introduces a graceful shutdown mechanism to allow ongoing uploads to complete before termination. The key enhancements include:

  • Handling shutdown signals:
    • The process now listens for termination signals (SIGINT, SIGTERM) and responds by allowing uploads to finish before exiting.
  • Tracking active uploads:
    • A global WaitGroup (uploadWaitGroup) is introduced to track ongoing uploads, incremented when a new droplet upload begins and decremented when it completes.
  • Graceful waiting before shutdown:
    Upon receiving a shutdown signal, cc-uploader will:
    1 Log the shutdown event.
    2 Wait for all active uploads to complete using the uploadWaitGroup.
    3 Gracefully shut down the HTTP servers (TLS and non-TLS).
  • Graceful server shutdown:
    Uses http.Server.Shutdown with a timeout to gracefully close both TLS and non-TLS servers.

Code Changes

  1. Signal Handling:
    • Introduced a signalChan to capture termination signals.
  2. WaitGroup for Uploads:
    • uploadWaitGroup ensures cc-uploader waits for all uploads to finish before stopping.
  3. Graceful Shutdown Logic:
    • The shutdown process now waits for active uploads to complete and cleanly shuts down the HTTP servers.

@kathap kathap marked this pull request as draft March 21, 2025 15:18
@kathap kathap marked this pull request as ready for review April 7, 2025 08:07
@jochenehret jochenehret self-requested a review April 10, 2025 08:47
@kathap kathap merged commit bd0f632 into main Apr 15, 2025
2 checks passed
ari-wg-gitbot added a commit to cloudfoundry/capi-release that referenced this pull request Apr 15, 2025
Changes in code.cloudfoundry.org/cc-uploader:

- More resilient droplet upload
    PR: cloudfoundry/cc-uploader#195
    Author: Katharina Przybill <[email protected]>
@moleske moleske deleted the add-draining-on-shutdown branch April 15, 2025 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants