Improve droplet upload draining #245
Open
+240
−147
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduce a global `draining` flag in `main.go` (atomic int32) to signal shutdown mode. - In `main.go` signal handler: • Set `draining = 1` on SIGTERM so new uploads are refused. • Invoke Ifrit group shutdown, then call `Shutdown()` on both HTTP servers with a 300s timeout. • Wait on `uploadWaitGroup` to let all in-flight uploads and polls complete before exit. - In `upload_droplet.ServeHTTP`: • Early-return HTTP 503 if `draining==1` (`rejecting-request-draining` log). • Retain existing logging and WaitGroup tracking for in-flight requests. - Enhance per-request flow to use `context.WithTimeout(r.Context(), timeout)`: • Automatically abort upload and polling if the client disconnects or the timeout elapses. • Pass `ctx.Done()` into both upload and poll routines. This ensures that ongoing droplet uploads finish cleanly while preventing new requests during shutdown.
5 tasks
2 tasks
kathap
force-pushed
the
improve-droplet-upload-draining
branch
from
2d87d5d to
f87cac1
Compare
kathap
force-pushed
the
improve-droplet-upload-draining
branch
from
07c903e to
b5eddc1
Compare
jochenehret
reviewed
Aug 8, 2025
kathap
force-pushed
the
improve-droplet-upload-draining
branch
from
fceb7cf to
a5b111a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In
main.gosignal handler:• Wait up to the timeout for in‑flight uploads to complete by racing a uploadWaitGroup.Wait() against timeout.
• Finally, signal the Ifrit process group (via monitor.Signal(os.Interrupt)) to hard‑stop any remaining runners and exit.
In
upload_droplet.ServeHTTP:• Retain existing logging and WaitGroup tracking for in-flight requests.
• replace deprecated CloseNotifier with context-based cancellation:
• Enhance per-request flow, introduce
context.WithTimeout(r.Context(), timeout)to produce a singlectx.Done()channel that:• Cancels when the client disconnects (
r.Context()).• Cancels when the configured timeout elapses.
This ensures that ongoing droplet uploads finish cleanly while preventing new requests during shutdown.
Related PRs:
cloudfoundry/capi-release#562
Related issue:
cloudfoundry/cloud_controller_ng#4351