Convert app import/export to be fully streamed end to end. #16653
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Replace tar package (6.2.1) with tar-stream (3.1.7) for memory-efficient processing - Upgrade tar-fs from 2.1.2 to 2.1.3 to fix security vulnerability - Implement streaming tar creation/extraction to prevent memory buffering - Update backup exports, imports, and CLI to use streaming TAR operations - Resolves 100% CPU usage and linear memory growth during backup operations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Implement streamExportAppDirect() that streams tar directly to HTTP response - Eliminates intermediate tar file creation reducing disk I/O and memory usage - Uses tar-stream → gzip → PassThrough pipeline for efficient streaming - Update backup controller to use direct streaming method - Fix TypeScript types: ReadStream → NodeJS.ReadableStream for broader compatibility 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
|
QA Wolf here! As you write new code it's important that your test coverage is keeping up. |
- Add streaming encryption transform using AES-256-CTR cipher - Replicate existing encryption logic but as stream transforms - Support both encrypted and unencrypted streaming exports - Stream files directly from object store to tar without temp files - Stream database export directly without intermediate files - Maintain full backward compatibility with existing encryption format - Eliminate all temporary file usage during export process This achieves the goal of fully streaming exports with no disk I/O for temp files while maintaining encryption compatibility. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
samwho
changed the title
mike12345567
reviewed
Aug 7, 2025
| } | ||
|
|
||
| try { | ||
| await objectStore.streamUpload({ |
There was a problem hiding this comment.
Previously the imports were handled in parallel with the promise.push - realistically that could kick off too much anyway, but should we do something similar to the export and have some level of parallelization to avoid large imports being very slow?
mike12345567
approved these changes
Aug 7, 2025
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We see very erratic memory and CPU usage in production and I spent time combing through the data and found that there is a correlation between the higher spikes and requests to export or import an app. Looking at the code, I think it's possible for the exports to be done entirely in memory.
This PR rewrites the import/export subsystems to be streamed end-to-end, in the hope that this lessens the memory spikes. I suspect CPU spikes may persist because compression is inherently CPU itensive. That said, if we stream direct to the response that means the stream will be constrained on the receiving bandwidth, so the CPU could be less.
Feature branch env
Feature Branch Link