Soft delete filled gaps #228751
Merged
Soft delete filled gaps #228751
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
denar50
added
release_note:skip
denar50
force-pushed
the
soft-delete-filled-gaps
branch
from
f96fb0c to
56c0f1f
Compare
|
/ci |
💚 Build Succeeded
Metrics [docs]
cc @denar50 |
|
Pinging @elastic/security-detection-engine (Team:Detection Engine) |
|
Starting backport for target branches: 8.19, 9.1 https://github.com/elastic/kibana/actions/runs/16420173132 |
kibanamachine
added a commit
to kibanamachine/kibana
that referenced
this pull request
Jul 21, 2025
## Summary This is a bug fix. Today the `softDeleteGaps` function is calling `processAllRuleGaps` without specifying the statuses of the gaps that should be fetched, therefore `processAllRuleGaps` falls back to fetching `unfilled` and `partially_filled`, but not `filled` gaps. As it is, if there are filled gaps, they will linger in the event log until the events expire. ## How to test Start with a clean installation of Kibana without any rules. Generate 100 rules, each with 100 gaps using [this tool](https://github.com/elastic/security-documents-generator). ``` yarn start rules --rules 100 -g 100 -c -i"5m" ``` Then do a manual run on some of the rules so that it covers some of the gaps. Wait until some gaps are filled. You can navigate to the dashboard at `/app/dashboards#/view/security-detection-rule-monitoring-default` and see if there are filled gaps. Then delete all the rules. The dashboard should not show any gaps. --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit f8b724c)
This was referenced Jul 21, 2025
kibanamachine
added a commit
to kibanamachine/kibana
that referenced
this pull request
Jul 21, 2025
## Summary This is a bug fix. Today the `softDeleteGaps` function is calling `processAllRuleGaps` without specifying the statuses of the gaps that should be fetched, therefore `processAllRuleGaps` falls back to fetching `unfilled` and `partially_filled`, but not `filled` gaps. As it is, if there are filled gaps, they will linger in the event log until the events expire. ## How to test Start with a clean installation of Kibana without any rules. Generate 100 rules, each with 100 gaps using [this tool](https://github.com/elastic/security-documents-generator). ``` yarn start rules --rules 100 -g 100 -c -i"5m" ``` Then do a manual run on some of the rules so that it covers some of the gaps. Wait until some gaps are filled. You can navigate to the dashboard at `/app/dashboards#/view/security-detection-rule-monitoring-default` and see if there are filled gaps. Then delete all the rules. The dashboard should not show any gaps. --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit f8b724c)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Jul 21, 2025
# Backport This will backport the following commits from `main` to `9.1`: - [Soft delete filled gaps (#228751)](#228751) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Edgar Santos","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-07-21T14:44:21Z","message":"Soft delete filled gaps (#228751)\n\n## Summary\nThis is a bug fix. Today the `softDeleteGaps` function is calling\n`processAllRuleGaps` without specifying the statuses of the gaps that\nshould be fetched, therefore `processAllRuleGaps` falls back to fetching\n`unfilled` and `partially_filled`, but not `filled` gaps. As it is, if\nthere are filled gaps, they will linger in the event log until the\nevents expire.\n\n## How to test\nStart with a clean installation of Kibana without any rules.\n\nGenerate 100 rules, each with 100 gaps using [this\ntool](https://github.com/elastic/security-documents-generator).\n```\nyarn start rules --rules 100 -g 100 -c -i\"5m\" \n```\n\nThen do a manual run on some of the rules so that it covers some of the\ngaps.\n\nWait until some gaps are filled. You can navigate to the dashboard at\n`/app/dashboards#/view/security-detection-rule-monitoring-default` and\nsee if there are filled gaps.\n\nThen delete all the rules.\n\nThe dashboard should not show any gaps.\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>","sha":"f8b724c060f4122d217df19de10a7e349c402a2c","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Detection Engine","backport:version","v9.1.0","v8.19.0","v9.2.0"],"title":"Soft delete filled gaps","number":228751,"url":"https://github.com/elastic/kibana/pull/228751","mergeCommit":{"message":"Soft delete filled gaps (#228751)\n\n## Summary\nThis is a bug fix. Today the `softDeleteGaps` function is calling\n`processAllRuleGaps` without specifying the statuses of the gaps that\nshould be fetched, therefore `processAllRuleGaps` falls back to fetching\n`unfilled` and `partially_filled`, but not `filled` gaps. As it is, if\nthere are filled gaps, they will linger in the event log until the\nevents expire.\n\n## How to test\nStart with a clean installation of Kibana without any rules.\n\nGenerate 100 rules, each with 100 gaps using [this\ntool](https://github.com/elastic/security-documents-generator).\n```\nyarn start rules --rules 100 -g 100 -c -i\"5m\" \n```\n\nThen do a manual run on some of the rules so that it covers some of the\ngaps.\n\nWait until some gaps are filled. You can navigate to the dashboard at\n`/app/dashboards#/view/security-detection-rule-monitoring-default` and\nsee if there are filled gaps.\n\nThen delete all the rules.\n\nThe dashboard should not show any gaps.\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>","sha":"f8b724c060f4122d217df19de10a7e349c402a2c"}},"sourceBranch":"main","suggestedTargetBranches":["9.1","8.19"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/228751","number":228751,"mergeCommit":{"message":"Soft delete filled gaps (#228751)\n\n## Summary\nThis is a bug fix. Today the `softDeleteGaps` function is calling\n`processAllRuleGaps` without specifying the statuses of the gaps that\nshould be fetched, therefore `processAllRuleGaps` falls back to fetching\n`unfilled` and `partially_filled`, but not `filled` gaps. As it is, if\nthere are filled gaps, they will linger in the event log until the\nevents expire.\n\n## How to test\nStart with a clean installation of Kibana without any rules.\n\nGenerate 100 rules, each with 100 gaps using [this\ntool](https://github.com/elastic/security-documents-generator).\n```\nyarn start rules --rules 100 -g 100 -c -i\"5m\" \n```\n\nThen do a manual run on some of the rules so that it covers some of the\ngaps.\n\nWait until some gaps are filled. You can navigate to the dashboard at\n`/app/dashboards#/view/security-detection-rule-monitoring-default` and\nsee if there are filled gaps.\n\nThen delete all the rules.\n\nThe dashboard should not show any gaps.\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>","sha":"f8b724c060f4122d217df19de10a7e349c402a2c"}}]}] BACKPORT--> Co-authored-by: Edgar Santos <[email protected]>
10 tasks
Bluefinger
pushed a commit
to Bluefinger/kibana
that referenced
this pull request
Jul 22, 2025
## Summary This is a bug fix. Today the `softDeleteGaps` function is calling `processAllRuleGaps` without specifying the statuses of the gaps that should be fetched, therefore `processAllRuleGaps` falls back to fetching `unfilled` and `partially_filled`, but not `filled` gaps. As it is, if there are filled gaps, they will linger in the event log until the events expire. ## How to test Start with a clean installation of Kibana without any rules. Generate 100 rules, each with 100 gaps using [this tool](https://github.com/elastic/security-documents-generator). ``` yarn start rules --rules 100 -g 100 -c -i"5m" ``` Then do a manual run on some of the rules so that it covers some of the gaps. Wait until some gaps are filled. You can navigate to the dashboard at `/app/dashboards#/view/security-detection-rule-monitoring-default` and see if there are filled gaps. Then delete all the rules. The dashboard should not show any gaps. --------- Co-authored-by: kibanamachine <[email protected]>
kibanamachine
added
the
backport missing
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
kibanamachine
added a commit
that referenced
this pull request
Jul 23, 2025
# Backport This will backport the following commits from `main` to `8.19`: - [Soft delete filled gaps (#228751)](#228751) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Edgar Santos","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-07-21T14:44:21Z","message":"Soft delete filled gaps (#228751)\n\n## Summary\nThis is a bug fix. Today the `softDeleteGaps` function is calling\n`processAllRuleGaps` without specifying the statuses of the gaps that\nshould be fetched, therefore `processAllRuleGaps` falls back to fetching\n`unfilled` and `partially_filled`, but not `filled` gaps. As it is, if\nthere are filled gaps, they will linger in the event log until the\nevents expire.\n\n## How to test\nStart with a clean installation of Kibana without any rules.\n\nGenerate 100 rules, each with 100 gaps using [this\ntool](https://github.com/elastic/security-documents-generator).\n```\nyarn start rules --rules 100 -g 100 -c -i\"5m\" \n```\n\nThen do a manual run on some of the rules so that it covers some of the\ngaps.\n\nWait until some gaps are filled. You can navigate to the dashboard at\n`/app/dashboards#/view/security-detection-rule-monitoring-default` and\nsee if there are filled gaps.\n\nThen delete all the rules.\n\nThe dashboard should not show any gaps.\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>","sha":"f8b724c060f4122d217df19de10a7e349c402a2c","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Detection Engine","backport:version","v9.1.0","v8.19.0","v9.2.0"],"title":"Soft delete filled gaps","number":228751,"url":"https://github.com/elastic/kibana/pull/228751","mergeCommit":{"message":"Soft delete filled gaps (#228751)\n\n## Summary\nThis is a bug fix. Today the `softDeleteGaps` function is calling\n`processAllRuleGaps` without specifying the statuses of the gaps that\nshould be fetched, therefore `processAllRuleGaps` falls back to fetching\n`unfilled` and `partially_filled`, but not `filled` gaps. As it is, if\nthere are filled gaps, they will linger in the event log until the\nevents expire.\n\n## How to test\nStart with a clean installation of Kibana without any rules.\n\nGenerate 100 rules, each with 100 gaps using [this\ntool](https://github.com/elastic/security-documents-generator).\n```\nyarn start rules --rules 100 -g 100 -c -i\"5m\" \n```\n\nThen do a manual run on some of the rules so that it covers some of the\ngaps.\n\nWait until some gaps are filled. You can navigate to the dashboard at\n`/app/dashboards#/view/security-detection-rule-monitoring-default` and\nsee if there are filled gaps.\n\nThen delete all the rules.\n\nThe dashboard should not show any gaps.\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>","sha":"f8b724c060f4122d217df19de10a7e349c402a2c"}},"sourceBranch":"main","suggestedTargetBranches":["9.1","8.19"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/228751","number":228751,"mergeCommit":{"message":"Soft delete filled gaps (#228751)\n\n## Summary\nThis is a bug fix. Today the `softDeleteGaps` function is calling\n`processAllRuleGaps` without specifying the statuses of the gaps that\nshould be fetched, therefore `processAllRuleGaps` falls back to fetching\n`unfilled` and `partially_filled`, but not `filled` gaps. As it is, if\nthere are filled gaps, they will linger in the event log until the\nevents expire.\n\n## How to test\nStart with a clean installation of Kibana without any rules.\n\nGenerate 100 rules, each with 100 gaps using [this\ntool](https://github.com/elastic/security-documents-generator).\n```\nyarn start rules --rules 100 -g 100 -c -i\"5m\" \n```\n\nThen do a manual run on some of the rules so that it covers some of the\ngaps.\n\nWait until some gaps are filled. You can navigate to the dashboard at\n`/app/dashboards#/view/security-detection-rule-monitoring-default` and\nsee if there are filled gaps.\n\nThen delete all the rules.\n\nThe dashboard should not show any gaps.\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>","sha":"f8b724c060f4122d217df19de10a7e349c402a2c"}}]}] BACKPORT--> Co-authored-by: Edgar Santos <[email protected]>
kibanamachine
removed
the
backport missing
kertal
pushed a commit
to kertal/kibana
that referenced
this pull request
Jul 25, 2025
## Summary This is a bug fix. Today the `softDeleteGaps` function is calling `processAllRuleGaps` without specifying the statuses of the gaps that should be fetched, therefore `processAllRuleGaps` falls back to fetching `unfilled` and `partially_filled`, but not `filled` gaps. As it is, if there are filled gaps, they will linger in the event log until the events expire. ## How to test Start with a clean installation of Kibana without any rules. Generate 100 rules, each with 100 gaps using [this tool](https://github.com/elastic/security-documents-generator). ``` yarn start rules --rules 100 -g 100 -c -i"5m" ``` Then do a manual run on some of the rules so that it covers some of the gaps. Wait until some gaps are filled. You can navigate to the dashboard at `/app/dashboards#/view/security-detection-rule-monitoring-default` and see if there are filled gaps. Then delete all the rules. The dashboard should not show any gaps. --------- Co-authored-by: kibanamachine <[email protected]>
crespocarlos
pushed a commit
to crespocarlos/kibana
that referenced
this pull request
Jul 25, 2025
## Summary This is a bug fix. Today the `softDeleteGaps` function is calling `processAllRuleGaps` without specifying the statuses of the gaps that should be fetched, therefore `processAllRuleGaps` falls back to fetching `unfilled` and `partially_filled`, but not `filled` gaps. As it is, if there are filled gaps, they will linger in the event log until the events expire. ## How to test Start with a clean installation of Kibana without any rules. Generate 100 rules, each with 100 gaps using [this tool](https://github.com/elastic/security-documents-generator). ``` yarn start rules --rules 100 -g 100 -c -i"5m" ``` Then do a manual run on some of the rules so that it covers some of the gaps. Wait until some gaps are filled. You can navigate to the dashboard at `/app/dashboards#/view/security-detection-rule-monitoring-default` and see if there are filled gaps. Then delete all the rules. The dashboard should not show any gaps. --------- Co-authored-by: kibanamachine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This is a bug fix. Today the
softDeleteGapsfunction is callingprocessAllRuleGapswithout specifying the statuses of the gaps that should be fetched, thereforeprocessAllRuleGapsfalls back to fetchingunfilledandpartially_filled, but notfilledgaps. As it is, if there are filled gaps, they will linger in the event log until the events expire.How to test
Start with a clean installation of Kibana without any rules.
Generate 100 rules, each with 100 gaps using this tool.
Then do a manual run on some of the rules so that it covers some of the gaps.
Wait until some gaps are filled. You can navigate to the dashboard at
/app/dashboards#/view/security-detection-rule-monitoring-defaultand see if there are filled gaps.Then delete all the rules.
The dashboard should not show any gaps.