Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Added concurrency limits and request throttling to prebuilt rule routes #209551

Merged
merged 1 commit into from
Feb 11, 2025
Merged

[Security Solution] Added concurrency limits and request throttling to prebuilt rule routes #209551

merged 1 commit into from
Feb 11, 2025
+574 −287

Conversation

xcrzx
Copy link
Contributor

@xcrzx xcrzx commented Feb 4, 2025 •
edited by kibanamachine
Loading

Resolves: #208357
Resolves: #208355

Summary

To prevent possible OOM errors, we need to limit concurrent requests to prebuilt rule routes (see attached tickets for more details).

  • installation/_perform and upgrade/_perform endpoints

    • Concurrency is limited to one parallel call. If another call is made simultaneously, the server responds with 429 Too Many Requests.
    • On the front end, all rule install and upgrade operations are retried in case of a 429 response. This ensures proper handling when a user clicks multiple times an update or install rule buttons

  • prebuilt_rules/_bootstrap endpoint

    • Install prebuilt rules and endpoint packages sequentially instead of in parallel to prevent from having them both downloaded into memory simultaneously.
    • Added a 30-minute socket timeout to prevent the proxy from closing the connection while rule installation is in progress.
    • Introduced a throttleRequests wrapper, ensuring the endpoint handler is called only once when multiple concurrent requests are received.
      • The first request triggers the handler, while subsequent requests wait for the first one to complete and reuse its result.
      • This prevents costly prebuilt rule package installation from running in parallel.
      • Reusing the response ensures the frontend correctly invalidates cached prebuilt rule queries. Since concurrent frontend requests should receive the same installed package information, responding with 421 and using the retry logic as in cases above is not an option here because the second request would receive a package installation skipped response leading to no cache invalidation.

  • installation/_review and upgrade/_review endpoints

    • Concurrency is limited to one parallel call. If another call is made simultaneously, the server responds with 429 Too Many Requests.
    • On the front end, all rule install and upgrade operations are retried in case of a 429 response. This ensures proper handling when a user clicks multiple times an update or install rule buttons

@xcrzx xcrzx self-assigned this Feb 4, 2025
@xcrzx xcrzx added release_note:fix Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team labels Feb 5, 2025
@xcrzx xcrzx marked this pull request as ready for review February 5, 2025 12:44
@xcrzx xcrzx requested review from a team as code owners February 5, 2025 12:44
@xcrzx xcrzx requested a review from maximpn February 5, 2025 12:44
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@xcrzx xcrzx added performance Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area labels Feb 5, 2025
@banderror

This comment was marked as resolved.

Sign in to view
@xcrzx xcrzx force-pushed the request-throttle branch 2 times, most recently from 87b0993 to a664983 Compare February 7, 2025 10:59
semd
semd approved these changes Feb 7, 2025
View reviewed changes
Copy link
Contributor

@semd semd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Threat hunting LGTM. just a NIT

maximpn
maximpn reviewed Feb 7, 2025
View reviewed changes
Copy link
Contributor

@maximpn maximpn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xcrzx,

local testing revealed a problem where React Query stops retrying mutations upon 429 HTTP response after 3 attempts. I set up rule upgrade (for example installed outdated package as described here) and tried to run bulk rule upgrade in two tabs in parallel. Upgrade succeeded in the first tab but was never completed in the second tab. React Query made 3 attempts and stopped.

Screen.Recording.2025-02-07.at.16.53.59.mov

@xcrzx xcrzx requested a review from maximpn February 10, 2025 18:10
@xcrzx xcrzx force-pushed the request-throttle branch 2 times, most recently from e21b161 to df1960b Compare February 11, 2025 10:03
@xcrzx
Copy link
Contributor Author

xcrzx commented Feb 11, 2025

local testing revealed a problem where React Query stops retrying mutations upon 429 HTTP response after 3 attempts. I set up rule upgrade (for example installed outdated package as described here) and tried to run bulk rule upgrade in two tabs in parallel. Upgrade succeeded in the first tab but was never completed in the second tab. React Query made 3 attempts and stopped.

@maximpn I tried multiple times but couldn’t reproduce the issue. Sometimes, due to exponential backoff, after multiple failed retries, you need to wait for some time (up to a minute or even more). Could that be the case?

This was referenced Feb 11, 2025
Open
Closed
@elasticmachine
Copy link
Contributor

elasticmachine commented Feb 11, 2025 •
edited
Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Jest Tests #1 / useReplaceCustomField calls the api when invoked with the correct parameters

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 6697 6699 +2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 21.4MB 21.4MB +1.7KB

History

cc @xcrzx

maximpn
maximpn approved these changes Feb 11, 2025
View reviewed changes
Copy link
Contributor

@maximpn maximpn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xcrzx Thanks for addressing comments and making improvements to increase robustness of the solution 👍

✅ I tested the PR locally in attempt to spot any regressions and didn't reveal issues. In particular I did the following

  • Run prebuilt rules bootstrap endpoint in parallel via API, via UI and both.
  • Run prebuilt rule's installation endpoint in parallel via API, via UI and both.
  • Run rules upgrade preview endpoint in parallel via API, via UI and both.
  • Run rules upgrade endpoint in parallel via API, via UI and both.

Direct parallel requests to API span up to 10 requests. UI testing used two web browser tabs.

Conclusion: Allowing execution of only request doesn't lead to regressions. Periodically rule upgrade page loads, rule upgrade review data and rule upgrade request are retried due to 429 HTTP error but it doesn't significantly affect users in average. Exponential backoff capped at 30 seconds sounds reasonable to avoid flocking endpoints. Suggesting users access rate limited endpoints in random time intervals only a limited number of users will have to wait minutes to load the data.

Caveats: There is a possibility to abuse rate limited API endpoints via automated direct requests. It will prevent any users from fetching any data since rate limiting works according to "first wins" scenario. In that case it will look like a DDoS attack.

@xcrzx xcrzx enabled auto-merge (squash) February 11, 2025 15:41
@xcrzx xcrzx merged commit c5557f3 into elastic:main Feb 11, 2025
9 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.17, 8.18, 8.x, 9.0

https://github.com/elastic/kibana/actions/runs/13268545459

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Feb 11, 2025
@xcrzx
[Security Solution] Added concurrency limits and request throttling t…
14d96cb 
…o prebuilt rule routes (elastic#209551)

**Resolves: elastic#208357
**Resolves: elastic#208355

## Summary

To prevent possible OOM errors, we need to limit concurrent requests to
prebuilt rule routes (see attached tickets for more details).

- `installation/_perform` and `upgrade/_perform` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

- `prebuilt_rules/_bootstrap` endpoint
- Install prebuilt rules and endpoint packages sequentially instead of
in parallel to prevent from having them both downloaded into memory
simultaneously.
- Added a 30-minute socket timeout to prevent the proxy from closing the
connection while rule installation is in progress.
- Introduced a `throttleRequests` wrapper, ensuring the endpoint handler
is called only once when multiple concurrent requests are received.
- The first request triggers the handler, while subsequent requests wait
for the first one to complete and reuse its result.
- This prevents costly prebuilt rule package installation from running
in parallel.
- Reusing the response ensures the frontend correctly invalidates cached
prebuilt rule queries. Since concurrent frontend requests should receive
the same installed package information, responding with 421 and using
the retry logic as in cases above is not an option here because the
second request would receive a package installation skipped response
leading to no cache invalidation.

- `installation/_review` and `upgrade/_review` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

(cherry picked from commit c5557f3)
@kibanamachine kibanamachine mentioned this pull request Feb 11, 2025
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Feb 11, 2025
@xcrzx
[Security Solution] Added concurrency limits and request throttling t…
096b0fe 
…o prebuilt rule routes (elastic#209551)

**Resolves: elastic#208357
**Resolves: elastic#208355

## Summary

To prevent possible OOM errors, we need to limit concurrent requests to
prebuilt rule routes (see attached tickets for more details).

- `installation/_perform` and `upgrade/_perform` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

- `prebuilt_rules/_bootstrap` endpoint
- Install prebuilt rules and endpoint packages sequentially instead of
in parallel to prevent from having them both downloaded into memory
simultaneously.
- Added a 30-minute socket timeout to prevent the proxy from closing the
connection while rule installation is in progress.
- Introduced a `throttleRequests` wrapper, ensuring the endpoint handler
is called only once when multiple concurrent requests are received.
- The first request triggers the handler, while subsequent requests wait
for the first one to complete and reuse its result.
- This prevents costly prebuilt rule package installation from running
in parallel.
- Reusing the response ensures the frontend correctly invalidates cached
prebuilt rule queries. Since concurrent frontend requests should receive
the same installed package information, responding with 421 and using
the retry logic as in cases above is not an option here because the
second request would receive a package installation skipped response
leading to no cache invalidation.

- `installation/_review` and `upgrade/_review` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

(cherry picked from commit c5557f3)
@kibanamachine kibanamachine mentioned this pull request Feb 11, 2025
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Feb 11, 2025
@xcrzx
[Security Solution] Added concurrency limits and request throttling t…
21e42b1 
…o prebuilt rule routes (elastic#209551)

**Resolves: elastic#208357
**Resolves: elastic#208355

## Summary

To prevent possible OOM errors, we need to limit concurrent requests to
prebuilt rule routes (see attached tickets for more details).

- `installation/_perform` and `upgrade/_perform` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

- `prebuilt_rules/_bootstrap` endpoint
- Install prebuilt rules and endpoint packages sequentially instead of
in parallel to prevent from having them both downloaded into memory
simultaneously.
- Added a 30-minute socket timeout to prevent the proxy from closing the
connection while rule installation is in progress.
- Introduced a `throttleRequests` wrapper, ensuring the endpoint handler
is called only once when multiple concurrent requests are received.
- The first request triggers the handler, while subsequent requests wait
for the first one to complete and reuse its result.
- This prevents costly prebuilt rule package installation from running
in parallel.
- Reusing the response ensures the frontend correctly invalidates cached
prebuilt rule queries. Since concurrent frontend requests should receive
the same installed package information, responding with 421 and using
the retry logic as in cases above is not an option here because the
second request would receive a package installation skipped response
leading to no cache invalidation.

- `installation/_review` and `upgrade/_review` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

(cherry picked from commit c5557f3)
@kibanamachine kibanamachine mentioned this pull request Feb 11, 2025
Merged
@kibanamachine
Copy link
Contributor

💔 Some backports could not be created

Status Branch Result
❌ 8.17 Backport failed because of merge conflicts
✅ 8.18
✅ 8.x
✅ 9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 209551

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Feb 11, 2025
@kibanamachine @xcrzx
[9.0] [Security Solution] Added concurrency limits and request thrott…
df87081 
…ling to prebuilt rule routes (#209551) (#210642)

# Backport

This will backport the following commits from `main` to `9.0`:
- [[Security Solution] Added concurrency limits and request throttling
to prebuilt rule routes
(#209551)](#209551)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Dmitrii
Shevchenko","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-02-11T17:12:03Z","message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","performance","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3"],"title":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule
routes","number":209551,"url":"https://github.com/elastic/kibana/pull/209551","mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x","8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209551","number":209551,"mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Dmitrii Shevchenko <[email protected]>
kibanamachine added a commit that referenced this pull request Feb 11, 2025
@kibanamachine @xcrzx
[8.18] [Security Solution] Added concurrency limits and request throt…
2bd85b1 
…tling to prebuilt rule routes (#209551) (#210640)

# Backport

This will backport the following commits from `main` to `8.18`:
- [[Security Solution] Added concurrency limits and request throttling
to prebuilt rule routes
(#209551)](#209551)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Dmitrii
Shevchenko","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-02-11T17:12:03Z","message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","performance","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3"],"title":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule
routes","number":209551,"url":"https://github.com/elastic/kibana/pull/209551","mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x","8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209551","number":209551,"mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Dmitrii Shevchenko <[email protected]>
kibanamachine added a commit that referenced this pull request Feb 11, 2025
@kibanamachine @xcrzx
[8.x] [Security Solution] Added concurrency limits and request thrott…
8018c82 
…ling to prebuilt rule routes (#209551) (#210641)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Added concurrency limits and request throttling
to prebuilt rule routes
(#209551)](#209551)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Dmitrii
Shevchenko","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-02-11T17:12:03Z","message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","performance","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3"],"title":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule
routes","number":209551,"url":"https://github.com/elastic/kibana/pull/209551","mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x","8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209551","number":209551,"mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Dmitrii Shevchenko <[email protected]>
kapral18 added a commit to agusruidiazgd/kibana that referenced this pull request Feb 11, 2025
@kapral18
Merge branch 'main' into agusruidiazgd__fix/styled-components-migrati…
dead195 
…on-206439

* main: (402 commits)
  [Search]: Fix Number type field to have correct property (elastic#210462)
  Change filter for rule monitoring gaps (elastic#209983)
  Update Logs Explorer deprecation messages (elastic#201307)
  [APM] Remove `error.id` in `getErrorGroupMainStatistics` query as it's not used (elastic#210613)
  [Embeddable] Fix presentation panel styles (elastic#210113)
  [ci] enable Scout reporter for on-merge-unsupported-ftrs (elastic#210627)
  [Fix][Synonyms UI]Add navigation link to the Detail breadcrumb. (elastic#209574)
  chore(dep): bump `store2` from `2.12.0` to `2.14.4` (elastic#210530)
  [scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (elastic#209761)
  [Cloud Security] Asset Inventory table flyout controls  (elastic#208452)
  [ML] Fix model deployment check in file uploader (elastic#209585)
  Updates archive again (elastic#209828)
  [Security Solution] Added concurrency limits and request throttling to prebuilt rule routes (elastic#209551)
  [Search] [Onboarding] Update search api to use EventEmitter instead of Provider (elastic#209784)
  [maps] lazy load map actions (elastic#210252)
  [Cloud Security] Adding telemetry collection condition based on render condition (elastic#208758)
  [Solution nav] Use flyout for Stack Management in Search and Observability solutions (elastic#208632)
  [Search] Fix Add Inference Endpoint API call (elastic#210243)
  [Agentless Connectors] Integration overview panel (elastic#210222)
  [Lens] Restore dynamic colouring by value for Last value agg (elastic#209110)
  ...
xcrzx added a commit to xcrzx/kibana that referenced this pull request Feb 12, 2025
@xcrzx
[Security Solution] Added concurrency limits and request throttling t…
a6c4954 
…o prebuilt rule routes (elastic#209551)

**Resolves: elastic#208357
**Resolves: elastic#208355

## Summary

To prevent possible OOM errors, we need to limit concurrent requests to
prebuilt rule routes (see attached tickets for more details).

- `installation/_perform` and `upgrade/_perform` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

- `prebuilt_rules/_bootstrap` endpoint
- Install prebuilt rules and endpoint packages sequentially instead of
in parallel to prevent from having them both downloaded into memory
simultaneously.
- Added a 30-minute socket timeout to prevent the proxy from closing the
connection while rule installation is in progress.
- Introduced a `throttleRequests` wrapper, ensuring the endpoint handler
is called only once when multiple concurrent requests are received.
- The first request triggers the handler, while subsequent requests wait
for the first one to complete and reuse its result.
- This prevents costly prebuilt rule package installation from running
in parallel.
- Reusing the response ensures the frontend correctly invalidates cached
prebuilt rule queries. Since concurrent frontend requests should receive
the same installed package information, responding with 421 and using
the retry logic as in cases above is not an option here because the
second request would receive a package installation skipped response
leading to no cache invalidation.

- `installation/_review` and `upgrade/_review` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

(cherry picked from commit c5557f3)

# Conflicts:
#	x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/capped_exponential_backoff.ts
#	x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/retry_on_rate_limited_error.ts
#	x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/use_bootstrap_prebuilt_rules.ts
#	x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/use_perform_specific_rules_upgrade_mutation.ts
#	x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/use_bootstrap_prebuilt_rules.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/bootstrap_prebuilt_rules/bootstrap_prebuilt_rules_handler.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/perform_rule_upgrade_route.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_installation/review_rule_installation_handler.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_installation/review_rule_installation_route.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_upgrade/review_rule_upgrade_handler.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_upgrade/review_rule_upgrade_route.ts
#	x-pack/plugins/security_solution/server/utils/throttle_requests.test.ts
#	x-pack/plugins/security_solution/server/utils/throttle_requests.ts
#	x-pack/solutions/security/plugins/security_solution/public/detection_engine/rule_management/api/hooks/use_bootstrap_prebuilt_rules.ts
#	x-pack/solutions/security/plugins/security_solution/public/detection_engine/rule_management_ui/components/rules_table/upgrade_prebuilt_rules_table/translations.tsx
#	x-pack/solutions/security/plugins/security_solution/public/detection_engine/rule_management_ui/components/rules_table/upgrade_prebuilt_rules_table/upgrade_prebuilt_rules_table_context.tsx
@xcrzx xcrzx mentioned this pull request Feb 12, 2025
Merged
@xcrzx
Copy link
Contributor Author

xcrzx commented Feb 12, 2025

💚 All backports created successfully

Status Branch Result
✅ 8.17

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@xcrzx xcrzx deleted the request-throttle branch February 12, 2025 11:26
xcrzx added a commit that referenced this pull request Feb 12, 2025
@xcrzx
[8.17] [Security Solution] Added concurrency limits and request throt…
6259401 
…tling to prebuilt rule routes (#209551) (#210773)

# Backport

This will backport the following commits from `main` to `8.17`:
- [[Security Solution] Added concurrency limits and request throttling
to prebuilt rule routes
(#209551)](#209551)

<!--- Backport version: 9.6.4 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Dmitrii
Shevchenko","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-02-11T17:12:03Z","message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","performance","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3"],"title":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule
routes","number":209551,"url":"https://github.com/elastic/kibana/pull/209551","mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},"sourceBranch":"main","suggestedTargetBranches":["8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210642","number":210642,"state":"MERGED","mergeCommit":{"sha":"df87081d8a574c6b0bb2d9fc026776502622dc11","message":"[9.0]
[Security Solution] Added concurrency limits and request throttling to
prebuilt rule routes (#209551) (#210642)\n\n# Backport\n\nThis will
backport the following commits from `main` to `9.0`:\n- [[Security
Solution] Added concurrency limits and request throttling\nto prebuilt
rule
routes\n(#209551)](https://github.com/elastic/kibana/pull/209551)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Dmitrii\nShevchenko\",\"email\":\"[email protected]\"},\"sourceCommit\":{\"committedDate\":\"2025-02-11T17:12:03Z\",\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:fix\",\"performance\",\"v9.0.0\",\"Team:Detections\nand
Resp\",\"Team: SecuritySolution\",\"Team:Detection
Rule\nManagement\",\"Feature:Prebuilt
Detection\nRules\",\"backport:version\",\"v8.18.0\",\"v9.1.0\",\"v8.19.0\",\"v8.17.3\"],\"title\":\"[Security\nSolution]
Added concurrency limits and request throttling to
prebuilt\nrule\nroutes\",\"number\":209551,\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.18\",\"8.x\",\"8.17\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.18\",\"label\":\"v8.18.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"number\":209551,\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.17\",\"label\":\"v8.17.3\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Dmitrii Shevchenko
<[email protected]>"}},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210640","number":210640,"state":"MERGED","mergeCommit":{"sha":"2bd85b19aa84575ec0745ba2ad24d4c7718d824f","message":"[8.18]
[Security Solution] Added concurrency limits and request throttling to
prebuilt rule routes (#209551) (#210640)\n\n# Backport\n\nThis will
backport the following commits from `main` to `8.18`:\n- [[Security
Solution] Added concurrency limits and request throttling\nto prebuilt
rule
routes\n(#209551)](https://github.com/elastic/kibana/pull/209551)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Dmitrii\nShevchenko\",\"email\":\"[email protected]\"},\"sourceCommit\":{\"committedDate\":\"2025-02-11T17:12:03Z\",\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:fix\",\"performance\",\"v9.0.0\",\"Team:Detections\nand
Resp\",\"Team: SecuritySolution\",\"Team:Detection
Rule\nManagement\",\"Feature:Prebuilt
Detection\nRules\",\"backport:version\",\"v8.18.0\",\"v9.1.0\",\"v8.19.0\",\"v8.17.3\"],\"title\":\"[Security\nSolution]
Added concurrency limits and request throttling to
prebuilt\nrule\nroutes\",\"number\":209551,\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.18\",\"8.x\",\"8.17\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.18\",\"label\":\"v8.18.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"number\":209551,\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.17\",\"label\":\"v8.17.3\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Dmitrii Shevchenko
<[email protected]>"}},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209551","number":209551,"mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210641","number":210641,"state":"MERGED","mergeCommit":{"sha":"8018c82f7d92b1d4798dea3c0a677bae740d4f8a","message":"[8.x]
[Security Solution] Added concurrency limits and request throttling to
prebuilt rule routes (#209551) (#210641)\n\n# Backport\n\nThis will
backport the following commits from `main` to `8.x`:\n- [[Security
Solution] Added concurrency limits and request throttling\nto prebuilt
rule
routes\n(#209551)](https://github.com/elastic/kibana/pull/209551)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Dmitrii\nShevchenko\",\"email\":\"[email protected]\"},\"sourceCommit\":{\"committedDate\":\"2025-02-11T17:12:03Z\",\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:fix\",\"performance\",\"v9.0.0\",\"Team:Detections\nand
Resp\",\"Team: SecuritySolution\",\"Team:Detection
Rule\nManagement\",\"Feature:Prebuilt
Detection\nRules\",\"backport:version\",\"v8.18.0\",\"v9.1.0\",\"v8.19.0\",\"v8.17.3\"],\"title\":\"[Security\nSolution]
Added concurrency limits and request throttling to
prebuilt\nrule\nroutes\",\"number\":209551,\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.18\",\"8.x\",\"8.17\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.18\",\"label\":\"v8.18.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"number\":209551,\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.17\",\"label\":\"v8.17.3\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Dmitrii Shevchenko
<[email protected]>"}},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
@xcrzx xcrzx mentioned this pull request Feb 24, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maximpn maximpn maximpn approved these changes

@semd semd semd approved these changes

Assignees

@xcrzx xcrzx

Labels
backport:version Backport to applied version labels Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area performance release_note:fix Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.17.3 v8.18.0 v8.19.0 v9.0.0 v9.1.0
Projects
None yet
Milestone
No milestone
6 participants
@xcrzx @elasticmachine @banderror @kibanamachine @maximpn @semd