Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http: request queue and timeouts #48

Merged
merged 3 commits into from
Jul 15, 2019
Merged

http: request queue and timeouts #48

merged 3 commits into from
Jul 15, 2019

Conversation

mastercactapus
Copy link
Member

  • Identified the issue which this PR solves.
  • Read the CONTRIBUTING document.
  • Code builds clean without any errors or warnings.
  • Added appropriate tests for any new functionality.
  • All new and existing tests passed.
  • Added comments in the code, where necessary.
  • Ran make check to catch common errors. Fixed any that came up.

Description:
This PR introduces request queueing to the concurrency limiter.

The concurrency limiter was designed to limit the number of simultaneous integration keys, service, and/or user-requests that are processed concurrently per-instance.

The present-day implementation limits concurrency, effectively isolating busy consumers, but all "pending" requests are then processed in random order. This results in latency spikes under load (per individual key) and 502 responses due to "backend timeout".

This change introduces a FIFO queue for all pending requests instead of the random group.

  • For requests beyond the maximum (250 per key), they are immediately returned with status code 429 (too many requests)
  • Requests stuck in the queue for longer than 25s are also returned with 429

Rejected requests are still scoped per integration key/service/user as before but will now allow a distinction between actual hung requests and throttling.

Describe any introduced API changes:
Throttled requests will now return status code 429 instead of timing out and returning 502.

@mastercactapus mastercactapus requested review from m17ch and arurao July 11, 2019 14:29
m17ch
m17ch previously approved these changes Jul 11, 2019
Copy link
Contributor

@m17ch m17ch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@arurao arurao self-requested a review July 15, 2019 14:41
@mastercactapus mastercactapus merged commit fafb486 into master Jul 15, 2019
@mastercactapus mastercactapus deleted the request-queue branch July 15, 2019 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants