Add option to disable workqueue bucket rate limiter#4451
Open
Okabe-Junya wants to merge 2 commits intoactions:masterfrom
Open
Add option to disable workqueue bucket rate limiter#4451Okabe-Junya wants to merge 2 commits intoactions:masterfrom
Okabe-Junya wants to merge 2 commits intoactions:masterfrom
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opt-in controller-manager flag to disable the controller-runtime workqueue’s global token-bucket rate limiter, improving reconciliation throughput in large-scale runner scale set environments.
Changes:
- Introduces
--disable-workqueue-bucket-rate-limiterand wires it to set a typed item-based workqueue rate limiter. - Extends Actions controllers’
SetupWithManagerto accept optional controller options (rate limiter, etc.). - Exposes the flag via the Helm chart deployment template and documents it in values.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
main.go |
Adds CLI flag and conditionally passes a custom typed workqueue rate limiter into controller setup options. |
controllers/actions.github.com/options.go |
Adds a reusable WithWorkqueueRateLimiter controller option and a Request alias for typed rate limiter generics. |
controllers/actions.github.com/ephemeralrunnerset_controller.go |
Updates SetupWithManager to accept options and apply them via builderWithOptions. |
controllers/actions.github.com/autoscalingrunnerset_controller.go |
Updates SetupWithManager to accept options and apply them via builderWithOptions. |
controllers/actions.github.com/autoscalinglistener_controller.go |
Updates SetupWithManager to accept options and apply them via builderWithOptions. |
charts/gha-runner-scale-set-controller/values.yaml |
Documents the new Helm-exposed flag (currently mispositioned/indented; see comments). |
charts/gha-runner-scale-set-controller/templates/deployment.yaml |
Conditionally adds --disable-workqueue-bucket-rate-limiter based on Helm values. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cf. mercari#6
What
Add a
--disable-workqueue-bucket-rate-limiterflag to the controller manager. When set, the controller workqueue uses only per-item exponential backoff (DefaultTypedItemBasedRateLimiter) instead of the defaultDefaultTypedControllerRateLimiter, which includes an overall token bucket limiter (10 QPS, 100 bucket size).Why
In large-scale environments with many runner scale sets, the default token bucket rate limiter (10 QPS overall) can become a bottleneck for reconciliation throughput. When many
EphemeralRunnerSetorAutoscalingRunnerSetresources need to be reconciled simultaneously, the 10 QPS cap delays processing and causes runner provisioning latency.The per-item exponential backoff is still preserved to prevent tight retry loops on individual failing resources. Removing only the overall bucket constraint allows the controller to process independent reconciliation requests without artificial global throttling.
Note: This flag defaults to
false, so the existing behavior is unchanged unless explicitly opted in.