Skip to content

Add option to disable workqueue bucket rate limiter#4451

Open
Okabe-Junya wants to merge 2 commits intoactions:masterfrom
mercari:Okabe-Junya/feature/add-workqueue-rate-limiter-option
Open

Add option to disable workqueue bucket rate limiter#4451
Okabe-Junya wants to merge 2 commits intoactions:masterfrom
mercari:Okabe-Junya/feature/add-workqueue-rate-limiter-option

Conversation

@Okabe-Junya
Copy link
Copy Markdown

cf. mercari#6

What

Add a --disable-workqueue-bucket-rate-limiter flag to the controller manager. When set, the controller workqueue uses only per-item exponential backoff (DefaultTypedItemBasedRateLimiter) instead of the default DefaultTypedControllerRateLimiter, which includes an overall token bucket limiter (10 QPS, 100 bucket size).

Why

In large-scale environments with many runner scale sets, the default token bucket rate limiter (10 QPS overall) can become a bottleneck for reconciliation throughput. When many EphemeralRunnerSet or AutoscalingRunnerSet resources need to be reconciled simultaneously, the 10 QPS cap delays processing and causes runner provisioning latency.

The per-item exponential backoff is still preserved to prevent tight retry loops on individual failing resources. Removing only the overall bucket constraint allows the controller to process independent reconciliation requests without artificial global throttling.

Note: This flag defaults to false, so the existing behavior is unchanged unless explicitly opted in.

@Okabe-Junya Okabe-Junya marked this pull request as ready for review April 17, 2026 07:09
Copilot AI review requested due to automatic review settings April 17, 2026 07:09
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in controller-manager flag to disable the controller-runtime workqueue’s global token-bucket rate limiter, improving reconciliation throughput in large-scale runner scale set environments.

Changes:

  • Introduces --disable-workqueue-bucket-rate-limiter and wires it to set a typed item-based workqueue rate limiter.
  • Extends Actions controllers’ SetupWithManager to accept optional controller options (rate limiter, etc.).
  • Exposes the flag via the Helm chart deployment template and documents it in values.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
main.go Adds CLI flag and conditionally passes a custom typed workqueue rate limiter into controller setup options.
controllers/actions.github.com/options.go Adds a reusable WithWorkqueueRateLimiter controller option and a Request alias for typed rate limiter generics.
controllers/actions.github.com/ephemeralrunnerset_controller.go Updates SetupWithManager to accept options and apply them via builderWithOptions.
controllers/actions.github.com/autoscalingrunnerset_controller.go Updates SetupWithManager to accept options and apply them via builderWithOptions.
controllers/actions.github.com/autoscalinglistener_controller.go Updates SetupWithManager to accept options and apply them via builderWithOptions.
charts/gha-runner-scale-set-controller/values.yaml Documents the new Helm-exposed flag (currently mispositioned/indented; see comments).
charts/gha-runner-scale-set-controller/templates/deployment.yaml Conditionally adds --disable-workqueue-bucket-rate-limiter based on Helm values.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread controllers/actions.github.com/options.go
Comment thread charts/gha-runner-scale-set-controller/values.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants