API Rate Limiting

If your Spinnaker deployment has API clients interacting with it, enabling and knowing how to operate the API rate limiter can help keep Spinnaker reliable through spikes of heavy traffic or rogue clients.

Intro

The API Rate Limiter currently supports limiting individual authenticated and anonymous principals, bucketing requests into windows that are refreshed every interval as well as global & per-principal learning mode. Both the capacity, as well as the window size, are configurable globally and per-principal.

Some metrics are generated by the rate limiter:

• rateLimit.throttling: Counter. Number of enforced throttled requests.
• rateLimit.throttlingLearning: Counter. Number of un-enforced throttled requests.
• rateLimit.principal.throttled: Counter. A tagged metric recording the number of throttled requests per-principal. Tag key is principal.
• rateLimit.principal.remaining: Gauge. A tagged metric recording the number of remaining requests per-principal. Tag key is principal.

Combined with controller.invocations, these metrics can give you a good sense of your on-going rate limiting configuration.

In the case of failures (Redis unavailable) or configuration errors (principal included in both enforcing and ignoring lists), a request will gracefully fallback to learning mode.

Enabling in gate.yml

The rate limiter requires Redis to be available for Gate.

rateLimit:
  enabled: true
  learning: true
  redis:
    enabled: true
  # The rate (in seconds) that the bucket will be replenished
  rateSeconds: 10
  # The number of allowed requests in the window of rateSeconds
  capacity: 500
  # Specific principals can be given different capacities
  capacityByPrincipal:
  - principal: anonymous
    override: 1000
  # Similarly, rateSeconds can be overridden per-principal
  rateSecondsByPrincipal:
  - principal: anonymous
    override: 5
  # A list of principals that are being enforced. Handy for cases where you want
  # to incrementally enable the rate limiter
  enforcing:
  - example@example.com
  # A list of principals that are in learning mode. This can be useful if you
  # want to give some principals unlimited power. Reconsider doing this :)
  ignoring:
  - unmetered-example@example.com

Integrating clients

The rate limiter exposes four HTTP Headers on each request that a client can ingest to play nicely with the rate limiter:

• X-RateLimit-Capacity: Int. Total capacity assigned to the principal.
• X-RateLimit-Remaining: Int. Total number of requests left before reset.
• X-RateLimit-Reset: Date. When the bucket will be replenished.
• X-RateLimit-Learning: Bool. Whether or not the rate limiter is enforcing the principal.

Anonymous clients

If your Spinnaker deployment allows anonymous API access, by default all requests will be bucketed into the same rate limit window. With a little trust, you can allow your users to assign themselves their own anonymous rate limit bucket by sending a X-RateLimit-App header, where the value is the name of their application.

Dynamic configuration

Re-deploying gate to respond to real-world events is non-ideal. Redis has the capability to override any configuration defined by the static configuration. First, a list of key patterns:

• rateLimit* - All rate limiter-related keys.
• rateLimit:* - Principal rate limiter.
• rateLimit:learning - Global learning flag.
• rateLimit:enforcing - A list of principals being enforced.
• rateLimit:ignoring - A list of principals in learning mode.
• rateLimit:capacity:* - Per-principal capacity.
• rateLimit:rateSeconds:* - Per-principal rateSeconds.

Examples

$ redis-cli

# Set a new capacity
127.0.0.1:6379> set rateLimit:capacity:chaosmonkey@example.com 2000

# Add the chaosmonkey@example.com principal to the enforcing list
127.0.0.1:6379> sadd rateLimit:enforcing chaosmonkey@example.com
(integer) 1

Reference

Code backing the rate limiter can be found in the gate repository .