Choose API rate limiting algorithms according to the resource you need to protect: request rate, tolerated burst, concurrent work or commercial quota. A token bucket permits a bounded burst and replenishes capacity over time. A concurrency limiter controls active work. Neither replaces authorization, workload isolation or a complete overload strategy.

Key takeaways

• Define the caller identity, scope and cost of an operation before choosing a limit.
• Rate limits and concurrency limits protect different boundaries.
• Client retries need a deadline, attempt budget and duplicate-safe operation contract.
• Test multiple application instances: local counters do not create a fleet-wide quota.

Token bucket, leaky bucket and other API rate limits

In a token-bucket design, available permits accumulate up to a capacity. Accepted work consumes permits; configured replenishment restores them. Capacity represents tolerated burst, while replenishment represents the longer-running allowance. Their values should come from measured workload and downstream headroom.

Leaky bucket is an overloaded term. A queueing shaper drains accepted work at a controlled pace; a metering interpretation may reject traffic that does not fit its timing model. State which behavior your implementation provides. Do not assume every product called a leaky bucket holds requests in a queue.

A request-count policy treats a cheap lookup and a large export equally unless you define separate policies or weighted permits. Protect expensive operations independently. A broad allowance that works for small reads may overwhelm document conversion or database-intensive reports.

Distinguish protection from billing. A short-lived overload limiter and a contractual monthly allowance have different durability, audit and reset requirements. Do not use an ephemeral process counter as the authoritative record of customer consumption.

Where should the limiter run?

Caller → edge / gateway → authentication → endpoint admission → business work
             │                  │                 │
       broad protection    trusted identity   local capacity
             └────────── shared quota, if explicitly required ──────────┘
Rejected work → clear response → bounded client retry or user action

Apply early broad protection where it prevents unnecessary processing, then use authenticated identity for caller-specific fairness. Never trust an arbitrary tenant header as proof of identity. Partition keys should come from validated identity and an approved mapping to the business tenant.

A single NAT address can represent many legitimate people. Conversely, one customer may use many network addresses. Pick identity from the policy’s business purpose, and document the behavior for unauthenticated callers. Bound the number of partitions so uncontrolled key creation cannot consume memory indefinitely.

The ASP.NET Core rate-limiting guidance documents named policies and per-instance in-memory counters. For a shared quota, use a suitable shared enforcement layer or explicitly designed distributed implementation. Test the enforcement scope during scaling and failover.

For a hypothetical three-replica deployment, a local allowance configured on every replica is not one centrally enforced customer allowance. Load balancing, sticky routing and replica restarts affect observed behavior. Decide whether approximate protection is sufficient or a stronger shared policy is required.

A shared store also creates a dependency. Specify what happens when quota evaluation is unavailable. Failing open may protect availability but increase downstream load; failing closed may block legitimate traffic. Make that choice per operation and risk, with monitoring and an explicit recovery path.

A tested .NET token-bucket example

This small console example targets .NET 10 and references the installed Microsoft.AspNetCore.App shared framework. It exercises the limiter directly; it does not expose a network endpoint. Automatic replenishment is disabled so the demonstration controls when replenishment is attempted.

using System.Threading.RateLimiting;

using var limiter = new TokenBucketRateLimiter(new TokenBucketRateLimiterOptions
{
    TokenLimit = 3,
    TokensPerPeriod = 1,
    ReplenishmentPeriod = TimeSpan.FromMilliseconds(100),
    AutoReplenishment = false,
    QueueLimit = 0
});
int checks = 0;
void Check(bool condition, string message)
{
    if (!condition) throw new Exception(message);
    checks++;
}
using var first = limiter.AttemptAcquire(3);
Check(first.IsAcquired, "Initial burst should fit.");
using var rejected = limiter.AttemptAcquire(1);
Check(!rejected.IsAcquired, "Empty bucket should reject.");
Check(rejected.TryGetMetadata(MetadataName.RetryAfter, out var delay) && delay > TimeSpan.Zero,
    "Rejection should estimate replenishment delay.");
await Task.Delay(TimeSpan.FromMilliseconds(250));
Check(limiter.TryReplenish(), "Manual replenishment should run.");
using var later = limiter.AttemptAcquire(1);
Check(later.IsAcquired, "A request should fit after replenishment.");
Console.WriteLine($"{checks} checks passed; local single-process token bucket only.");

Verification result: the local run passed five checks: initial burst acceptance, rejection when empty, positive retry metadata, manual replenishment and a later successful acquisition. It is a single-process demonstration, not a load benchmark or proof of distributed enforcement.

To run it in a console project, set TargetFramework to net10.0 and add <FrameworkReference Include="Microsoft.AspNetCore.App" /> inside an ItemGroup. Use dotnet run. The short delay is for this test, not a production polling schedule.

For an HTTP application, configure the appropriate endpoint policy and place middleware consistently with routing and identity requirements. Configure the rejection response deliberately. Check framework-version documentation and test that the intended endpoint actually uses the policy.

Keep queue size bounded. Queuing may help short bursts, but a long queue consumes resources and makes the accepted request older. When work can outlive an HTTP deadline, consider an explicit durable-job contract rather than holding connections until clients abandon them.

429 responses, retries and exponential backoff

RFC 6585 defines 429 Too Many Requests and permits a Retry-After header. Do not assume every response supplies one. HTTP semantics defines Retry-After as an HTTP date or delay in seconds.

A client policy should parse supported server guidance, account for its remaining deadline and stop when the wait is too long for the user operation. If guidance is absent, a bounded exponential delay with jitter can spread retries. A cap is not permission to retry earlier than an explicit longer server delay: defer the job or return a useful result instead.

For each retryable attempt:
  verify operation is safe to repeat or uses supported idempotency
  verify attempt budget and overall deadline remain
  use valid server retry guidance when present
  otherwise calculate bounded backoff with jitter
  stop or defer if the required wait exceeds the remaining deadline
  wait with cancellation; then make the next attempt

Choose one layer to own retries. If an SDK, service client and workflow each retry independently, one user action may trigger far more requests than intended. Propagate a correlation identifier and record attempt counts without logging credentials or sensitive payloads.

Never treat a timeout as proof that a write did not occur. Before repeating a non-idempotent operation, use the provider’s operation identifier or reconciliation mechanism. A rate-limit response and an uncertain write result require different decisions.

For scheduled work, spread start times rather than launching every tenant at the top of the minute. Use per-tenant budgets and bounded parallelism so one large import cannot starve interactive traffic. Priorities should be explicit rather than accidental consequences of queue order.

Troubleshooting API throttling

Observe allowed and rejected requests, queue delay, active operations and downstream saturation. Segment by policy and a controlled identity dimension. Avoid high-cardinality metric labels containing every user or request ID.

A practical rollout checklist

1. Document the protected resource and caller contract.
2. Measure representative arrival patterns and expensive operations.
3. Define burst, steady rate, concurrency and queue age separately.
4. Test rejected responses and client cancellation.
5. Exercise retry storms, replica changes and dependency failures.
6. Check fairness across small and large tenants.
7. Roll out with monitoring and a reversible configuration change.

Do not choose production values from the tiny demonstration. Establish them with a representative load test and review customer impact before enforcement. Rate limiting supports capacity management; it is not a complete DDoS defense or a substitute for access control.

See the webhook, polling and streaming comparison for arrival patterns, and the integration architecture guide for ownership boundaries.

Common questions

Protect APIs without creating retry storms

Bring traffic profiles, downstream limits and customer expectations. Request an API integration architecture review to define admission, retry and recovery behavior.