Every FastAPI codebase that eventually falls over shares one root cause: it was written as a stack of endpoints instead of a system with clear boundaries. When traffic ramps, the endpoints do too much — auth, validation, DB calls, third-party fan-out — and the event loop starves. The fix is not more workers. The fix is treating the service like a small operating system.
The mental shift
An OS is a scheduler surrounded by rings of privilege. Requests enter at the outermost ring, pass through middlewares (auth, tracing, rate limits), get validated into typed intents, and only then reach handlers — the equivalent of syscalls. Long-running work is delegated to background workers, never blocked on inline.
The kernel model
Below is the shape I converge on for any FastAPI service pushing beyond ~500 rps per pod. Every arrow is await-safe; every box has one job.
Middleware as a ring
Order matters. Trace context first (so failures upstream still get spans), then request logging, then auth, then rate limiting, then body validation. Anything that can reject a request cheaply belongs earlier than anything that touches a database.
app = FastAPI()
app.add_middleware(TraceMiddleware)
app.add_middleware(AccessLogMiddleware)
app.add_middleware(AuthMiddleware)
app.add_middleware(RateLimitMiddleware, per_route=True)Scheduling & backpressure
The event loop is the CPU. If a handler does synchronous DB work, you have introduced a global lock. Use asyncpg or databases, wrap CPU-heavy work in run_in_executor, and expose a bounded semaphore per downstream so one slow dependency cannot drain the pool.
asyncio.Semaphore(64) in front of every third-party HTTP client is the single highest-leverage change I have shipped. It turns cascading failures into graceful degradation.Background workers
Anything longer than the P99 request budget goes to a queue — arq, Celery, or a plain Redis stream. Handlers stay boring: validate, enqueue, return an ID. Idempotency keys on the enqueue path save you the first time a client retries.
Observability
- OpenTelemetry spans on every middleware and every outbound call.
- RED metrics per route: Rate, Errors, Duration — nothing more, nothing less.
- Structured logs keyed by
trace_id, sampled at 10% in prod.
Takeaways
- Handlers are syscalls. Keep them ten lines or fewer.
- Every downstream gets a semaphore, a timeout, and a circuit breaker.
- Async all the way down, or don't bother going async at all.
- The bottleneck is almost always the DB pool — size it, measure it, alert on it.