LLM Inference Server
Serving a large language model is not a request-response problem. The work is queuing, admission, and knowing which of a small number of very expensive machines is free — everything around the model, rather than the model itself.
I designed the architecture and built the API layer clients talk to, plus the management interface used to operate it.
The problem
GPUs are scarce, expensive, and slow to warm. An inference server has to absorb bursts without dropping work, keep long-running generations from starving short ones, and stay observable while doing it — none of which the model has an opinion about.
That makes the serving layer the actual engineering surface, and the reason it needed designing before it needed writing.
Approach
The system separates the path a request takes from the machinery that operates it. Clients meet a FastAPI surface; work moves through a queue rather than being handed straight to hardware, so load is something the system absorbs instead of something it forwards.
Postgres holds the durable state, Redis the state that only needs to be fast, and Kafka carries the traffic between stages. Nothing exotic — the difficulty here is in the boundaries, not the components.
The management interface
An inference server that cannot be inspected is a liability, so the Angular control plane was built alongside the API rather than after it. It is the difference between a service someone can run and a service someone has to babysit.
What it taught me
- When the expensive resource is the constraint, the queue in front of it is the architecture.
- The operations interface is not a follow-up task. Building it in parallel is what makes the design's assumptions visible early.