GPU Model Deployment Platform
Getting a large model running on a GPU machine once is an afternoon. Making that repeatable, for different models with different shapes and different traffic patterns, is a platform.
This one supports both configurations: serverless, where capacity appears for a workload and goes away again, and clustered, where it stays up.
Two shapes of the same need
Serverless and clustered deployment are usually treated as competing answers. They are not — they are the right answer to different traffic. Intermittent, unpredictable demand should not hold a GPU open; steady demand should not pay a cold start on every call.
Supporting both meant treating the deployment mode as configuration rather than as a fork in the codebase, so the platform has one path to maintain instead of two that drift.
Deployment times
A related and unglamorous win: reworking how the Docker images were configured brought build and deploy times down. On a platform whose whole promise is that shipping a model is routine, the length of that loop is a feature.
What it taught me
- If two deployment modes share a purpose, make the difference a configuration value — a second code path becomes a second product.