LLM Cloud Deployment & Cost Benchmarking

For a cloud computing project, a friend and I deployed a 1.5B-parameter LLM (DeepSeek-R1-Distill-Qwen) on CPU-only infrastructure, testing whether a small business could realistically serve a model without paying for a GPU. We set up a FastAPI inference server packaged into a Docker image and ran it on GKE. To stress test the system, we configured a Horizontal Pod Autoscaler (HPA) to scale pods (from 1 to 3 based on CPU utilization) so we could see how autoscaling affected performance and cost under load. Traffic came from K6 load tests simulating steady and burst conditions across three CPU instance types, and for each run we tracked p50/p95 latency, tokens per second, and cost per 1,000 tokens.

One detail we missed during implementation was that the model takes 2–3 minutes to load into memory at container startup, which Kubernetes wasn't built to handle by default, since it assumes containers start fast. The clearest example was that our readiness probe hit /health, which returned 200 OK as soon as the FastAPI process was up but not once the model had actually loaded. So when HPA created a new pod under burst traffic, Kubernetes marked it Ready and started routing requests to it immediately, and those requests would time out. The primary fix was to delay /health until the model had loaded, but to be safe we also decided to push initialDelaySeconds up to 180 and pace HPA's scale-up to one pod per 120 seconds so new pods got a warm-up window instead of getting hit with traffic at once.

In the end, our best configuration (c3-standard-4 with HPA under burst traffic) hit 2.38 tokens/sec with zero failed requests at $0.0136 per 1,000 tokens, making it about 21% cheaper per month at our traffic volume, though its lower throughput makes it more expensive per token than the GPU option. The tradeoff was latency: p95 landed around 40–50 seconds, which rules out anything interactive but works fine for batch or background workloads.