Optimizing LLM Inference Costs: Reducing Compute Spend by 65% with vLLM & AWQ
How to deploy custom open-weights models on private GPU clusters, cutting API costs while improving throughput by 4x.
Beyond Commercial API Lock-In
Relying exclusively on proprietary SaaS API endpoints for high-throughput AI agent workloads creates unpredictable recurring costs and latency bottlenecks.
Key Optimization Techniques
- PagedAttention Engine: Implementing vLLM’s memory management to maximize batch sizes on NVIDIA A100/H100 GPUs.
- AWQ 4-bit Quantization: Reducing VRAM footprint by 75% with negligible accuracy drop.
- Speculative Decoding: Using a 1B draft model to accelerate a 70B target model’s output generation.
Start With Measurement
Before optimizing, establish a baseline: cost per thousand requests, tokens per second, time to first token, and p95 latency under realistic load. Optimizations that help throughput can hurt latency, so decide which matters for your workload. Interactive chat prioritizes time to first token. Batch document processing prioritizes total throughput and cost per token.
Why vLLM and PagedAttention Help
The key-value cache for attention grows with sequence length and can waste memory through fragmentation. PagedAttention manages it in fixed-size blocks, similar to virtual memory paging, which reduces waste and lets the server pack more concurrent requests onto each GPU. Combined with continuous batching, where new requests join a running batch instead of waiting for it to finish, GPU utilization rises substantially. This is where most of the throughput gain comes from.
Quantization: Trade-offs, Not Magic
AWQ stores weights in 4 bits while protecting the small fraction of weights that matter most for accuracy. That cuts memory by roughly three quarters versus 16-bit weights, so larger models fit on fewer GPUs. Accuracy loss is usually small, but it is not zero and it varies by model and task. Always evaluate the quantized model on your own tasks, especially for reasoning, code, and long-context work, before shipping. Other options include GPTQ and FP8, which suit recent hardware.
Speculative Decoding
A small draft model proposes several tokens, and the large target model verifies them in a single pass. When the draft is often right, you get the target model’s quality at a lower latency. Gains depend on the acceptance rate, which depends on how well the draft matches the target and on the task, so measure it before committing.
Cheaper Levers Often Overlooked
- Prompt caching and prefix sharing: reuse computation for long shared system prompts.
- Model routing: send easy requests to a small model and escalate only hard ones.
- Output limits: cap max tokens and instruct concise answers.
- Response caching for repeated or near-identical queries.
- Context discipline: retrieve fewer, better passages instead of stuffing the window.
Self-Hosting vs API: A Fair Comparison
Self-hosting wins when volume is high and steady, data cannot leave your environment, or you need custom models. APIs win when traffic is spiky, volume is low, or you cannot staff GPU operations. Count the full cost of ownership: GPU utilization, idle capacity, engineering time, on-call burden, and upgrades. Savings depend heavily on GPU utilization, so model your own traffic before assuming any headline percentage.
Operating It Reliably
Autoscale on queue depth, not CPU. Keep a fallback to a hosted API for overflow and outages. Track GPU memory and the request queue, and load-test with realistic prompt lengths, since average-length benchmarks hide tail behavior.
Related Resources
Read about our AI strategy and consulting and MLOps deployment services, or the LLM fine-tuning and quantization capability. For a cost review of your workload, book a consultation.