Smaller KV cache, faster first token
· 6 min read
For our graduate capstone this summer, a team of four built a latency stack on vLLM serving Llama-3.1-8B on a single RTX 3090. My part was the KV-cache layer underneath the scheduler. This post covers what that layer did and why it moved first-token latency more than anything else we tried.