Skip to main content

A 27B model on one RTX 3090

· 4 min read
Kit Kong
Kit Kong
Engineer in Vancouver, AI systems and trading infrastructure

Most of my local inference runs on one RTX 3090 with 24 GB of VRAM. This post covers what fits on it, what doesn't, and why the limit is usually not the model weights.

What runs on the card​

My daily coding model is a Qwen 27B in GGUF, run through llama.cpp with CUDA. At 4 bits, either Q4_K_M or IQ4_XS, the weights fit with room left for a long context. For image understanding I use Qwen3-VL-8B at Q6_K. I use Ollama for quick one-off runs, and ComfyUI for image generation. The same card also ran the vLLM experiments for my KV-cache capstone.

GGUF K-quants are the reason a 27B model fits at all. They quantize weights in small blocks, each with its own scale, so 4-bit weights lose much less than a naive 4-bit cast would. IQ4_XS is slightly smaller than Q4_K_M. On a 24 GB card, that difference goes straight into context length.

My current setup is the 27B at IQ4_XS in llama.cpp: 215K context with vision. The community benchmark for that recipe puts it at 47 and 56 tokens per second on its two workloads.

Context is a KV cache budget​

Once the weights load, the question is how much context you can afford, and that comes down to the KV cache. For a given model, most of the difference in maximum context comes from how the KV cache is quantized, not from the model.

For the 27B I checked, a q4_0 KV cache costs about 18,432 bytes per token and q8_0 about 34,816, almost exactly twice as much. On a single-card llama.cpp IQ4_NL setup, that meant a ceiling of 131K tokens with q8_0 KV. The 200K context reported for an earlier version of the model depended on q4_0 KV.

Two traps:

  • "It allocated" is not "it works." Flash attention's scratch memory grows as the context actually fills, so a context that allocates at startup can still fail once real input fills it. Long context only counts once you've run a fill ladder or a needle-in-a-haystack test at the size you claim.
  • Hybrid models break the paper math. Some newer models mix a few full-attention layers with many linear-attention (DeltaNet) layers. Those layers carry recurrent state that also takes VRAM, so a back-of-envelope estimate that counts only KV comes out too optimistic.

What Ampere can't do​

The 3090 is Ampere (sm_86). It has no native FP8 or FP4 math. NVFP4 weights do run, through a Marlin W4A16 fallback: weights stay 4-bit, but the math runs in 16-bit, so you get the memory savings without Blackwell's speedup. Separately, the newest Triton attention backend in vLLM errors out with an fp8 KV cache on this card, so I keep the KV in fp16 or bf16 there.

That doesn't mean low-bit KV is off the table. In my capstone, TurboQuant's rotation-based 4-bit KV scored 0.83 on GSM8K against 0.80 for bf16, so 4-bit KV doesn't have to cost quality.

The real ceiling is system RAM​

The constraint I hit most often isn't VRAM. It's the host. The machine has 32 GB of RAM, and WSL2 gets 24 GB of it. Under Ollama on Windows, only about 11 GB was available, not enough for a model that needed 36 GB. Some of the newer large models people pair with a single card need 84–148 GB of system RAM, which rules them out entirely.

I looked at the usual upgrades: a 96 GB workstation card, multiple 3090s, large-memory Macs, Intel Arc, Ryzen AI Max, a DGX Spark. I bought none of them. For what I run, upgrading system RAM to 64 GB comes before any new GPU.

Training on it, too​

The card isn't only for inference. I trained an OPT-125M ranker, the kind of small length predictor that learning-to-rank schedulers use, and published the model and its dataset on Hugging Face. A 125M-parameter model is a single-card training job, and it's a good reminder that a lot of useful ML systems work needs a small model in the right place, not a big one.

What's next​

Community recipes for single-3090 setups change quickly. The latest READMEs report patched vLLM builds running a Qwen 27B about 1.5–2× faster than my llama.cpp setup, including at long context, but without vision. Those numbers are the authors' own benchmarks at different power limits, so the next step is to rerun them on my card before switching.

The general rule I've kept: when someone posts 100+ tokens per second, check the model, the context length and the concurrency before comparing it with your own setup.