Qwen2.5 72B GPU Memory Calculator

Qwen2.5-72B at a glance

Qwen2.5 is Alibaba Cloud's Qwen team's September 2024 model series, ranging from 0.5B to 72B parameters and pre-trained on up to 18 trillion tokens. The flagship Qwen2.5-72B has 72.7 billion parameters (70.0 billion excluding embeddings), 80 layers and grouped-query attention with 64 query heads and 8 key-value heads. Its model card lists a context length of 131,072 tokens with up to 8,192 generated tokens. The default configuration file is set for 32,768 tokens; to go beyond that you enable YaRN rope scaling, which the card notes may slightly affect quality on short texts when applied statically. Unlike most of the series, which is Apache 2.0, the 72B model uses the Qwen licence agreement, so check its commercial terms.

A practical advantage of this model is that the Qwen team publishes its own quantized builds: Qwen2.5-72B-Instruct is available as AWQ, GPTQ-Int4, GPTQ-Int8 and GGUF in addition to BF16, so you are not relying on third-party conversions.

How the estimate works for this model

Weights: 72.7 × bits ÷ 8 gigabytes. Cache: the calculator scales a 0.5 MB-per-token reference by 72.7 ÷ 70, giving about 0.52 MB per token at 16-bit and half that at 8-bit or below, then multiplies by context length and concurrent requests. Overhead: 1.8 GB. Cards needed: the total divided by 92% of one card.

The model's real BF16 cache per token is 2 × 80 layers × 8 KV heads × 128 head dimensions × 2 bytes = 327,680 bytes, about 0.31 MiB. That is the same as Llama 3.1 70B because the attention layout matches. The calculator's figure is about 1.7 times the real one, a safe overestimate.

Default inputs, worked through

As with other 64-head models, three-way tensor parallelism is not supported by common engines, so an unquantized deployment runs on 4 × H100, or on 2 × H200 where 282 GB holds the weights with plenty of cache space.

A long-context example

Qwen2.5-72B is often chosen for long-document work. Set FP8, a 32,768-token context and 4 concurrent requests: the calculator gives 72.7 GB of weights and 33.2 GB of cache, 107.7 GB, or 2 × H100. With the real attention layout and an FP8 cache the 131,072 cached tokens need about 21.5 GB, or about 43 GB with a 16-bit cache. Either way two 80 GB cards are enough, which makes FP8 on a two-GPU node a common choice for this model.

Matching precision to GPUs

• Prefer the official AWQ or GPTQ-Int4 builds for 4-bit GPU serving in vLLM or SGLang; prefer GGUF for llama.cpp, Ollama and CPU or Apple-silicon use.
• 8-bit on a single 80 GB card is a near-miss: weights alone are 72.7 GB, leaving almost nothing for cache. Use two cards or drop to 4-bit.
• If you only need English chat or coding at this size, compare against Llama 3.3 70B too; the hardware is the same, so the decision can rest on quality for your tasks and on licence terms.

Limitations

• The cache rule overstates this model's real cache, and does not model YaRN or prefix caching.
• Embedding and output layers (2.7B parameters together) are usually left in 16-bit in 4-bit builds, so 4-bit files are a few GB larger than 72.7 × 0.5 GB.
• Prefill of a 128K-token prompt needs activation memory beyond the flat 1.8 GB; engines handle this with chunked prefill, but leave margin.
• The throughput figure is a heuristic, not a benchmark of this model.