Llama 3.1 405B GPU Memory Calculator

Llama 3.1 405B: the facts that drive memory

Meta released Llama 3.1 in July 2024 in 8B, 70B and 405B sizes, and the 405B model was the first openly downloadable model of that scale. It is a dense transformer, so unlike DeepSeek's MoE models every one of its roughly 405 billion parameters is used for every token. Meta's published configuration has 126 layers, a hidden size of 16,384, 128 query heads and 8 key-value heads (grouped-query attention), a 128,256-token vocabulary and a 128K-token context window. It is released under the Llama 3.1 Community License, which has an acceptable-use policy and extra terms for very large companies.

Meta trained it in BF16 and also published an official FP8 version of the instruct model. Meta's stated reason for the FP8 release was to let the 405B model run within a single 8-GPU server, and the arithmetic on this page shows why that was necessary.

From inputs to GPU count

The calculator multiplies the parameter count by the bytes each weight takes (2 for FP16/BF16, 1 for FP8 or INT8, about 0.56 for a 4.5-bit GGUF and 0.5 for 4-bit AWQ/GPTQ). It then adds a KV-cache allowance of context × concurrent requests × a per-token size, where the per-token size is a reference 0.5 MB for a 70B model scaled linearly by parameter count, halved when you pick 8-bit or lower. A fixed 1.8 GB covers the CUDA context. The GPU count assumes you can use 92% of each card.

Linear scaling is a poor fit for Llama 405B. Its true BF16 cache per token is 2 (keys and values) × 126 layers × 8 KV heads × 128 dimensions × 2 bytes = 516,096 bytes, about 0.49 MiB. The calculator's rule gives 2.89 MB, about six times more, because grouped-query attention means the cache grows with layer count and KV heads, not with total parameters.

Worked example (default inputs)

1. Weights: 405 billion × 2 bytes = 810.0 GB.
2. KV cache, calculator rule: 8,192 tokens × 2 requests × 2.89 MB ÷ 1024 = 46.3 GB.
3. Add 1.8 GB overhead: 858.1 GB total.
4. Divide by 73.6 GB usable per H100 80 GB: 11.7, so 12 GPUs.

Recomputed with the architecture's real cache size, the same 16,384 cached tokens need about 8.5 GB, giving roughly 820 GB in total. The headline does not change: BF16 405B needs more than one 8 × 80 GB server (640 GB). Twelve is also not a usable tensor-parallel size, since engines want the 128 attention heads and 8 KV heads to split evenly. The usual BF16 layout is 16 GPUs, tensor parallel 8 within each node and pipeline parallel 2 across two nodes, joined by InfiniBand or RoCE.

Which precision fits which hardware

• One 8 × H100 node with FP8 is the standard production choice. 405 GB of weights leaves over 200 GB of HBM for cache across the node, which with a 0.49 MiB per-token cache (or about half that with an FP8 cache) is enough for long contexts and real batching.
• One 8 × H200 node (1,128 GB) can hold the BF16 weights in a single server, which avoids cross-node pipeline stages if you need unquantized output.
• 4-bit builds make 405B possible on 4 × 80 GB cards or on large unified-memory workstations, but this is the size class where the hardware cost of 405B starts to compare poorly with a well-served 70B model. Test whether your workload actually benefits.
• Use NVLink-connected SXM GPUs for tensor parallelism. On PCIe-only cards every layer's all-reduce crosses a much slower link, and decode latency suffers.

Speed expectations, derived rather than benchmarked

A dense model must stream all its weights from memory for each decoding step. One H100 SXM5 has 3.35 TB/s of HBM3 bandwidth, so eight of them provide about 26.8 TB/s together. Reading 405 GB of FP8 weights per step therefore caps a single stream at roughly 66 tokens per second before communication overheads, and real systems land well below that ceiling. Larger batches share each weight read across many requests, which is why serving throughput rises with concurrency while per-user speed does not. The calculator's own throughput line is a simple heuristic, not a benchmark.

Limitations

• The KV estimate assumes cache size scales with parameter count; for Llama 405B it overstates the BF16 cache by about six times.
• GPU counts are memory minimums. Valid parallel layouts (powers of two that divide 8 KV heads) usually push the number up.
• Activation memory during long-prompt prefill, CUDA graphs and fragmentation are covered only by the flat overhead and the 8% margin.
• The licence, not the hardware, can be the deciding constraint. Read Meta's terms before planning a commercial deployment.