Exact VRAM memory allocation, dynamic KV-cache requirements, and tensor parallelism slicing for Llama-3.1 70B Enterprise quantized in FP16 Uncompressed Native deployed on NVIDIA H100 80GB SXM5.
The default case totals 149.8 GB, a memory minimum of 3 × 80 GB. Engines need an even split of 64 heads, so BF16 serving normally uses 4 GPUs.
Barely: the calculator total is 75.8 GB, leaving almost no room for KV cache. Two GPUs is the practical FP8 setup.
80 layers × 8 KV heads × 128 dimensions × 2 (keys and values) × 2 bytes is 327,680 bytes per token in BF16, about 0.31 MiB.
Llama 3.1 70B is the middle model of Meta's July 2024 Llama 3.1 family, published as both a pretrained base model and an instruction-tuned chat model. Its exact size is about 70.6 billion parameters; this calculator rounds the default to 70. It has 80 transformer layers, a hidden size of 8,192, 64 query heads sharing 8 key-value heads, and a 128K-token context window, up from 8K in Llama 3. Because both base and instruct weights exist, 3.1 70B is the usual starting point for fine-tuning. Llama 3.3 70B, released later, is an instruct-only model with the same architecture, so the memory figures below apply to both.
For a 70B model the calculator's 0.5 MB-per-token reference is used directly, so this is the page where its KV rule is closest to its source. The model's real BF16 cache is 2 × 80 layers × 8 KV heads × 128 dims × 2 bytes = 327,680 bytes per token (0.31 MiB), so the rule still overstates it by about 60%. Halving the cache at 8-bit corresponds to running an FP8 KV cache, which vLLM and TensorRT-LLM support but do not enable by default.
| Line | Working | Shown |
|---|---|---|
| Model weights | 70 × 2 bytes | 140.0 GB |
| KV cache | 8,192 × 2 × 0.5 MB ÷ 1024 | 8.0 GB |
| Overhead | constant | 1.8 GB |
| Total VRAM | 140 + 8 + 1.8 | 149.8 GB |
| Minimum GPUs | 149.8 ÷ 73.6 = 2.04 → round up | 3 × 80 GB |
Three GPUs is the memory minimum, but most engines cannot split 64 attention heads and 8 KV heads three ways. A BF16 deployment therefore normally uses tensor parallelism across 4 GPUs, or 2 × H200 (141 GB each). With the exact 70.6B parameter count the weights come to about 141 GB, which does not change the conclusion.
Raise the context to the full 131,072 tokens for a single request and the calculator's cache becomes 64 GB at 16-bit, for a total of 205.8 GB. With the real per-token size the cache is about 43 GB. Either way, one full-length 128K conversation costs roughly a third to half as much memory as the model itself. If you plan for long documents, size for the longest prompt plus output you will accept, multiplied by how many such requests may be in flight.
| Setup | Precision | Calculator total | Comment |
|---|---|---|---|
| 4 × H100 80 GB | BF16 | 149.8 GB at 8K × 2 | Standard unquantized serving with room for batching |
| 2 × H100 80 GB | FP8 | 75.8 GB | Common production choice; leaves about 70 GB for cache |
| 1 × H100 80 GB | FP8 | 75.8 GB | Technically fits under the 95% line, but almost no cache room |
| 1 × 48 GB workstation card | 4-bit | 40.8 GB | Single-user use with short contexts |
| 2 × 24 GB consumer cards | 4-bit | 40.8 GB | Works with llama.cpp or exllama-style splitting; slower than one larger card |
Note an inconsistency you may see when you choose FP8 with the default settings: the status line says the model fits on a single 80 GB GPU, because 75.8 GB is under 95% of 80 GB, while the GPU count says 2, because it uses the stricter 92% figure. Treat the stricter number as the planning answer: a single card with only 4 GB spare cannot hold a useful batch.