Exact VRAM memory allocation, dynamic KV-cache requirements, and tensor parallelism slicing for Gemma-2 27B Google Research quantized in FP16 Uncompressed Native deployed on NVIDIA H100 80GB SXM5.
Yes. In BF16 the default case totals 59.3 GB, under the 80 GB card's limit. Allowing for the larger real cache, expect about 61–62 GB.
It has 16 KV heads, so each full-attention layer stores 8 KiB per token. Half of the 46 layers use a 4,096-token sliding window, giving roughly 276 KiB per token at full context, against the calculator's 199 KiB.
Yes, with a 4-bit or Q4_K_M build of roughly 16–17 GB. The calculator gives 17.9 GB at one request and 8K context, which fits 24 GB.
Gemma 2 is Google DeepMind's open-weight model family released in June 2024 in 2B, 9B and 27B sizes. The largest has about 27.2 billion parameters and, according to the technical report, was trained on 13 trillion tokens (the smaller two were trained with knowledge distillation; the 27B was not). Its configuration has 46 layers, a model width of 4,608, 32 query heads and 16 key-value heads of dimension 128. The context window is short by current standards: 8,192 tokens. The weights are published under Google's Gemma terms of use.
Two architectural details matter in practice. Gemma 2 alternates local and global attention: every other layer only looks at the most recent 4,096 tokens (a sliding window), while the others attend to the whole context. It also uses logit soft-capping in attention, which some attention kernels did not support when the model launched, so use an engine release that lists Gemma 2 support.
It takes 27.2 × bits ÷ 8 for weights, adds a cache of context × requests × (27.2 ÷ 70 × 0.5 MB) at 16-bit, which is about 199 KiB per token, plus 1.8 GB of overhead, and divides by 92% of one GPU.
Here the cache rule underestimates, unlike on the larger model pages. Gemma 2 27B keeps 16 KV heads, so a full-attention layer stores 2 × 16 × 128 × 2 bytes = 8 KiB per token, and 46 layers make 368 KiB per token if every layer kept the whole context. Because half the layers keep only the last 4,096 tokens, at the full 8,192-token context the average is about three-quarters of that, roughly 276 KiB per token, still about 40% above the calculator's figure. Engines that do not implement the sliding-window saving use the full 368 KiB.
So the real total is closer to 61–62 GB, but the conclusion holds: BF16 Gemma 2 27B is a single-H100 model. Setting the context field above 8,192 produces numbers for a configuration the model does not support; leave it at 8,192 or less.
| Card | Suggested build | Calculator total | Notes |
|---|---|---|---|
| 80 GB (H100, A100 80 GB) | BF16 | 59.3 GB at 8K × 2 | Unquantized serving with moderate batching |
| 48 GB (RTX 6000 Ada, A6000) | 8-bit | 30.6 GB at 8K × 2 | BF16 (59.3 GB) does not fit |
| 24 GB (RTX 3090 / 4090) | 4-bit or Q4_K_M | 17.9 GB at 8K × 1 (Q4_K_M) | Real Q4_K_M files are about 16–17 GB; keep context modest |
| 16 GB | Consider Gemma 2 9B | — | 27B at 4-bit leaves too little room for cache |
On a single 24 GB card, the 4-bit 27B model is a popular local setup. Because the real cache is larger than the calculator assumes, add 1–2 GB of margin before concluding that a long 8K conversation will fit.
Decoding reads all weights for each token, so the ceiling is memory bandwidth divided by model bytes. On an H100 SXM5 at 3.35 TB/s and 54.4 GB of BF16 weights, the ceiling is around 60 tokens per second for one stream; on an RTX 4090 at about 1 TB/s and a roughly 16 GB 4-bit file, it is in the same range. Real speeds are lower. The calculator's throughput estimate is a simple formula and not a measurement.