Skip to content
coderband
Book a call

FP8 and NVFP4 for Image Generation: What Changes and When It Pays

FP8 makes diffusion transformers 1.2–1.9x faster with about 40% less memory from Ada GPUs on; NVFP4 adds 1.1–1.7x on Blackwell only. Costs, quality, tooling.

coderband engineering12 min read

In published benchmarks, FP8 makes the diffusion transformer 1.2–1.9x faster than BF16 and cuts its memory by about 40%, on any NVIDIA GPU from Ada (L40S, RTX 4090) onward. NVFP4 adds another 1.1–1.7x on top of FP8, but only on Blackwell. Using NVIDIA’s end-to-end FLUX.1 [dev] timings on an RTX 5090 at RunPod’s $1.19 an hour, that is $2.21 per 1,000 images in FP8 and $1.27 in FP4. The catch is quality. 4-bit post-training quantization visibly hurts small text and fine detail unless you correct it, so FP4 pays only after it passes an eval on your own prompts.

Key takeaways

  • FP8 is the default on Ada, Hopper and Blackwell. The PyTorch team saw “very negligible differences” on FLUX.1 [dev].
  • NVFP4 is Blackwell-only, and B200 (compute capability 10.0) and RTX 5090 (12.0) are different targets with different tooling support.
  • Without hardware support you save memory, not time. The weights are dequantized back to BF16 for every matmul.
  • Quantize the transformer. It takes 96–98% of the time in NVIDIA’s FLUX profiles.
  • Many headline speed-ups mix in a compiler change or an offloading baseline. Use like-for-like numbers.
  • FP4 post-training quantization degrades small text first. QAT or SVDQuant recover most of it.
  • Measure quality with paired, same-seed metrics and targeted prompt sets, not FID on a few hundred images.

This post goes deep on one lever from our pillar guide, How to Cut GPU Inference Costs for Image Generation, which covers the cost model, distillation, compilation and utilization.

What the formats are

Low-precision formats give up bits, so they need a scale factor that maps each group of values into the small range the format can hold. The size of that group, and the format of its scale, is most of what separates the options.

Format Element bits Scale Largest value Where it runs in hardware
FP8 E4M3 4 exponent, 3 mantissa Per tensor or per row, chosen by software ±448 Ada, Hopper, Blackwell
FP8 E5M2 5 exponent, 2 mantissa Per tensor or per row ±57,344 Ada, Hopper, Blackwell
MXFP8 E4M3 or E5M2 One E8M0 (power-of-two) scale per 32 values Same as element Blackwell (data center, in torchao today)
MXFP4 E2M1 One E8M0 scale per 32 values ±6 per element Blackwell
NVFP4 E2M1 One FP8 E4M3 scale per 16 values, plus an FP32 per-tensor scale ±6 per element Blackwell

Sources: the FP8 Formats paper, the OCP Microscaling (MX) spec (AMD, Arm, Intel, Meta, Microsoft, NVIDIA, Qualcomm) for MX formats and the value limits, and NVIDIA’s NVFP4 explainer.

What matters in practice:

  • E4M3 is the inference format. The FP8 paper notes that inference uses E4M3, and E5M2’s extra range is for training gradients.
  • FP4 accuracy lives in the scales. E2M1 has one mantissa bit. NVFP4’s smaller blocks and finer FP8 scale mean one outlier distorts fewer neighbours. At about 4.5 bits per value, NVIDIA puts it 3.5x smaller than FP16 and 1.8x smaller than FP8.
  • Speed needs weights and activations quantized (W8A8, W4A4). Weight-only schemes (W4A16, NF4, most GGUF files) shrink the model but compute in BF16, and the Diffusers torchao guide notes they keep “the memory peaks for activation computation”.

Which GPUs support them

GPU Architecture Compute capability FP8 tensor cores FP4 tensor cores
A100 Ampere 8.0 No No
L4, L40S, RTX 4090 Ada 8.9 Yes No
H100, H200 Hopper 9.0 Yes No
B200, GB200 (B300 is 10.3) Blackwell data center 10.0 Yes Yes
RTX 5090, RTX PRO 6000 Blackwell Blackwell client and workstation 12.0 Yes Yes

Compute capabilities are from NVIDIA’s CUDA GPU list. torchao’s FP8 needs compute capability 8.9 or higher, and the Diffusers NVFP4 kernels for Black Forest Labs checkpoints need 10.0 or higher. FP4 comes from Blackwell’s second-generation Transformer Engine.

Three consequences catch teams out:

  1. No FP8 on A100. Without native support, the weights are “dequantized to compute_dtype in each forward pass” and activations stay in 16-bit. You save VRAM, not time.
  2. No FP4 on H100. For 4-bit before Blackwell, SVDQuant’s INT4 path is the main option, and the Diffusers Nunchaku Lite kernels don’t support Hopper yet.
  3. sm_100 and sm_120 differ. torchao’s prototype NVFP4 and MXFP8 configs require SM100+ and don’t mention consumer Blackwell. The BFL path supports 12.0 cards with CUDA 12.9 or newer. Check your exact GPU against your exact library.

What gets quantized in a diffusion pipeline

A pipeline has text encoders, a denoising backbone (a UNet in SDXL, a diffusion transformer in FLUX and SD3.5) and a VAE decoder. Only the backbone runs once per step, so it is the target. NVIDIA measured the transformer at about 96% of FLUX.1 Kontext’s processing time and 98% of FLUX.1 [dev] latency on an RTX 5090.

Inside the transformer, not everything gets the same treatment:

  • Linear layers go to FP8 or FP4. That is where the tensor-core speed-up lives.
  • Attention mostly stays in FP8, even in “FP4” builds. NVIDIA’s FLUX.1 [dev] FP4 recipe runs multi-head attention in FP8, and its Kontext recipe keeps softmax in FP32. That is why FP4 gains less over FP8 than the bit count suggests.
  • Embeddings, output layers and norms stay in higher precision in NVIDIA’s FLUX.1 recipe. NVIDIA’s FLUX.2 example retains layers by name pattern (embedder, norm_out, proj_out).
  • Convolutions are awkward. Diffusers’ ModelOpt integration has a disable_conv_quantization flag because block quantization “generally don’t work well with convolutional layers”, which matters for UNets and VAEs.

Text encoders are reasonable FP8 candidates for memory: NVIDIA found FLUX.2 robust with an FP8 text encoder. The VAE usually stays in BF16, as in Stability’s SD3.5 Large TensorRT release, which ships only the MMDiT in FP8.

Quantizing only the transformer shrinks the whole pipeline less than you expect. Black Forest Labs measured peak memory for FLUX 3 Image Klein, a 5.6B transformer with a separate 4B text encoder, on a B200 at 1024x1024: 31 GB in BF16, 26 GB in FP8 and 25 GB in NVFP4. Peak memory dropped by less than a fifth. On larger models the picture is better: NVIDIA reports SD3.5 Large going from over 18 GB to 11 GB with FP8, and FLUX.2 [dev] needing 40% less VRAM.

The tooling, as of November 2026

The quickest real FP8 test on an L40S or H100, which the Diffusers docs recommend pairing with torch.compile:

import torch
from diffusers import FluxPipeline, PipelineQuantizationConfig, TorchAoConfig
from torchao.quantization import Float8DynamicActivationFloat8WeightConfig, PerRow

quant = PipelineQuantizationConfig(
    quant_mapping={
        "transformer": TorchAoConfig(
            Float8DynamicActivationFloat8WeightConfig(granularity=PerRow())
        )
    }
)
pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    quantization_config=quant,
    dtype=torch.bfloat16,
).to("cuda")

# FP8 matmuls only pay off once the graph is compiled and fused.
pipe.transformer.compile(mode="max-autotune", fullgraph=True)

Published speed and memory numbers

Model and setup GPU 16-bit FP8 FP4 Source
FLUX.1 [dev], 30 steps, end to end, TensorRT RTX 5090 10.93 s (FP16, low-VRAM mode) 6.68 s 3.85 s NVIDIA
FLUX.1 [dev], VRAM, default mode RTX 5090 35.7 GB (BF16) 24.6 GB 21.67 GB same
FLUX.1 Kontext [dev], per step, transformer RTX PRO 6000 607 ms 317 ms 254 ms NVIDIA
SD3.5 Large, 30 steps, end to end, TensorRT both H100 8.10 s (BF16) 5.71 s n/a Stability model card
FLUX.1 [schnell], batch 1, torchao not stated 0.418 s 0.346 s 0.320 s torchao docs
FLUX.2 [klein], 1024x1024 RTX 5080/5090 baseline up to 1.6x faster up to 2.7x faster Black Forest Labs

These are vendor or maintainer numbers, so treat them as upper bounds. Three things explain the spread:

  1. The runtime matters as much as the format. NVIDIA’s “2.3x” for SD3.5 Large compares FP8 TensorRT with BF16 PyTorch. With TensorRT on both sides, Stability’s card shows the MMDiT going from 7,945 ms to 5,605 ms: 1.42x for precision, the rest from TensorRT.
  2. Some baselines were offloading. NVIDIA’s 16-bit runs on the 32 GB RTX 5090 needed low-VRAM mode, so its FLUX.1 [schnell] “speed-up” from 4.43 s to 0.91 s mostly measures not offloading. FP8 to FP4 (1.73x on [dev]) is the clean comparison.
  3. Fused kernels beat generic ones. torchao’s generic NVFP4 is only 1.08x over its FP8 on FLUX.1 [schnell], and its roofline shows dynamic NVFP4 at 0.46x of BF16 speed on 1024-sized matmuls. Nunchaku’s fused engine reports NVFP4 FLUX.1 [dev] 3.1x faster than BF16 on an RTX 5090. Small models and small resolutions gain less.

Quality: where quantization hurts

Most vendor posts show a handful of side-by-sides, not metrics. Run your own eval.

How to measure it

  1. Pair every image: same prompt, seed and scheduler, BF16 against the candidate. That separates the precision change from seed-to-seed variation.
  2. LPIPS against the BF16 output (Zhang et al.) measures how far pixels moved, not whether the change is worse.
  3. CLIP score (Hessel et al.) and ImageReward (Xu et al.) check prompt adherence and preference. CLIP metrics carry CLIP’s biases, and neither checks spelling.
  4. FID only at scale. FID is biased at finite sample sizes, in a model-dependent way. SVDQuant used 5,000 prompts. With 300 images, skip it.
  5. Targeted failure-mode sets: rendered text (PartiPrompts has a “Writing & Symbols” category), faces, hands, hair, fabric, foliage and logos. For text, OCR both outputs and compare strings.
  6. Blind human A/B with people who know the product. Fix the pass threshold before anyone sees the speed numbers.
import lpips, torch
from torchmetrics.functional.multimodal import clip_score

lp = lpips.LPIPS(net="alex").cuda()
CLIP = "openai/clip-vit-large-patch14"

def to_lpips(x):  # uint8 NCHW in [0, 255] -> float in [-1, 1]
    return x.float().div(127.5).sub(1).cuda()

def compare(base, cand, prompts):
    # base, cand: uint8 tensors (N, 3, H, W) generated with identical seeds
    drift = lp(to_lpips(base), to_lpips(cand)).mean().item()
    clip_b = clip_score(base, prompts, model_name_or_path=CLIP).item()
    clip_c = clip_score(cand, prompts, model_name_or_path=CLIP).item()
    return {"lpips": drift, "clip_base": clip_b, "clip_cand": clip_c}

The cost math per 1,000 images

Same formula as the pillar post: price per hour divided by images per hour, at batch size 1 and full utilization. On-demand prices are from RunPod (updated 4 November 2026) and Modal (per-second rate times 3,600), checked 5 November 2026. Latencies are the end-to-end vendor numbers above.

Setup Seconds per image Images per hour RunPod Secure RunPod Community Modal
FLUX.1 [dev], RTX 5090, FP16 (offloading) 10.93 329 $3.61 ($1.19/h) $2.10 ($0.69/h) n/a
FLUX.1 [dev], RTX 5090, FP8 6.68 539 $2.21 $1.28 n/a
FLUX.1 [dev], RTX 5090, FP4 3.85 934 $1.27 $0.74 n/a
FLUX.1 [dev], RTX 4090, FP8 (offloading) 10.62 339 $2.63 ($0.89/h) $1.00 ($0.34/h) n/a
SD3.5 Large, H100, BF16 8.10 444 $8.98 ($3.99/h) $6.05 ($2.69/h) $8.89 ($3.95/h)
SD3.5 Large, H100, FP8 5.71 631 $6.33 $4.27 $6.26

How to read it:

  • FP8 on Hopper saved about 30% on SD3.5 Large, like for like in TensorRT.
  • Price the GPU and the precision together. FP8 on a $0.89 RTX 4090 costs about twice as much per image as FP4 on a $1.19 RTX 5090, because the 4090 has to offload.
  • Utilization still dominates. The FP8 RTX 5090 row at 50% utilization costs $4.42, worse than FP16 running fully busy.
  • The saving has to pay for the work. At 1 million FLUX.1 [dev] images a month, FP8 to FP4 saves about $940 a month at Secure Cloud prices, which pays for an eval and an engine build. At 50,000 a month it saves under $50. For weighing engineering spend against run cost, see our AI feature cost breakdown.

When FP8 pays, when FP4 pays, when to stay on BF16

Your situation Choose Why
A100 or older BF16; INT4 SVDQuant only if you need it to fit No FP8 tensor cores. FP8 weights get dequantized, so you save memory but not time
L40S, L4, RTX 4090, H100, H200 FP8 transformer, BF16 VAE Native FP8 support, the smallest quality drift in public data, and mature tooling (torchao, TensorRT)
B200 or RTX 5090/PRO 6000, high volume, photo or illustration output NVFP4 linear layers, FP8 attention, after an eval 1.25–1.7x over FP8 in fused runtimes. Use a vendor checkpoint, QAT or SVDQuant rather than plain PTQ
Text in images, logos, product packaging, UI mock-ups FP8, or BF16 if FP8 fails your OCR check Small text is the first documented casualty of FP4 PTQ
Fixed seeds must reproduce compositions BF16 or FP8 NVFP4 can change composition, not just texture
4-step distilled model at 512–1024 px FP8; test FP4 last Small matmuls and CPU overhead shrink the gain, and generic NVFP4 can be slower than BF16 on small shapes
Users bring their own LoRAs FP8 in PyTorch, or SVDQuant SVDQuant applies LoRAs without re-quantization. TensorRT engines need a rebuild per change
Low volume, or GPUs idle most of the day Stay where you are Scale-to-zero and batching save more. See the pillar

The rule of thumb: on Ada and Hopper, FP8 is a default you need a reason not to take. On Blackwell, NVFP4 is an option you need evidence to take. If an outside team does the work, a fixed scope with a measurable pass bar is easier to hold them to; our notes on fixed-price vs time-and-materials explain why.

Getting a measured answer for your pipeline

Every number above is someone else’s model on someone else’s GPU. Yours depends on your model, resolution, step count, LoRAs and quality bar, so it has to be measured.

That is what the GPU Inference Speed Audit does. It is a fixed $4,900, paid upfront, and takes one week. On day one we record a reproducible baseline on your hardware and agree a quality bar with you. Then we profile the pipeline and prototype the precision changes that fit it (FP8, NVFP4, SVDQuant or a TensorRT engine, plus whatever else the profile turns up), each measured against that bar. You get the benchmark scripts, raw results, a ranked plan with effort and risk for each change, and a fixed quote to implement it if you want one.

If we can’t show at least 30% lower cost per output or 30% lower latency, measured on your workload at the agreed quality bar, you get a full refund automatically and keep the report.