Hardware FP4 inference for speech on Blackwell

Blackwell GPUs multiply 4-bit floating-point matrices in hardware. The text-LLM stack adopted this within months; speech had, as far as I could find, not been tested at all. So I tested it: three architectures — Orpheus-3B (autoregressive LLM TTS), F5-TTS (flow-matching diffusion transformer, 336M), and Whisper large-v3 (encoder-decoder ASR, 1.5B) — on one Blackwell workstation card, measured from single GEMMs up to a 500-phrase evaluation. Code and logs are in the fp4-speech repo.

Four findings.

1. FP4 weights are safe everywhere I looked. Quantizing all 515 Whisper linear layers moves WER from 3.37% to 3.36% on 500 LibriSpeech samples. F5-TTS is flat across all 167 layers. Orpheus W4A16 passes the sanity set.
2. FP4 activations are where it breaks, and the failure mode depends on the architecture: the autoregressive model stops generating on some prompts, the diffusion model trades its harmonic structure for a noise floor.
3. Whether FP4 buys speed is a roofline question, and CUDA graphs decide it. The same NVFP4 checkpoint runs at 82 tok/s without graphs and 300 with them.
4. At scale, quantization noise behaves like a regularizer. Over 500 phrases, NVFP4 generated successfully more often than BF16 itself — at measurably higher WER. A trade-off, not a ranking.

Why speech needs its own study

Text-LLM results do not transfer for four reasons. The output is a continuous waveform, so quantization error becomes audible distortion rather than a wrong word. The architecture zoo is wider: autoregressive codec LMs, flow-matching diffusion transformers, encoder-decoder sequence models. The workloads span both roofline regimes — AR decode at batch 1 is memory-bound, a DiT running 32 denoising passes is compute-bound, and FP4 helps these differently. And errors propagate differently: an AR model carries every mistake forward through hundreds of decode steps, while a diffusion model starts each denoising step fresh.

NVFP4, and the two ways it fails

NVFP4 stores each value as E2M1 — one sign, two exponent, one mantissa bit. Sixteen code points:

$$q \in \{0,\ \pm 0.5,\ \pm 1,\ \pm 1.5,\ \pm 2,\ \pm 3,\ \pm 4,\ \pm 6\}$$

Every 16 consecutive values share one FP8 (E4M3) scale factor, chosen so the block maximum lands on the largest code point:

$$s = \mathrm{cast}_{\text{E4M3}}\!\left(\frac{\max_i |x_i|}{6}\right), \qquad \hat{x}_i = s \cdot q_i$$

That is the whole format (NVIDIA’s spec: 16-value micro-blocks with E4M3 scales are what distinguish NVFP4 from MXFP4’s 32-value power-of-two blocks). Two failure modes follow directly from it.

Outliers crush their neighbors. The smallest nonzero magnitude a block can represent is $0.5\,s = \max|x|/12$; anything below $\max|x|/24$ rounds to zero. One outlier of magnitude 1000 in a block of 16 zeroes the other fifteen values — I verified this directly in my fakequant implementation before touching any model. Speech keeps important information in exactly those small values: high-frequency detail, timing, prosody.

The scale itself can overflow. E4M3 tops out at 448, so a block maximum above $6 \times 448 = 2688$ produces a scale the scale type cannot represent. In a naive single-level implementation that is a NaN; hardware paths add a per-tensor FP32 scale and saturate instead (Transformer Engine’s recipe), which repositions the problem without removing it. Keep the number 2688 in mind — it comes back in the Orpheus autopsy.

Setup

One machine, pinned end to end:

• GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q (SM120, 96 GB GDDR7, 1.79 TB/s, 300 W board power, cap range 250-325 W)
• Stack: CUDA 13.0, driver 580.159.03, PyTorch 2.11.0+cu130, Triton 3.6.0, vLLM 0.22.1, llmcompressor 0.11.0, FlashInfer 0.6.11
• Models: Orpheus-3B (Llama backbone, 28 layers, SNAC codec tokens), F5-TTS 336M (mel DiT + Vocos vocoder), Whisper large-v3
• Quantization paths: my own NVFP4 fakequant for sensitivity maps, llmcompressor oneshot checkpoints for Orpheus (NVFP4 and W4A16), a drop-in FP4Linear over torch._scaled_mm for the F5-TTS W4A4 path, Triton tl.dot_scaled kernels for the gaps
• Metrics: WER (Whisper large-v3 transcription, text-normalized), generation success rate (fraction of prompts producing non-empty audio), RTF, tok/s, tok/J, spectrograms

Do the FP4 tensor cores even engage?

First question on a young stack: is the hardware path real? BF16 vs NVFP4 GEMM through cuBLAS, speech-relevant shapes:

M	N	K	BF16 TFLOPS	NVFP4 TFLOPS	ratio
4096	4096	4096	301.6	1028.7	3.4x
2048	5120	1280	267.8	761.9	2.8x
1024	4096	1280	228.4	527.6	2.3x
1024	1024	1024	149.8	209.7	1.4x
512	1024	1024	101.1	105.8	1.0x
256	1024	1024	43.8	52.8	1.2x
128	5120	1280	91.2	136.5	1.5x
128	1024	1024	19.1	26.4	1.4x

The cores engage: 3.4x on the large square case. For calibration, the datasheet quotes 3,511 AI TOPS — FP4 with sparsity, so about 1,755 dense — and NVIDIA’s spec-sheet arithmetic puts dense FP4 at roughly 8x dense BF16. Measured reality on real shapes: 2.8x on the F5-TTS FFN shape, 1.0-1.5x on decode-like shapes. The benefit is shape-dependent; large-M compute-bound work sees most of it.

Two constraints surfaced that appear in no documentation: cuBLAS rejects NVFP4 GEMMs with M < 128 outright — exactly where batch-1 speech decode lives — and M must additionally be padded to a multiple of 128. I filed the first as vLLM #48491.

Weights are safe

Fakequant every linear layer’s weights to NVFP4, leave activations alone, measure. Whisper large-v3 on 500 LibriSpeech test-clean samples:

component	layers	WER
baseline BF16	0	3.37%
encoder all	193	3.21%
decoder all	322	3.28%
encoder self-attention	128	3.40%
encoder MLP	64	3.39%
decoder self-attention	128	3.37%
decoder cross-attention	128	3.36%
all weights	515	3.36%

Every difference is inside the noise. F5-TTS is the same story — 4.7% WER in every cell of the component grid, at both 32 and 16 denoising steps:

component	layers	WER @ NFE 32	WER @ NFE 16
baseline	0	4.7%	4.7%
DiT attention	88	4.7%	4.7%
DiT MLP	44	4.7%	4.7%
DiT all	154	4.7%	4.7%
text embedding	9	4.7%	4.7%
whole model	167	4.7%	4.7%

Orpheus agrees from the deployment side: the W4A16 Marlin checkpoint generates 10/10 sanity phrases at 193 tok/s, 1.27x over BF16. Trained weight distributions are roughly Gaussian with moderate dynamic range; a per-16 FP8 scale tracks them well, and the coarse E2M1 palette is enough once the scale is right. That held for all three architectures.

Activations are where it breaks

Quantize activations too — hardware W4A4 — and the two TTS architectures fail in opposite ways.

Orpheus stops talking. The first NVFP4 checkpoint (10 calibration prompts, eager mode) produced audio for 7 of 10 sanity phrases; the failures emit a stop token within the first 0-2 tokens. Recalibrating with 50 prompts and enabling CUDA graphs did not fix it — 6/10. This is not a calibration artifact. A layer-group probe says why:

activations quantized	logit MSE	top-1 token kept
layers 0-3	0.04	yes
layers 4-7	0.04	yes
layers 8-11	0.27	yes
layers 12-15	1.23	yes
layers 16-19	0.87	yes
layers 20-27	0.44	yes
all 28 at once	0.57	no

No single group is the culprit; only the whole stack flips the next token. And mid-network activations reach $|3616|$ — the induced block scale, $3616/6 = 603$, is past the E4M3 maximum of 448. The overflow threshold from the format section is not hypothetical in this model.

F5-TTS keeps talking through the noise. Every phrase produces audio, but quantization noise floods the spectrum. The spectrograms show it directly: the clean formant bands of BF16 become uniform noise texture, silent regions pick up a noise floor, and the damage concentrates at high frequencies where the original energy is low.

F5-TTS BF16, phrase 00

F5-TTS W4A4, phrase 00

One mechanism, two symptoms. Block scaling zeroes small values next to outliers and overflows on extreme ones. In the AR model those per-step errors feed every later step; over 28 layers times hundreds of decode steps the output distribution drifts until a premature stop token fires. In the DiT nothing compounds across denoising steps — each starts fresh — but the spectral detail lost inside a step never comes back, so the damage shows up as a persistent noise floor instead of a crash. Consistent with that, F5-TTS under FP4 scores the same WER at 32 steps and 16: the noise does not accumulate. The opposite of the AR case.

What FP4 actually buys

Orpheus-3B across regimes, 10-phrase sanity set, per-phrase means:

regime	mode	tok/s	RTF	VRAM	vs BF16
BF16	eager	152	0.57	6.2 GB	1.0x
W4A16 Marlin	eager	193	0.46	2.31 GB	1.27x
NVFP4	eager	82	1.07	2.44 GB	0.54x
NVFP4	CUDA graphs	300	0.29	2.44 GB	2.0x

CUDA graphs are not optional for NVFP4. Without them, per-kernel launch overhead dominates and the quantized model is slower than BF16 — 82 against 152 tok/s, a 3.7x swing on the same checkpoint. With them, the memory-bandwidth advantage finally shows: 2x throughput in 2.5x less VRAM. Small-M GEMM math never got faster (see the microbenchmarks); the win is memory traffic plus launch amortization.

The eager rows above understate the un-quantized baselines, so here is the uniform comparison: in the 500-phrase run below, all three regimes ran the same default CUDA-graph mode and averaged 169 / 324 / 309 tok/s (BF16 / W4A16 / NVFP4). NVFP4’s at-scale lead over BF16 is 1.8x, and W4A16 Marlin is actually the raw-throughput winner at scale — while losing both quality axes, as the next section shows.

F5-TTS is the compute-bound counterexample. Weight-only quantization moves nothing, and the naive W4A4 path is much slower than what it replaces — the overhead is Python-level activation quantization, not the FP4 GEMM:

regime	RTF (phrases 0-4)	output
BF16, NFE 32	0.11	clean
W4A16	0.13	clean, no speedup — compute-bound
W4A4, naive	0.68	noise floor, 6x slower

The cheaper win for diffusion TTS is upstream of quantization: halving NFE from 32 to 16 halves inference time (RTF 0.091 to 0.056) with no WER change — and it still holds with all 167 layers in FP4. The two optimizations do not interact; stacking them stacks the savings without stacking the damage.

Energy: the free 10%

Orpheus NVFP4 under four power caps, measured cap verified per point:

cap	tok/s	avg power	tok/J
250 W	316.1	231.3 W	1.367
275 W	316.8	240.4 W	1.318
300 W	317.7	245.8 W	1.292
325 W	318.1	257.0 W	1.238

Throughput is flat across the whole range — under 1% — while draw climbs from 231 to 257 W. The workload is not power-limited; the extra wattage becomes heat. The minimum cap is +10.4% tokens per joule at under 1% speed cost. (An earlier automated sweep applied caps without verifying they took effect and produced non-monotonic garbage; it was discarded and redone with per-point verification. Locks and caps: always read back what you set.)

500 phrases: a trade-off, not a ranking

Sanity sets flatter everyone. All three Orpheus regimes generated the same 500 LibriSpeech transcripts; Whisper large-v3 transcribed the results.

regime	successful	success rate	WER (on successful)	avg tok/s
BF16	235/500	47.0%	41.5%	169
W4A16	208/500	41.6%	50.4%	324
NVFP4	258/500	51.6%	47.2%	309

Three readings. First, absolute numbers are poor everywhere, BF16 included — Orpheus is trained on short conversational text and LibriSpeech transcripts are long literary sentences. The cross-regime comparison on identical inputs is still valid; the absolute numbers are not. Second, NVFP4 has the highest success rate of any regime, above BF16 itself — 258 against 235. That matches what I first noticed by ear: FP4 noise suppresses the repetition and vowel-elongation loops that derail autoregressive decoding, behaving like implicit noise injection. Third, the stability is not free — NVFP4 WER sits 5.7 points above BF16, and W4A16, the “safe” pick from the weight-sensitivity maps, unexpectedly loses on both axes at once.

So: BF16 for per-utterance accuracy, NVFP4 for generation stability and throughput, W4A16 dominated on this benchmark — but validate on in-domain text before believing any of it for a product. The regularizer effect is a hypothesis with one strong data point; mapping noise magnitude against stability against accuracy is the obvious follow-up.

Two Triton kernels for the SM120 gaps

Triton 3.6.0’s tl.dot_scaled with e2m1 operands compiles and runs correctly on SM120, which made two gap-fillers practical.

A fused-quantization W4A16 GEMM — BF16 activations, pre-packed FP4 weights, quantization inside the kernel — runs the F5-TTS FFN shape (2048x5120x1280) in 0.201 ms against 0.098 ms for cuBLAS BF16 and 0.037 ms for cuBLAS NVFP4. Half of cuBLAS BF16, but 3.5x faster than the Python activation-quantization path it replaces, which was the actual bottleneck in the F5 W4A4 numbers above.

A small-M kernel covers M < 128, the regime cuBLAS refuses. It runs correctly at M = 1 through 64 at 0.5-0.7x cuBLAS BF16 speed — and that is the expected result, not a failure: at M = 1 the launch overhead (~10 us) dwarfs the arithmetic (~100 ns), so the number format is irrelevant to latency. At small M, FP4 buys memory capacity, not speed; speed comes from launch amortization, which is CUDA graphs.

Ten SM120 findings the documentation does not mention

Collected while everything above kept hitting walls. Two are now upstream issues with minimal reproducers.

1. System toolkits older than CUDA 12.8 do not know compute_120; every from-source build fails until the toolkit is upgraded.
2. FlashInfer’s check_cuda_arch() compared compute capability as strings, so “12” < “75” and SM 12.0 failed the “sm75 or higher” check. Filed as FlashInfer #3945, since fixed.
3. cuBLAS NVFP4 rejects M < 128, so batch-1 decode permanently falls back to Marlin. Filed as vLLM #48491.
4. vLLM V1’s engine child process can die silently; the parent reports only EngineDeadError. Workaround: VLLM_USE_FLASHINFER_SAMPLER=0, eager mode, and the direct LLM API while bisecting.
5. A crashed EngineCore can linger as a zombie holding ~90 GB of GPU memory; only a manual pkill frees it.
6. “SM 12.x requires CUDA >= 12.9” warnings at startup are non-blocking noise on a working CUDA 13 stack.
7. VLLM_USE_V1 and VLLM_MAX_MODEL_LEN are silently unrecognized by vLLM 0.22.1.
8. The Orpheus package’s import surface is not what its name suggests: module orpheus_tts, class OrpheusModel, and generate_speech yields PCM int16 byte chunks.
9. FlashInfer’s JIT needs CCCL headers compatible with the system CUDA, not the pip-installed one; a system CUDA 13 install resolved it.
10. enforce_eager masks NVFP4’s real performance (82 vs 300 tok/s). Any FP4 benchmark without CUDA graphs is measuring launch overhead.

Limitations

Perceptual evaluation rests on automatic metrics and spectrograms; I skipped a formal CMOS test on purpose, because with ~50% success rates on out-of-domain text the listening sample would be biased toward phrases that happened to generate, and a small listener panel cannot resolve effects of this size. In-domain CMOS is future work. Absolute Orpheus quality on LibriSpeech text is poor in every regime; only the deltas are interpretable. One GPU model was tested — datacenter Blackwell may dispatch cuBLAS differently and has different power behavior. Whisper never ran on the hardware NVFP4 path: llmcompressor’s oneshot calibration crashes on the encoder-decoder layout (it materializes an empty calibration set and the sampler refuses), so every Whisper number here is fakequant sensitivity, not hardware FP4 — a toolchain gap, not a model property. CosyVoice2, the hybrid LLM-plus-flow architecture, was not evaluated.

Related work, or the lack of it

Low-bit speech inference exists as INT4: quantization-aware training for Conformer ASR, INT4 LSTM ASR, INT4 Whisper on CPU runtimes, and INT4/FP8 deployment stacks for production TTS. Hardware FP4 work — NVFP4 and MXFP4 both — clusters entirely around text LLMs, with a side branch into image diffusion. A documented sweep (2026-07-31: arXiv, Interspeech/ICASSP indexes, NeurIPS/ICML efficiency workshops, NVIDIA/TensorRT-LLM/vLLM trackers; the query log ships alongside this article’s source) found no published account of hardware FP4 inference for speech models. To my knowledge this is the first published measurement of speech on hardware NVFP4 — a claim I will retract the day someone shows me prior art.

Reproduction

Everything pinned above, plus: quantization recipes are committed next to the checkpoint metadata (QuantizationModifier, targets: [Linear], ignore: [lm_head], scheme NVFP4 or W4A16); the 500-phrase evaluation uses temperature 0.6, top-p 0.9, max_tokens 1200, repetition penalty 1.1, one request per generate call, outputs at 24 kHz, WER via jiwer on normalized text. Scripts: bench/ (GEMM, fakequant, F5 paths, Triton kernels, spectrograms), orpheus/ (baselines, quantization, eval_500.py), whisper/ (sensitivity), env/ (install logs and smoke test) in the repo. Model weights and generated audio are not committed — weights are gitignored, audio is reproducible from the scripts. The LibriSpeech-500 evaluation set ships in the repo as an Arrow shard so the exact sample selection is fixed.