LFM2.5-24B-A2B, quantized to GPTQ

Liquid AI's LFM2.5-24B-A2B is a 24-billion-parameter mixture-of-experts model that activates only about 2 billion parameters per token. That makes it fast, but at bf16 it's around 48 GB, too big for a single 20–24 GB card. This 4-bit GPTQ build fits on one, and it's published on Hugging Face.

What makes this model different

• A hybrid stack. 40 layers: 30 short-convolution layers and 10 full-attention layers. Only the 10 attention layers keep a KV cache, about 20 KiB per token at bf16, so long context stays cheap.
• 64 experts, 4 per token. Most of the weights live in the experts, so that's where 4-bit pays off.
• Not a Llama. Recipes written for dense Llama models get the module layout and the group size wrong for this architecture.

How it was built

1. Research first. The Hermes modeler agent running the quantization kept hitting the same errors, so it first built a source-verified reference from the GPTQModel source and tests, vLLM's docs and the issue tracker. It covered the right API, the LFM2 module tree, group size and checkpoint/resume, plus nine recurring errors and their fixes.
2. Quantize on rented hardware. The run happened on a Hugging Face Space with an A100 80 GB, never on a local machine: GPTQModel 7.6.0, 4-bit weights, group size 32, calibrated on 1,024 C4 samples, with checkpointing so an idle-killed Space resumes from the last layer.
3. Verify the artifact. 13.4 GB on disk. Every one of the 2,432 experts (38 MoE layers × 64) is stored as 4-bit GPTQ, along with the dense FFN, the short-conv projections and attention q/k/v.

Design decisions

• Group size 32, not 128. The experts' projections are small (1,536 wide). Finer groups preserve more of each one, which is the setting GPTQModel tests for LFM2.
• Keep the router precise. The MoE router gates, attention output projections, conv kernels, norms and embeddings stay in bf16. They're a small share of the weights and matter most for routing and stability.
• Native vLLM layout. Experts are written as separate per-expert w1/w3/w2 quantized layers, the layout vLLM's native lfm2_moe loader expects, so no --trust-remote-code is needed.

What's next

• Serving tests. Load it in vLLM on a local 20–24 GB card and measure tokens per second and long-context behaviour.
• 4-bit vs. full precision. Compare it against the bf16 original, separating weight-quantization loss from FP8 KV-cache loss.
• NVFP4. A Blackwell-native 4-bit version with NVIDIA Model Optimizer, which needs a custom expert-only recipe for this architecture.

The model is on Hugging Face: aiprimebuilds/LFM2.5-24B-A2B-GPTQ.