Ant Group and the vLLM community have open-sourced Humming, a just-in-time compiled GEMM kernel library for quantized large language model inference. The project is released under Apache 2.0 from Ant Group's inclusionAI organization and mirrored under the vLLM project.
Humming aims at a common problem in quantized serving: every mix of weight bit width, activation format, scaling scheme, dense versus mixture-of-experts layout, and GPU generation tends to need its own specialized kernel. The library converts supported export formats into a shared quantization description, then JIT-compiles kernels from reusable CUDA components for the target shape, device and compute options.
According to the project, one architecture covers integer weights from INT1 to INT8 and floating-point weights from FP4 to FP8, paired with FP16, BF16, FP8, FP4, INT8 or INT4 activations where the formats are compatible. It supports both dense GEMM and MoE GEMM, and targets NVIDIA GPUs from Turing onward. Dependencies are limited to PyTorch and NVCC, and the pip package is described as only about 100 KB.
A China-primary angle is hardware reach beyond NVIDIA. The release says Humming already fully supports T-Head's Zhenwu 810E, a CUDA-compatible AI accelerator, so the same kernel and tuning path can run on that device. T-Head is Alibaba's chip unit; Zhenwu 810E has appeared in earlier Chinese AI serving deployments.
Performance figures in the joint announcement are from the teams' own microbenchmarks and serving tests. At small batch sizes, dense W4A16 is reported at roughly 3 to 4 times cuBLAS on several devices. On B300, W4A8 versus a DeepGEMM baseline is quoted at about 1.6 times in some small-to-medium batch cases. In an end-to-end DeepSeek-V4.1-Flash test on eight H20 GPUs, switching MoE from Marlin W4A16 to Humming W4A8 raised output throughput by about 14% to 43% and cut mean time to first token by about 25% to 32%, with the largest gains at higher concurrency. Those results have not been independently verified.
Starting with vLLM 0.29.0, CUDA installs of vLLM include Humming as a dependency. Operators can select it with --linear-backend humming and --moe-backend humming. Humming is a joint Ant Group and vLLM community project, not an Ant-only product.
SEE ALSO: Ant Group Open-Sources Ming-3.0-tts · T-Head Unveils Zhenwu V900 AI Accelerator · METASTONE Launches TokenFleet