METASTONE, an independent compute operator, has introduced TokenFleet, a distributed scheduling framework for large language model inference. The company announced it during the 2026 CCF National High Performance Computing Conference.
TokenFleet does not make individual models run faster. Instead it targets cluster-level problems: requests piling up on some GPU instances while capacity sits idle elsewhere, long-document jobs crowding out real-time chat, and timeouts or cancellations that burn compute without producing a usable answer. METASTONE frames the goal as raising effective token output, meaning tokens from requests that finish successfully and meet their service targets.
The framework has three parts. TokenPilot picks an instance inside a model pool. It weighs the gain from reusing a cached prompt prefix, which skips repeated prefill computation, against the cost of waiting in a longer queue on that instance. It estimates workload from input size, output budget and reusable prefix, then scores candidates using queue depth, in-flight tokens, KV cache pressure and predicted latency.
FleetMind handles routing across data centers. It tracks which sites actually have the right model version deployed and ready, along with live capacity and network conditions, then sends new requests to another site only when permissions and data-residency rules allow it and the cost comparison favors it.
SLOMax manages mixed workloads in one cluster. It classifies requests by input length, output budget, task type and tenant service level, then admits and queues them in separate pools to protect interactive capacity from long-context and batch jobs.
According to figures from METASTONE's conference slides, predicted-latency routing on a tuned GLM5.3 deployment cut P95 time-to-first-token by 32.4% and P50 by 71.3% under the tested cluster and load. In a separate estimate for SLOMax, total output throughput rose about 1.5%, while on-target output throughput rose about 29.4%, as the share of requests meeting service targets went from about 67% to about 86%. METASTONE describes the second set as a projected example, and notes that lower latency does not directly equal lower cost.
TokenFleet follows the company's Meta-Infer inference engine, which focuses on single-instance efficiency. METASTONE says it operates more than 10 supercomputing and intelligent computing centers. None of the figures have been independently verified.
SEE ALSO: Token Factories Emerge as WAIC 2026 Theme · Zhipu Confirms GLM-5.3-Flash Is Served on 100,000 Domestic Chips