Meet FreeToken: An Edge-Native MoE Serving Engine that Runs 753B GLM-5.2 on a Single Workstation GPU

Frontier open-weight models are shipping faster than the hardware assumptions around them. Kimi-K3, GLM-5.2 and DeepSeek-V4-Flash are closing the capability gap with proprietary systems, but releasing parameters only determines who can obtain a model — not who can afford to run it. Serving them still assumes datacenter-class GPU clusters, and as agentic workloads push inference demand up, that cost lands hardest on individual developers and small teams. Meanwhile, more than a hundred million consumer machines already carry discrete GPUs. A team of researchers from UC Berkeley and UT Austin propose FreeToken. The research team argued the missing piece is not hardware but a serving system: it treats a personal machine as a unified, elastic inference platform rather than a small GPU, and continuously maps computation and model state onto whatever GPU, CPU, memory and interconnect bandwidth the machine actually has. The result is a 35B model at interactive speed on an 8 GB laptop GPU, 284B on a gaming desktop, and the 753B GLM-5.2 on a single workstation card.

Is it deployable?

Yes, FreeToken is Apache-2.0 on GitHub, published on PyPI as freetoken v0.1.2 (uv pip install “freetoken[accel]”), and shipped as a one-click desktop app for Windows and Linux at flashml.ai. The CLI targets Linux x86_64 with an NVIDIA GPU on driver r580+ (CUDA 13). ft serve exposes OpenAI- and Anthropic-compatible endpoints on port 1919, and ft launch claude wires up Claude Code, Codex, OpenCode or OpenClaw against your own box.

Who it fits: solo developers, startups and SMB engineering teams whose agent token bills already exceed the cost of a GPU they own; enterprises should treat it as an air-gapped or regulated-workload path, not a datacenter replacement. Strongest industry fit: healthcare and legal (data never leaves the machine), defense, finance, and IP-heavy R&D. Typical applications: local coding agents, private code review, offline contract analysis, synthetic-data generation, batch evals.

The gap it targets

Mixture-of-Experts makes local frontier inference arithmetically feasible. DeepSeek-V4-Flash activates 6 of 256 routed experts in each of 43 layers, so only 13B of its 284B parameters participate in any single token. Sparsity does not shrink the expert pool, though — at FP4 the full set is roughly 140 GB, so inactive experts sit in host memory and enter the execution path on demand.

The research team isolates three failure modes in existing engines (llama.cpp, KTransformers, Ollama, MoE-Infinity):

• Prefill destroys sparsity: Thousands of tokens per layer route to nearly the whole expert set, so a prefill pass streams the entire pool across PCIe — about two seconds on an RTX 5090, five on PCIe 4.0 desktops, ten or more on the x8 links common in laptops.
• Static placement misses decode traffic: llama.cpp assigns MoE tensors at load time; KTransformers pins a “hot” subset. Routing shifts every token, so most expert evaluations fall to the CPU while the GPU and the PCIe link sit idle.
• Consumer CPUs cannot carry the remainder: Dual-channel DDR5 delivers 80–90 GB/s against the 1–1.8 TB/s an RTX 4090 or 5090 draws from on-package memory.

Three mechanisms

• Bandwidth-adaptive execution (the q* policy): Because DMA transfers and CPU expert execution read from the same host-memory subsystem, a saturated PCIe link leaves a residual bandwidth of B_H − B_P. FreeToken splits each step’s m cache misses accordingly: q* ≈ m × B_P / B_H experts are filled into the GPU cache, the rest are computed in place on the CPU, and the two partial sums merge exactly — no approximation, no router modification. Both bandwidths are profiled on the deployed machine (ft bench bw), which matters: measured B_P:B_H is 52.7:77.3 on an RTX 5090 server but 11.8:47.5 on a 4060 laptop.
• Semantic-aware caching: During prefill, full-layer double buffering streams layer l+1 while the GPU computes layer l. Recurrent-state checkpoints are anchored at special-token boundaries — thinking blocks, tool calls, tool outputs — precisely where agent harnesses truncate context, so an edit re-prefills only the new suffix. During decode, a shared LRU expert cache spanning all MoE layers follows the router instead of a placement frozen at load time.
• Elastic memory management: At scheduler safe points the GPU expert cache is rebuilt under a revised VRAM budget without restarting the engine or reloading the host pool. Experts are read from disk straight into their final host layout, then pinned; no GPU warmup is required because the first request is served with a cold cache.

Results

On an RTX 5090, FreeToken sustains 77–83 tok/s on Qwen3.6-35B-A3B (BF16) and 22–25 tok/s on DeepSeek-V4-Flash (MXFP4) — 1.5–2.3× the strongest baseline, with decode staying within 12% of the single-turn rate across three agentic workloads. Worst-case TTFT stays below 44 s in every cell; llama.cpp hits 232 s, Ollama 179 s and KTransformers 946 s somewhere in the matrix, past the point where agent clients time out.

At equal cache capacity (37% of the Qwen3.6 pool), the global LRU misses 16% of decode-time expert reads against 41% for KTransformers and 62% for llama.cpp. On an 8 GB RTX 4060 laptop the NVFP4 build serves 35B at 39.3 tok/s — above the 33 tok/s median decode speed measured for Codex in production traces. On a single RTX PRO 6000, GLM-5.2 (753B, 40B active) runs at 14.9 tok/s versus llama.cpp’s 7.3.

Data Check

Key Takeaways

• FreeToken splits MoE cache misses between PCIe fills and CPU execution using measured bandwidths, not a fixed offload rule.
• Expert output stays bit-exact — no router changes, no expert substitution, no precision relaxation.
• 1.5–2.3× decode throughput over llama.cpp, Ollama and KTransformers, with tail TTFT under 44 s.
• 35B at 39.3 tok/s on an 8 GB laptop GPU; 753B GLM-5.2 on one workstation GPU.
• Apache-2.0, on PyPI and as a Windows/Linux desktop app — deployable this afternoon.

Check out the PAPER, GITHUB REPO and PROJECT. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us