The results
| Model | Worker and runtime | Measured result | Workload |
|---|---|---|---|
| DeepSeek-V4-Flash | 3× RTX 5090 · FreeToken | 30.62 end-to-end tok/s ~33.3 decode tok/s | Warmed 255-token probe |
| Qwen3.6 35B-A3B | 3× RTX 5090 · FreeToken | 33.76 end-to-end tok/s ~41.0 decode tok/s | Warmed 256-token probe |
| GLM-5.2 NVFP4 | 4× RTX PRO 6000 Blackwell Max-Q · FreeToken Triton offload | 7.158 end-to-end tok/s 9.235 steady decode tok/s | Warmed 255-token probe; served successfully, below the 12 tok/s qualification target |
| Kimi-K3 0.40B dev checkpoint | 1× RTX 3090 · FreeToken offload | 567.043 serial output tok/s 32/32 requests passed in the first concurrency ladder | Architecture proof only; not comparable to the full 2.8T model |
| Qwen3.6 35B-A3B | 8× RTX 5060 Ti · vLLM tensor parallelism 8 | 1,574 aggregate output tok/s | 64 concurrent requests; two successful trials |
| GPT-OSS 120B | 8× RTX 5060 Ti · vLLM tensor parallelism 8 | 1,451 aggregate output tok/s | 64 concurrent requests; two successful trials |
| Gemma4 31B NVFP4 | 8× RTX 5060 Ti · vLLM tensor parallelism 8 | Did not complete | Out of memory during warm-up at an 8k context window |
Do not turn this table into one ranking
The FreeToken rows include warmed single-request probes on multi-GPU workers and a separate concurrency test on one RTX 3090. The vLLM rows measure aggregate output across a fixed ladder on eight lower-memory GPUs. Those workloads are not interchangeable.
What the Kimi run proved
FreeToken loaded the public eight-layer, 0.40B MXFP4 Kimi-K3 development checkpoint and served it through the OpenAI-compatible API on a 24GB RTX 3090. A 256-token serial request produced 255 reported output tokens in 0.450 seconds, or 567.043 tokens/s. The first ladder completed all 32 simultaneous requests.
This proves the new KDA/MLA, latent-MoE, SiTU, MXFP4 loading, batching, and API path. It says nothing about how fast the full 2.8T Kimi-K3 model will run.
During a sustained follow-up, the live-swapped NVIDIA driver lost the passed-through GPU on the second 16-request sample. We kept the completed samples and recorded the failure. We did not turn the earlier burst peak into a sustained-throughput claim.
Review the FreeToken implementation and the saved benchmark artifacts.
What this changes for routing
Parameter count is not enough. Placement needs the weight format, active memory footprint, context limit, runtime, and expected concurrency.
A cheap community worker can be right for one workload and unable to load another. We keep raw results and failed runs so that decision can be checked later.
Qwen3.8 27B remains a separate candidate for a 24GB RTX 3090. Its fit has been assessed, but it has no TEKIZ.AI benchmark result yet.
Method notes
The vLLM runs used fixed 128-token completions, temperature 0, a fixed prompt seed, tensor parallelism across eight GPUs, and two trials at each concurrency level through 64. FreeToken requests streamed over the OpenAI-compatible API. The Kimi run ignored EOS to exercise the requested token budget. We saved metrics, not generated text or credentials. These are throughput tests, not quality scores or maximum-capacity claims.