Cloudflare Revolutionizes AI: Making Kimi & GLM Models 41% Faster and 30% Cheaper
How Cloudflare's breakthrough optimization techniques are transforming large language model deployment
The AI infrastructure landscape just experienced a seismic shift. Cloudflare has successfully optimized two of the most demanding open-source AI models—Moonshot AI's Kimi K2.6 and Z.ai's GLM 5.2—achieving remarkable performance gains that are reshaping how we think about serving large language models at scale.
The Challenge: Memory-Hungry Giants
Kimi and GLM represent the cutting edge of AI capabilities. These massive mixture-of-experts (MoE) models boast context windows exceeding 262,000 tokens and deliver exceptional performance across multiple languages and complex tasks. However, their size creates a critical bottleneck: GPU memory constraints.
Unlike smaller models, the primary memory consumer isn't the model weights themselves—it's the KV (key-value) cache that stores attention data for previously processed tokens. For long-context conversations, this cache grows exponentially, quickly exhausting available GPU memory.
The Solution: Three Revolutionary Techniques
Cloudflare's engineering team, working with the open-source SGLang inference framework, implemented three game-changing optimizations:
KV Cache Quantization
Compressing the KV cache from BF16 to FP8 doubled context capacity from 686,000 tokens to 1.37 million tokens.
+41% throughput · −30% cost per token
Model Weight Compression
Advanced quantization on model weights further reduces memory footprint without sacrificing accuracy, so more requests share the same GPU hardware efficiently.
Cache Protection
With more requests packed onto shared hardware, Cloudflare added robust safeguards so cached data stays reliable at scale.
Throughput: BF16 vs FP8 KV Cache
On Kimi K2.6 decoding, FP8 keeps scaling where BF16 runs out of memory:
| Concurrent requests | BF16 (tok/s) | FP8 (tok/s) |
|---|---|---|
| 32 | 1,558 (limit) | 1,489 |
| 64 | Out of memory | 2,192 |
Result: 41% higher throughput and 30% lower cost per token at peak FP8 concurrency.
Real-World Impact
These optimizations are now live in production on Cloudflare's Workers AI platform, where developers worldwide can access Kimi K2.6, Kimi K2.7 Code, and GLM 5.2 models. The improvements mean:
Lower costs
AI application developers pay less per token in production.
Higher throughput
Production workloads sustain more concurrent requests per GPU.
Zero accuracy loss
Benchmarks stay indistinguishable from unoptimized versions.
Faster responses
End users get lower latency on long-context conversations.
Why This Matters for Chinese AI Models
This breakthrough is particularly significant for Chinese AI models entering the global market. Kimi and GLM have demonstrated exceptional capabilities in multilingual understanding, tool calling, and reasoning tasks—but deployment costs have been prohibitive.
Cloudflare's optimization effectively "lowers the barrier" for deploying these sophisticated models internationally, making them economically viable for a broader range of applications. As one observer noted, this represents a "dimensional reduction attack" on deployment challenges for Chinese models going global.
The Technical Edge
What sets Cloudflare's approach apart is the strategic application of different optimizations at different stages:
Prefill phase
Maintains BF16 precision where compute is the bottleneck.
Decode phase
Applies FP8 quantization where memory is constrained.
Result: optimal performance at each inference stage.
Looking Forward
With GLM 5.2 and Kimi K2.7 Code now available on Workers AI, developers have access to frontier-scale models with 262,144 token context windows, function calling, vision capabilities, and advanced reasoning—all optimized for production deployment.
This isn't just an incremental improvement; it's a fundamental rethinking of how we serve large language models efficiently. As AI models continue growing in size and capability, innovations like Cloudflare's will be essential for making advanced AI accessible and affordable.
The message is clear: The future of AI isn't just about building bigger models—it's about serving them smarter.
