Google TPUs Hit 700 Tokens Per Second in Meta Data Centers

Google has successfully integrated its custom tensor processing units into Meta’s production AI infrastructure through a new software bridge. This move marks a significant shift in the AI hardware market, as it is the first time Google has sold TPUs to an external customer for large-scale production use. The announcement follows a close collaboration between Meta, Google, and the SGLang team, resulting in a native PyTorch version of the SGLang inference engine. By removing the need for code rewrites, the new stack allows Meta to deploy its existing PyTorch workloads directly onto Google’s hardware without migrating to the JAX framework.

The core of this development is the SGLang-TorchTPU engine, which is now running in production at Meta. Zongwei Zhou, Meta’s PyTorch-TPU lead at startup RadixArk, confirmed that the system is live today. Zhou described the result as a remarkably mature, PyTorch-native serving stack. This achievement stems from a three-way partnership where RadixArk, the creator of SGLang, built the engine alongside Google and Meta. The integration effectively neutralizes the software barrier that previously kept Meta tied to other hardware architectures.

Meta’s strategic interest in diversifying its hardware stack was further emphasized by Jana van Greunen, the company’s director of PyTorch engineering. Greunen stated that Meta cannot be constrained by a single hardware architecture. She noted that the new TorchTPU system enables the company to deploy its PyTorch workloads to Google TPUs seamlessly. Google confirmed that it co-developed this specific software layer with Meta’s PyTorch team. This joint effort highlights a growing trend among major tech firms to avoid dependency on any single chip supplier, particularly as AI compute costs continue to rise.

This is not the first time Meta has experimented with Google’s silicon, but it is the most significant deployment to date. Google describes the older PyTorch/XLA bridge as a predecessor to the new TorchTPU system. Industry reports indicate that Meta research teams used TPUs via the older bridge between 2020 and 2023. However, those initial contracts were eventually canceled. The current push by Google to support PyTorch on TPUs was largely aimed at winning back Meta, which had previously resisted migrating its vast codebase to Google’s JAX framework. The new native support addresses this long-standing friction point.

The commercial terms of this arrangement remain partially opaque, though industry chatter suggests a specific timeline. Reports from last year indicated that talks included renting Google Cloud TPUs in 2026, with direct purchases planned for 2027. Meta is believed to have signed a multi-year rental agreement to secure capacity. While the exact location of these production TPUs is not public, the scale of the deal signals a serious commitment from Meta to diversify its compute resources. This move places direct pressure on Nvidia, which has long held a near-monopoly in the AI training and inference markets.

The timing of this announcement is notable, arriving just one day after Amazon announced it would open-source TorchNeuron. TorchNeuron is a native PyTorch backend for Amazon’s Trainium2 and Trainium3 chips. Both Google and Amazon are making concerted efforts to loosen Nvidia’s software lock-in by providing better support for PyTorch, the dominant framework in the industry. By making their hardware more accessible to existing PyTorch users, both companies aim to capture a larger share of the AI workload market. This competitive dynamic is likely to drive further innovation in inference optimization.

Performance benchmarks provide concrete evidence of the hardware’s potential. A tuned vLLM setup serving Moonshot AI’s Kimi K3 model, which has 2.8 trillion parameters, reportedly exceeds 700 tokens per second for a single user on the seventh-generation Ironwood TPU. This test was conducted by Inferact, a startup founded by the creators of vLLM. The setup utilized 16 Ironwood chips with speculative decoding, a technique that uses a smaller draft model to speed up inference. In comparison, a similar setup on 16 Nvidia GB200 chips reached 452 tokens per second.

Beyond raw speed, the new software stack offers efficiency gains at smaller batch sizes. Kernels for the Kimi K3 and Qwen 3.8 27B models deliver roughly 1.4 to 2 times the throughput of the GB200 baseline when running without speculative decoding. This improvement is crucial for applications that require lower latency or operate with smaller concurrent user groups. The ability to match or exceed Nvidia’s performance while offering a different architectural approach gives Meta significant leverage in future hardware negotiations.

Looking ahead, the SGLang-TorchTPU project plans to expand its model support. The next additions will include DeepSeek V4.1 Flash and Reflection Beam. RadixArk is also working toward Day 0 TPU support for new open models, meaning the hardware will support new releases immediately upon their launch. This target is set for the first quarter of 2027. Google, Meta, and RadixArk are scheduled to share more details at the SGLang Summit 2026, taking place on November 12 and 13 in San Francisco. This upcoming event will likely serve as a key showcase for the maturation of the multi-vendor AI hardware landscape.