Amid the rise of AI agents and agentic use cases, Google is readying new silicon to support middleware changes and making network and storage improvements to cut training and inference time.
Google’s data center team is racing to turn its infrastructure into a well-oiled machine for AI and the onslaught of agents. At this year’s Google I/O, CEO Sundar Pichai shared startling numbers: Google’s data centers processed about 3.2 quadrillion tokens a month, roughly seven times more than the 480 trillion processed in May 2025.
“Multiple agents work together, and now you’ve got millions, billions of users around the world potentially spinning off agents to help them do things,” said Mark Lohmeyer, vice president and general manager for AI and computing infrastructure at Google.
Google’s new data-center blueprint includes updated hardware, software, and orchestration layers to keep always-running agents operational.
In the LLM era, users sent prompts and received responses, and Google’s infrastructure was designed for latency and throughput. But agents could increase inference transactions by up to 100 times non-agentic workloads, Lohmeyer said. Google’s redesigned AI data-center stack has the elasticity for agents to be widely distributed, run for long periods, and make decisions independently.
“We’re delivering new platforms every year, each one optimized for what we think the world is going to need for the age of agents going forward,” Lohmeyer said.
Efficient data flow is key so agents can act, reason, and decide faster.
Google adjusted the Google Kubernetes Engine into an agent-native environment, where agents could be quickly spun up in sandboxes and containers. “From an infrastructure perspective, you need to spin up a bunch of TPUs or GPUs very rapidly. Then you need to be able to run them and spin them back down,” Lohmeyer said.
Google also made drastic improvements to its silicon to support its middleware changes. It recently introduced new AI chips, with the TPU-8t for training, and TPU-8i for inference. The 8t chip has three times more computing power than the previous-generation Ironwood chip. The 8i chip has 384 megabytes of SRAM and 288GB of HBM3e memory, which is 50% more than the previous-generation chip.
The platform is optimized for KV cache (key-value cache), which stores important contextual information needed by agents to make decisions, which reduces the round trips to other memory and storage systems. “Being able to store more of the KV cache directly on the chip allows you to respond much more rapidly and cost-effectively,” Lohmeyer said.
A new CPU called Axion N4A is more power efficient at agentic workloads such as orchestration and tool calling, Lohmeyer said.
Google also made many network and storage improvements to cut training and inference time. A new technology called TPUDirect can move data from storage directly into the memory of the TPU quickly by bypassing any orchestration overhead, Lohmeyer said.
A networking technology called Virgo can coordinate 1 million TPUs across a widely distributed network. It can also link up GPUs such as Nvidia’s latest CPU-GPU package called Vera Rubin. “In the case of Vera Rubin, we’ll be able to connect up to 960,000 GPUs leveraging Virgo,” Lohmeyer said.
A new technology called Pathways is a distributed training framework that efficiently scales machine learning across millions of TPUs and GPUs. Pathways solves bottleneck issues typically associated with JAX, and both help coordinate across wide networks.
“The software to orchestrate these large-scale distributed training jobs is also just as important as the hardware that it runs on top of,” Lohmeyer said.
Weighing Google’s AI data-center stack
Google is the only provider with its own data centers, software, hardware and models, said Jack Gold, principal analyst at J. Gold Associates. Google can optimize each on a regular cadence, which “many data centers can’t easily afford given the high cost of new chips,” Gold said.
Google’s stack may not be best for every data center need compared to Nvidia’s general-purpose GPUs, CPUs, and networking. AWS and Microsoft are also creating their chips.
“There is no real risk of Nvidia being replaced by Google in a big way. But with an ever-expanding market, there is plenty of room for all players,” Gold said.
But Logan Wolfe, partner at Kyndryl’s global AI strategy and sovereign transformation, advised enterprises to adopt a multi-cloud strategy to reduce risk from system failures, however superior an infrastructure may be. “I think that kind of hybrid and liquid infrastructure, we’re definitely getting there,” Wolfe said.
The cost per token varies depending on the provider of inference, whether that’s Microsoft, Google, OpenAI or Anthropic. That will matter as AI moves from experimentation to a powerful tool that drives business changes.
“Ultimately it really comes down to how much money are we spending on AI to move a certain business outcome,” Wolfe said.
