AI and agents are driving unprecedented computing demands, and Google is looking at the future to keep its data centers ready.
Google’s AI offerings span its internal and cloud offerings. Its data centers are processing seven times more AI tokens compared to last year. To keep up, Google is upgrading its data-center hardware and software technologies at a faster clip. It plans to raise $80 billion to build new data centers. (See related story: Google transforms its data center architecture for agent era)
Network World spoke with Mark Lohmeyer, vice president and general manager of AI and computing at Google, about how the company’s infrastructure is keeping pace with AI demand.
Network World: What is the primary shift in infrastructure needs?
Mark Lohmeyer: We’ve seen the rise of agents and agentic use cases. Years ago, it was the chat phase: Ask a question, get an answer. Now we’re in the agentic era, where you express your intent, agents spin off multiple sub-agents, working in parallel, preserving state. This is a radical shift in what infrastructure needs to do; make them fast, cost effective, secure, reliable. We’re delivering infrastructure optimized for the age of agents.
NW: What’s the goal of the infrastructure buildout, and what should customers expect regarding costs?
ML: Ultimately, it’s about enabling customers with leading-edge capabilities and models at scale cost-effectively. With agents, inference transactions increase by 50x, 100x versus non-agentic workloads. We’re driving the cost per transaction down exponentially. In our latest platforms, we reduce the cost by almost 2x for the same work. Customers serve twice the number of users at the same cost, directly driving profitability.
NW: How are you addressing energy efficiency?
ML: Energy is a critical resource, and Google has optimized for years. We design data centers and compute [to drive] high PUE (power usage effectiveness). We introduced liquid cooling over five years ago, and these latest systems are all liquid cooled. For agentic workloads, CPUs come to the forefront… orchestrating agents, calling tools, doing evaluation loops in reinforcement learning. Our latest Axion-based CPU platform called N4A has energy efficiency and is significantly better than the prior generation and x86 comparables.
NW: How do you think about token efficiency as you build-out systems?
ML: Performance and efficiency gains are powered by co-design of the model and infrastructure. Gemini is trained on TPUs, primarily served on TPUs with high frontier model capability, in a token and cost-efficient way. This stems from co-design across the full stack.
NW: How do you project what infrastructure will be needed years in advance?
ML: Hardware cycles deliver a new next generation roughly every year, but design cycles are two years or more in advance. We work with DeepMind doing core research, to application teams taking models into production, to billions of users, to our team building infrastructure. We work upstream with DeepMind and application teams to understand what’s coming. Agents weren’t being broadly spoken of externally, but internally we had those insights around what they would need. That shows up in hardware design. We hit the timing right — these platforms are built for agents.
NW: What’s the eighth generation TPU platform?
ML: We deliver new platforms every year, and ones launched years ago are close to 100% utilized because demand for AI-optimized compute is high. The eighth-generation TPU platform is the first delivering two complete systems, from the chip all the way up to the network and storage and software, that are optimized.
TPU-8t is optimized for training, and TPU-8i is optimized for inference. For TPU-8i, we increased SRAM on the chip to 384MB — three times the prior generation — and increased the HBM by 50%.
NW: How are you approaching GPU and TPU compatibility?
ML: People in a single cluster do not commingle GPUs and TPUs. We offer both options based on specific workload needs. We’ve been investing on the TPU side in using software frameworks customers are comfortable with on GPUs and enabling those on TPUs. For example, PyTorch and vLLM. Customers could have a pool of GPUs and TPUs, running vLLM on top of that. Start with a workload on TPUs, but if the TPU pool is fully utilized, spill to GPUs or vice versa. This works because it’s all leveraging the same compatible software layer on top.
NW: How has the orchestration platform changed for agents?
ML: Kubernetes is becoming the orchestration platform of choice for AI. Google is transforming GKE [Google Kubernetes Engine] into an agent-native orchestration solution. When expressing intent to an agent and it spins up multiple sub-agents, compute needs to spin up rapidly — TPUs or GPUs — without long delays, then run and spin back down. We’re optimizing at every layer of the GKE stack: significantly improving node startup time and how rapidly we start and stop containers. Lovable demonstrates this with GKE, spinning up hundreds of sandboxes for live coding sessions on their platform in parallel, paying for infrastructure when needed.
NW: What is the role of the network and storage infrastructure?
ML: The network is critical for AI. This requires creating large-scale clusters of GPUs or TPUs and enabling them to talk to each other in a high-performance way. We created the Virgo network — a collapsed network architecture, non-blocking within a data center, where multiple pods or NVLink72 domains connect together.
In TPU8T, we can connect over a million TPUs together leveraging Virgo, creating large-scale, high-performance, reliable clusters that shrink innovation cycles. Storage is equally critical. In large-scale clusters, something is always failing. The ability to take snapshots and go back to a checkpoint is important.
We’ve introduced Managed Lustre 10T, with 10 terabytes per second of bandwidth, 18 petabytes of storage in single clusters. This is 10 times faster than last year and 20 times faster than competition. We have Rapid Bucket, low-latency storage backed by Google storage systems. Both are impactful in large-scale training environments.
NW: How does KV cache strategy differ between training and inference?
ML: For TPU-8i, we increased SRAM on the chip to 384 megabytes — three times the prior generation — and increased the HBM by 50%. Storing KV cache directly in chip memory allows responding to inference requests much more rapidly and cost-effectively than going to an external system. For inference workloads, storing as much KV cache as possible on-chip is critical.
We’re introducing a dedicated KV cache storage subsystem that works across GPUs and TPUs. As KV caches get larger, being able to fall back to this dedicated subsystem becomes critical. Loading model weights rapidly is important in dynamic inference environments where accelerators switch between models hour by hour.
