Microsoft Azure and University of Texas researchers found that multi-step AI workflows create CPU-GPU bottlenecks that conventional inference infrastructure struggles to handle efficiently.
Enterprises deploying agentic AI may need a new generation of AI servers as conventional GPU-centric infrastructure struggles to efficiently execute multi-step AI workflows, according to researchers from Microsoft Azure and the University of Texas at Austin.
Drawing on production telemetry from Microsoft’s Azure cloud and experiments with representative open-source agent frameworks, the researchers found that AI agents spend far more time coordinating models, tools, and orchestration software than conventional inference systems assume.
Rather than behaving like standalone LLM requests, agentic applications execute as dynamic workflows that repeatedly move work between CPUs, GPUs, and external services, exposing inefficiencies in today’s server designs.
“Our study shows that agentic execution is fundamentally fragmented and heterogeneous,” the researchers wrote in the paper. “Each request expands into a workflow of LLM inferences, tool invocations, and orchestration decisions that repeatedly crosses the CPU-GPU boundary.”
According to the paper, that execution pattern places the CPU on the application’s critical path because orchestration software and tools execute on the host while model inference runs on GPUs.
The researchers said conventional server architectures are poorly matched to those workloads because fragmented execution strands CPU and GPU resources, different host-side software roles have different resource requirements, and multiplexing multiple agents increases coordination overhead.
Production data points to fragmented execution
The researchers said a representative production request alternated between multiple LLM calls, tool discovery, tool execution, and orchestration before completing. In a controlled study using the CORAL framework, a single workload expanded into 580 LLM calls interleaved with 552 tool invocations, causing execution to “ping-pong between the two processors hundreds of times.”
The study also found that host CPU utilization remained low for extended periods before rising sharply during bursts of tool execution, while GPU utilization varied widely depending on workflow composition, leaving some accelerators saturated and others idle.
According to the researchers, the fragmented execution pattern leaves CPUs and GPUs underutilized on average while allowing either processor to become “a transient bottleneck on the workflow’s critical path,” making static resource provisioning inefficient for agentic workloads.
Sanchit Vir Gogia, chief analyst at Greyhound Research, said the findings show enterprises should evaluate agentic AI infrastructure differently from conventional inference deployments.
“Agentic AI is not a bigger chatbot; it is a distributed application with inference embedded inside it,” Gogia said. “The individual ingredients are familiar. The execution graph is new.”
“The GPU remains indispensable, but it no longer owns the entire clock,” he added. “Tool time matched or beat inference time in more than 27 per cent of requests, and average utilisation is beginning to lie to infrastructure teams.”
Researchers propose workflow-aware server design
Based on those findings, the researchers proposed a server architecture, called Agora, that dynamically reallocates CPU and GPU resources, separates scheduling, orchestration, and tool execution into dedicated host roles, and adapts resource allocation to workload behavior.
“Agora dynamically harvests idle CPU cores for co-located throughput work, while protecting agentic tail latency against tool spikes. It also oversubscribes GPU memory by placing more agents on each GPU, prefetching the next agent’s state to hide swap latency,” the researchers wrote in the paper. “To match the machine to the heterogeneous roles, Agora pools cores by role and applies affinity-aware scheduling to restore locality. These techniques substantially improve CPU and GPU utilization and per-server throughput while preserving agent tail latency.”
In their evaluation, the researchers reported that Agora increased host CPU utilization by about 30%, recovered about 95% of a co-located workload’s standalone throughput under low load, freed roughly one-third of GPUs through workload consolidation, increased generation throughput by 82%, and reduced tail latency by 2.5 times.
Gogia said the findings indicate that infrastructure procurement should focus less on individual processors and more on how entire AI workflows execute.
“The CPU is not returning to the throne; the throne itself is disappearing,” he said. “Competitive advantage is moving from the individual processor to the heterogeneous server, rack and runtime operating as one system.” He said organizations should “procure the workflow, not the box,” arguing that workload profiling and scheduling are likely to deliver greater benefits than sizing infrastructure based on model inference alone.
