When an AI workload is not performing as expected, the obvious place to look is the GPU. Do we have enough compute capacity? Would adding more GPUs solve the problem?
Not necessarily.
The GPUs may be perfectly capable of handling the workload, but if the network is not able to move data quickly and consistently, adding more compute could result in more expense without addressing the real bottleneck.
This is something I think we risk losing sight of in the rush to invest in AI. GPUs cannot do their job alone. They rely on huge amounts of data moving quickly between compute, storage and applications, which means the network has a big part to play in how well that investment actually performs.
For CIOs, this means looking beyond how much compute they have and asking whether the network and wider infrastructure can keep pace.
AI is changing what the network needs to do
Enterprise networks have always had to move data reliably between systems. The difference with AI is the volume, speed and pattern of that traffic. Large AI workloads are distributed across multiple GPUs because the processing requirements are simply too great for one processor to handle efficiently. Those GPUs need to communicate constantly, exchanging data and synchronizing their work. Training is particularly demanding because GPUs may need to exchange parameters and gradients throughout the process, creating large volumes of traffic. As the number of GPUs increases, so does the importance of the connections between them.
This changes the relationship between compute and the network. Adding more GPUs increases processing capacity, but it can also increase the amount of data that needs to move between systems. If the network can’t keep pace, it can become the bottleneck that prevents that additional compute from delivering the expected performance.
Organizations do not need thousands of GPUs before this starts to matter. Even relatively modest enterprise deployments can involve large datasets moving between storage and compute, multiple accelerators working together and applications accessing models in real time.
The important point is that AI readiness cannot be measured simply by counting GPUs. You can continue adding compute, but there comes a point where the network determines how effectively that compute can be used. If data cannot reach the GPUs quickly enough, or communication between them is slowed by congestion or latency, expensive resources can spend time waiting rather than processing.
At that point, the organization doesn’t have a compute shortage — it has an infrastructure problem.
More bandwidth does not solve every problem
The obvious response might be to add more network capacity. While bandwidth certainly matters, increasing it does not remove every potential constraint. AI workloads can be particularly sensitive to latency, congestion, packet loss and inconsistent performance. Cisco notes that AI pipelines create different traffic patterns across training and inference, with training requiring both high bandwidth and low latency as GPUs continuously exchange information. The infrastructure therefore needs to manage not just capacity, but how traffic moves through the environment.
That distinction matters.
Imagine hundreds of GPUs working together on the same task. Their value comes partly from their ability to operate in parallel, so if one part of the network slows communication between them, the effect can extend beyond that individual connection. GPUs may be left waiting for other parts of the workload to catch up. The fastest component in the environment is not much use if it is constantly waiting for the slowest.
Storage adds another dimension. AI depends on data, which means moving large datasets from storage systems into compute environments and, depending on the workload, moving information back again. A network that performs well for GPU-to-GPU communication could still encounter problems elsewhere in that journey.
AI infrastructure does not usually exist in isolation, either. Enterprises are introducing GPU clusters into data centers that already support conventional applications, storage platforms, cloud connectivity and other business-critical systems. AI traffic therefore becomes another part of an increasingly complex infrastructure estate.
This is where network design starts to have a direct bearing on the economics of AI.
If an organization has invested heavily in accelerated compute, it wants those resources doing useful work for as much of the time as possible. A network bottleneck that leaves GPUs underutilized is not simply an IT performance issue — it means the business is getting less value from some of its most expensive infrastructure.
You cannot fix what you cannot see
Another challenge is knowing where the problem actually is. Infrastructure teams can see whether devices are available, track bandwidth utilization and receive alerts when something fails. That remains important, but increasingly complex AI environments demand a broader view.
A switch can be working. A server can be healthy. The GPUs can be available. Storage can be online. Yet the workload can still be performing below expectations. That’s because understanding individual components does not necessarily tell you how well the overall system is working. This is where the distinction between monitoring and observability becomes useful.
Traditional monitoring generally tells teams whether infrastructure components are operating within expected parameters. Observability goes further, bringing together information from across the environment to help teams understand why a system is behaving in a particular way. IBM describes network observability as providing real-time visibility into the performance, behavior and health of the network by drawing on data from across the infrastructure, rather than focusing solely on predefined metrics and reactive troubleshooting. For AI environments, that wider view is increasingly important.
If a training job slows down, teams need to be able to establish whether the cause is compute, storage, congestion on a network path, packet loss, a degraded link or something else entirely. Looking at each device separately can make that difficult, particularly when the issue is not a complete failure but a gradual deterioration in performance. Those are often the harder problems to find. A component that fails completely usually generates an alert, but a connection that still works yet is performing badly can be much less obvious. The infrastructure is technically available, but the workload is taking longer, GPUs are being used less efficiently and costs are creeping up. End-to-end visibility gives infrastructure teams a better chance of seeing those relationships.
AI readiness is an infrastructure question
As AI moves deeper into business operations, the consequences of infrastructure performance will become harder to ignore. A bottleneck during an experiment might delay a development team, but the same problem affecting a production AI service could have consequences for employees, customers or other systems that depend on it.
This means the network needs to be considered as part of the AI investment from the outset, rather than something to revisit when performance problems appear.
That does not necessarily mean replacing the existing network or building an entirely separate environment for AI. In many cases, the infrastructure already in place may be perfectly capable of supporting the workloads being introduced. The important thing is knowing where its limits are. That requires an understanding of what those workloads actually need, where genuine constraints exist and whether IT teams have enough visibility across compute, networking and storage to spot problems as they emerge.
Ultimately, it is not how much compute an organization has that matters, but how well it can use it. If CIOs are investing heavily in GPUs, they need to make sure the infrastructure around them is not holding them back.