AI Datacenter Network Architecture | Why the Fastest GPUs Are Not Enough: The Defining Role of Network Infrastructure in AI Workloads

Author: Burak Salihoğlu, Network & Security Engineer – Sekom

Today, the diversity and widespread use of Artificial Intelligence (AI) models has reached levels that can no longer be underestimated. As users, we always see the front-end interfaces of these AI services that touch almost every aspect of our lives. Yet the real workload runs in the background on a massive network infrastructure where thousands of GPUs communicate with each other. In this article, we will take a close look at the AI Network architecture that determines how fast an AI model will be trained or respond.

We can evaluate AI Network architecture under two main headings.

Frontend is the part that is visible above water and accessible to everyone. Backend is the computation and data management infrastructure that keeps the entire system functional.

Frontend Network

Frontend is the layer of AI infrastructure that opens to the outside world and communicates directly with users. The main purpose of this network is to handle user requests — that is, inference requests — to pull datasets into the system from external sources, and at the same time to provide the necessary communication for monitoring and maintenance processes. It generally operates over standard Ethernet protocols, TCP/IP. Although latency is an important factor at this layer, it is not as critical as the backend side; here the main priority is security, accessibility, and healthy communication of the system with the outside world.

Further Reading: Visibility in Modern Data Centers: Discover Cisco MDS SAN Analytics

For example; a user sends a request from a web browser to an AI application (such as a chatbot). This request first goes to the application servers via the frontend network. The prompt typed by the user is received here and forwarded to the backend for processing. ChatGPT and Gemini Chatbot, which are heavily used today, can be given as examples.

Backend Network

The infrastructure where the AI model, LLM (Large Language Model), is trained, data management is provided, and computation operations are performed is called the backend network. Within this structure, billions of parameters must be synchronized between GPUs during model training, and lossless data transmission technologies such as RDMA (RoCE v2) or InfiniBand are generally used in this process. Even a 1-millisecond delay or a single packet loss that may occur in this infrastructure can cause GPUs to remain idle and the training time (JCT) to be significantly extended.

Sometimes it even leads to loss of time due to the termination of the entire workload, and the energy used is wasted. To give a brief example; this could be a task that takes weeks being interrupted on its last day due to a problem in the AI network infrastructure.

Our main focus in this article will be the backend network side.

Lossless Architecture and Traffic Management in Artificial Intelligence Networks

As artificial intelligence (AI) and Machine Learning (ML) models grow larger, infrastructure designs are generally determined only based on the number of GPUs. In fact, even if you have the world’s fastest GPUs, if your network is not capable of managing this traffic, a large portion of your hardware investment will be wasted as idle time.

So what distinguishes AI workloads from traditional data center traffic, and how is a modern AI Network architecture built?

1 – Transitioning from Standard Ethernet to “AI Fabric”

Classic network architectures can operate in a “lossy” structure; this situation is tolerated by the TCP protocol through a retransmission mechanism. However, in artificial intelligence workloads, this approach creates a serious problem, and a single packet loss means thousands of GPUs waiting idle for seconds, which significantly extends training times.

• Elephant Flows: During AI training, massive data blocks are synchronized. These flows are so large that they can instantly fill the buffer memory of a standard switch.
• Job Completion Time (JCT): In the AI world, the success metric is not “packets per second” but how long the training takes to complete. A 1-millisecond congestion in the network can extend the total training time by days.

Further Reading: Ensuring Reliability and Governance in Artificial Intelligence

2 – Congestion Management

One of the most critical problems in AI networks is the incast situation. Many GPUs simultaneously sending data to a single target GPU or node causes serious bottlenecks, especially during distributed training. This situation leads to switch buffers filling up, packet losses, and increased latency, directly affecting the model training time.

Congestion management in AI networks is not only about preventing packet loss, but also a multi-layered optimization problem to ensure low latency, high throughput, and deterministic performance.

3 – Congestion Resolution Methodologies

• PFC (Priority Flow Control)

When a switch’s buffer in the network starts to fill up, the switch sends a “stop” signal to the sender for a specific traffic class. This way, the relevant traffic is temporarily halted, preventing buffer overflow.

While classic flow control stops the entire line, PFC only affects the congested traffic class. This allows other traffic types (such as management or best-effort traffic) to continue flowing. This mechanism is critically important for achieving lossless behavior in Ethernet-based infrastructures.

Negative Effect

In cases of excessive or incorrect use, Head-of-Line Blocking can occur. In this case, a congested traffic class can indirectly affect others as well, causing congestion to spread across the network (congestion spreading).

• ECN (Explicit Congestion Notification)

In traditional networks, dropping packets when a switch’s buffer is full (packet drop) is an expected behavior. This mechanism works as a natural congestion control method, especially in TCP-based communication.

Depending on the application running in the background and the protocol used, these packet losses can be tolerated to a certain extent through retransmission mechanisms, and the sustainability of the system can be maintained. However, in AI workloads that require low latency and high synchronization, even a single packet loss can halt the entire computation process (Incast Problem), which is why this approach in traditional network architecture falls short. This is exactly where ECN comes into play.

When the switch queue threshold is exceeded, instead of dropping the packet, it marks the two-bit ECN field in the IP header as “Congestion Experienced” (CE). The receiving GPU that receives this marked packet sends an “ECE – Echo” message to the sender indicating that the path is congested and that it needs to slow down. Thanks to this feedback, the sender starts to optimize traffic gradually by reducing its data transmission rate before any packet loss has occurred.

• Packet Spraying

It is a Load Balancing strategy that tries to prevent congestion from occurring in the first place. Traditional networks (ECMP) always send a data flow through the same path. If that path is full, congestion occurs. Packet Spraying, in order to use all the bandwidth in the network to its fullest and to prevent a single line from becoming congested, ensures that packets belonging to the same flow are distributed equally across all available paths.

Negative Effect

Due to packets taking different paths, out-of-order delivery problems can occur at the destination. This situation can cause performance issues for some protocols or applications.

• RoCE (RDMA over Converged Ethernet)

It is a network technology used to provide very low latency and high-performance data transfer in data centers. Over a standard Ethernet network, it provides the ability to write and read data between servers through direct memory-to-memory communication without burdening the processor (CPU). RoCEv2 encapsulates RDMA packets within UDP/IP, offering Layer 3 routing capability. This feature ensures scalability in modern AI data centers. For the RoCE architecture to work efficiently, lossless configuration on the network side is mandatory. At this point, mechanisms such as PFC (Priority Flow Control), ECN (Explicit Congestion Notification), and appropriate buffer management come into play.

Further Reading: Sekom’s End-to-End Monitoring Engineering

GPU Manufacturers’ Network Architecture Approaches | Nvidia vs. Intel

When designing the AI Network Architecture, the characteristics of the GPU used directly determine your network structure. Each GPU manufacturer has its own network architecture. Today, when AI models have reached billions of parameters, the Compute Fabric structure that connects this power together is as critically important as the computational power itself. At this point, we will examine the most widely used today — both Nvidia and Intel GPU network architectures.

  1 – Nvidia GPU Network Architecture

In the Nvidia ecosystem, Compute Fabric is shaped through InfiniBand or Ethernet-based RoCEv2. In this sense, Nvidia can offer us two different alternatives. The focus here is for the GPU and the network to work separately but at the same time synchronized in the best way. Although Nvidia has its own InfiniBand network, Ethernet-based architectures are also widely preferred today, especially due to their flexibility and cost advantage.

Nvidia’s architecture is called rail-optimized architecture. As seen in the topology below, each server has 8 single-port GPUs at 400 Gbps speed, and each server’s GPU port is connected to different leaf switches in sequence. A total of 32 port 400 Gbps GPU connections are available on one leaf switch. Similarly, each leaf switch must provide 32 port 400 Gbps connection speed to the spine switch. The reason for this is to prevent any bottleneck that may occur in terms of bandwidth.

Rail-Optimized Network Interconnecting Topology

  2 – Intel Gaudi Architecture

Intel Gaudi adopts an “Ethernet-Native” approach. Instead of external network cards, it integrates RDMA (RoCEv2) capabilities directly into the processor. While reducing intra-server cabling, it brings the network design closer to a “standard scale-out Ethernet” structure. Like Nvidia, Intel Gaudi architecture also prefers to use the RoCEv2 standard.

Intel Gaudi architecture is called 3-Ply. As seen in the topology below, each server has 6 Intel Gaudi GPUs, and these GPUs are connected to 3 different independent leaf switches in groups of 2 ports. The most important feature of this architecture is that it does not require a spine switch in the environment.

  3-Ply Network Interconnecting Topology

Growing Responsibility in Network Infrastructure

In the traditional approach, network infrastructure was positioned as a component with critical responsibilities of providing data transfer between servers. Today, this scope of responsibility has been significantly expanded together with the requirements brought by artificial intelligence workloads, and it has become a central role that determines not only the connection but the overall performance of the system.

The success of AI projects is directly tied to how intelligently this infrastructure is designed and managed.

A mere 2% packet loss that can occur due to neglected details in the design will cause an 8-fold delay in the completion time of the total workload.

Establishing a lossless fabric structure, choosing the right congestion management protocols, and determining the topology appropriate for your GPU architecture are not only technical choices but also strategic decisions that directly affect the return on investment (ROI) of your AI investments. To build these critical decisions on the right foundations, contact Sekom for more information and to request an expert assessment.