Name: Performance Evaluation of Interconnect Technologies for AI Scale Up Computing UAL vs UALoE SU
Uploaded: 2025-10-25T23:18:16.000Z
Duration: 19 min 49 s
Description: Thousands of YouTube videos with English-Chinese subtitles! Now you can learn to understand native speakers, expand your vocabulary, and improve your pronunciation...

So today we are going to talk about the interconnects for scale ups.

Relative clauses (Restrictive vs. nonrestrictive)

So over the last couple of days we heard a lot of keynote speakers and presentations that talk about the importance of interconnects for scale ups.

So here we want to talk about the two technologies, what we know about the UAL and Ethernet and we're gonna talk about the three approaches to doing that UL over Sue and Rocky and I'm Sri Vengarisna and here's my colleague Srinivas Gangam.

First let's look at the two approaches of doing that.

Like once we have the UAL which is based on memory semantics that use load store and atomic kind of memory transactions and we have Ethernet which has UL over Ethernet or scale up Ethernet and these are basically again memory transactions which are encapsulated over optimized Ethernet header.

So as we heard it recently too that okay, the memory transactions can be multiple transactions which can be packed in a given packet so that we can amortize the overhead of the IPG and the Ethernet header.

For RDMA transport it basically has the complete header stack of IP Ethernet, IP and the IB header.

So this distinction is very important as we basically go over the performance results what we're going to show later.

So looking at the key advantages for scale up networking, the first one is load balancing.

When we say load balancing, the load balancing can happen at a very fine granular.

Basically we can do load balancing at 256 bytes.

All the memory transactions that have been read from the compute can be load balanced across all the Rails.

So the Rails all can be optimized for high bandwidth.

The second one is the latency characteristics for UAL.

If you look at it, the DL Flit is always 640 bytes.

But for Ethernet as we heard from the previous presentation too, the latency can be variable depending on the packet size, depending on how many memory transactions you're packing within a given packet.

And with memory semantics one of the biggest advantage what we can have is it can be push model.

When you say push model, once the COMPUTE is done, we can directly push the data from the compute tile to the networking tile and it basically doesn't incur the knock RTT delay what we see in an soc and all the transactions what are basically started at the compute tile, they all can be completed out of order.

When I say completed out of order, all the transactions are load balanced across the rails and all these transactions can complete by using the maximum Bandwidth and then the completion can be done after that.

So we can get the maximum utilization of the bandwidth.

From the implementation point of view, a lot of choices about this implementation depends on the scale up switch, what you're going to deploy the network, for example, if at all the switch is capable of doing packets per second with no limitation, then the packing logic, what you can have in the accelerator can be very efficient.

It can basically pack only one memory transaction within a packet.

But that also adds the inherent bandwidth penalty because we have the overhead for a given transaction which is the Ethernet header.

And for the QS behavior, like depending on the number of traffic classes you have, especially the no loss traffic classes.

These traffic classes are required in Ethernet because we need to separate the request and response channels, which are different for Ethernet.

But for UL we already have a request and response channel which are separated out at the EPLI layer.

So it can get better performance without any QAS properties.

But here for Ethernet we need to have separate queues and these need to be scheduled properly for end to end workload.

All these implementation choices have a direct impact on latency and the bandwidth.

What we can observe in the network, this is a typical simple framework.

What we are showing up, we have seen it multiple flavors of it.

Here we can see on the left hand side we have a compute tile.

Right hand side we have a networking tile.

And within the networking tile the TX and the RX are connected to the switch.

And here the switch can be any out of the shelf ethernet switch.

And here as the experience done and wanted to transfer the payload, it basically needs to load balance across all the networking tiles.

And here as I was saying before, the load balancing can happen very efficiently because it can load balance at 256 bytes.

And as the transactions arrive at the ethernet tile, it basically needs to go to the packing logic first.

When you talk about the packing logic, what it is trying to do is pack multiple of these memory transactions within an Ethernet packet.

But at the same time work conserving and not packing basically will affect your bandwidth.

But if at all you pack more, you inherit adds latency.

So that's kind of the drawback of packing.

Then it goes to the queuing and scheduling logic.

And here again as we talked about the queuing, we need to have separate the requests and response channels so that we basically can have the end to end workflow.

Here it can be an optimized L2 header, it can be etherne or it can be a SUE header.

Then at the link layer it basically needs to have end to end reliability.

So we basically need to at least support the LLR capability within the link layer so that we can at least have link to link protected.

The reason is it only corrects bit errors but not end to end reliability.

Then defies common across Ethernet and ul.

And on the receive side we basically have a parsing logic that can parse the packet and then it needs to unpack and basically write the packets into the memory.

The reason it is simpler is that okay, it doesn't have the packing logic, it doesn't have the scheduling logic.

All this stuff is to some extent incorporated in the DLNDL layer because DL inherently has separate channels for requests and responses.

So here, the biggest takeaway from this here is the logic is simplified here because the interfaces are much more similar because the NOC already has an AXI kind of interface or a load store kind of interface which ties very well with the UPLI interface here.

The amount of logic what we have and the area saving what we can have is significant because of this.

And now we go with the rocky workload here.

Like we have a simplified rocky workload.

Basically talking about the control path which is not performance critical here.

The connection, the memory registration and the key exchanges can happen at a very slow rate.

From the core operations point of view, we only need to support RDMA write and write with immediate.

The reason for that is okay, all the transactions are bulk transfers and we only wanted to do transfers by load balancing the large payload.

And what is not required is the send and receive operations.

So from the feature set point of view, multipathing is very important because large IO needs to be load balanced across all the paths.

So all the packets need to be sprayed across all the paths which are available.

When I say out of order delivery it means every packet needs to have an address so that it can be placed in the destination GPU as and when the packet arrives without waiting for reorder or any other complex logic.

From the congestion point of view we need to have selective acks and nacks.

The packets are basically written into the memory.

They can be ACK back and NACS is for any of the congestion related issues when a packet is dropped or then we can send an AC so that the end to end reliability will resend the packet and from the completion point of View we can do a RDMA write with immediate so that completion can be sent to the destination accelerator.

So from the rocky semantics point of view, the biggest thing is the load balancing.

As we talked about, a large I O now needs to be partitioned across multiple DMA engines within the networking tile.

And depending on the I O size, the the partition of these jobs across the networking tiles can be chunky if at all.

It's a small transfer that needs to be done, then the load balancing is not efficient across the networking tiles.

And here most of the ROCKY engines are a pull model.

When I say pull model, usually the transactions, the DMA thread is sitting in the networking tile wherein it needs to pull the data from the host memory or the GPU memory and then it needs to reorder the transactions and form a packet and send it over the interface.

Here from the rail optimization point of view, like the bandwidth across these rails may not be completely load balanced to the extent of what memory semantics can achieve.

Because here the IO chunking can be little bit more grad like little more chunkier.

And for the ACK generation again we have the same issue.

When we are writing the packet at the receiving side to write into the memory, it needs to wait for all the acks of the memories which memory transactions sent on the NOC for the acks to come back to generate the ack for the packet.

And from the resiliency point of view, again as we talked about, is a hierarchical kind of a load balancing IO chunk is basically partitioned to multiple smaller chunks within the networking tile.

And if at all a link is lost within a networking tile, then the load balancing across the networking tiles needs to be a little bit adjusted and that gets a little bit more complicated.

Going over the simple again, the XPU kind of a framework here is the same thing what we talked about before, we have a DM engine here which is kind of reading the transactions from the memory and then sending it out.

And we have the queuing and scheduling logic and most of the logic can be similar down the road here I talked about having Ethernet IP and AIB header, but

it can be a SU optimized header too here which can reduce the load.

Okay, from the workflow comparison point of view, like I'll just walk through that quickly, we went over that.

Like load balancing, it can be very efficient for UAL and for rocky it can be a little bit chunkier for the switch latency can be the same across all it will be the 200-300Ns the write responses are very efficient for UL and for Rocky they can be a little bit more delayed because of the ACK accumulation.

And the bandwidth utilization is very good for UL from the functionality comparison.

Again, here is the same thing what we talked about before.

The dmitradius pull and push and the packing logic is there for UL over Ethernet, which potentially can add some latency.

And for channel based scheduling for ual, as we talked about, the requests and responses have separate channels.

But for Ethernet we need to separate them out to separate queues from the attributes.

Subtitles ListPlay Video

Performance Evaluation of Interconnect Technologies for AI Scale Up Computing UAL vs UALoE SU

assume

critical

multiple

basically