Name: Source Routing for AI Fabrics
Uploaded: 2025-10-25T00:00:19.000Z
Duration: 15 min 41 s
Description: Thousands of YouTube videos with English-Chinese subtitles! Now you can learn to understand native speakers, expand your vocabulary, and improve your pronunciation...

Adverbs of degree

I'm a Product Manager at Marvell, handling data center software Sonic Inside.

I am also Director of Software Product Management at Marvell.

So today we're going to present an alternate approach to managing flows in an AI cluster.

I want to be very clear when we say alternate, we're not saying better, we're just saying different.

So we'll give you a brief introduction to the AI training traffic background, the fabrics stuff that David has probably already mentioned.

The impact of job completion time, especially when it comes to congestion.

Other latency dynamics within the fabric.

Some existing approaches to improve JCT as well as our proposal and our call to action.

So with that I'll hand it over to my partner here.

Okay, so before we go into the solution of what we are proposing, we wanted to kind of set a base of the different AI models.

We have different network architectures required.

So there were like typically the old generation ones where we had small AI trainings which could have been accommodated in just a single chassis with all the CPUs and the GPUs required with all the PCIe links.

And then we have the next generation where our models kind of expanded, we needed bigger clusters.

So there we went into kind of the medium and large AI training fabrics where you had dedicated AI fabrics.

It is only dedicated for AI training and it has always made sure that you have only the AI workloads running on them.

So it could have dedicated networks and it could have even just proprietary networks.

But what we are seeing from there, going ahead with a lot of 2023 becoming genai, more and more AI applications coming into the market and more and more people integrating AI into their applications.

We see that we need a multi tenant fabric because every application today, every software application, whether it is cloud or enterprise, they are looking at integrating AI into their whole software suit.

So what that means is we can't have a completely dedicated network just for AI.

That is where the multi tenant aspect of the network comes in.

We have to cater for different kinds of applications, different kinds of AI workloads.

And even like even in the AI, the requirement for each AI applications are different.

So what we need to make sure is that we cater for all of these and every company doesn't have the bandwidth to put dedicated AI networks because their LLMs might not be as huge as typically what we have seen in a chatgpt AI, right?

So that is where multi tenant networks becomes more important.

And with multi tenant networks becoming more important, what we are seeing is Ethernet becomes a default because you have different kinds of traffic, different kinds of requirements of applications.

We see that Ethernet becomes more adaptable to these kind of networks instead of having a dedicated and proprietary solution into the AI networks.

So when we think about AI, if you think how the AI workloads are placed, how they're interacting with each other, there are multiple steps goes even before it comes onto the server, right?

So there are dedicated servers who are planning the whole AI orchestration, the work job orchestration.

There is a mapper sitting on a server thinking, okay, I have this much workload which I have to work on and I have this much capacity on my CPUs and my GPUs, so where can I place them and how do I interact to get all the results?

One of the key factors for that is job completion time.

How can I get the job completed at the fastest time possible?

the mapper has already mapped all the workloads, usually on your data center, very specific to the racks and the servers, everything mapped.

So it has a complete view and it will make sure that it knows the ideal state, where it wants to be and how it can place its jobs on different infrastructure products.

So if you think of that, the training goes in multiple stages.

So we have the initial testing parameters, we provide all the inputs into the learning model, we get intermediate results, we work on them multiple iterations to kind of get into the final trained model.

All of this has a lot of east west traffic, and

all of these east west traffic, if it is not planned, tends to have congestion on some of the resources.

Because every result you are getting out of each training step is going into one or the other resource which is concentrating all these results together.

What happens there is we hit the congestion and we have to mitigate these congestion and plan ahead.

So there are multiple ways which is kind of addressing all these congestion issues already.

So it is like we have the packet spraying method, we have the VOQ method, we have the telemetry adaptive method.

All of these are great in different scenarios.

So one of the other alternative we are trying to propose today is because you have a mapper already, you have someone who is already looking into your network and all of your resources from a higher level and it knows exactly where the job is going to be placed and how these jobs are going to interact.

We are saying that it might be better or it is a alternative approach to look at the network and the resources from a higher level.

Don't just think from a network or a server or a GPU or XPU level, but

And if you already know how you're interacting with each of these modules, then maybe it is better to configure upfront what the flows are.

So to go deeper into the whole proposal itself, I'll hand it over to Kishore who can take you into more details on this.

So like Pratya mentioned, this is an alternative approach.

There is a mapping function that takes pieces of your workload and puts it maps it to certain compute elements in your network in your infrastructure.

So our approach is to kind of piggyback on that.

So assuming that there is a controller or a mapper, there will be a way for us to pre program the flows, the traffic flows in the fabric from point A to point B.

So that approach is something that we're calling flow path routing or source routing.

The idea is to assign the paths to the flows in advance before you actually start your workload pre program your fabric.

Then we use the controller to prep the fabric, install all the routes and install the paths into your fabric as an initial step.

Your controller can load balance your flows using source routes.

Assuming it has some information about your network.

We'll talk about that a little bit later.

Your fabric nodes as defined in your fabric as the top or your leaf spine layer do not really need to do a lookup.

And we'll talk about how we go about that.

Your destination ports can be embedded into your packet ahead of time just like any old source route, similar to what you see in SRV6.

And we'll talk about the difference what this proposal has versus SRV6 in a little bit.

And we can eliminate lookup latency in your fabric.

And the idea here is to use the information in your packet to immediately make the decision.

So how does this new packet format look like and how is it embedded into a typical IP?

So on the left hand side you have your layer 2 header, your IP header and your usual transport protocol of choice.

Whether it's udp, tcp, Rocky, rdma, doesn't matter.

Right in the middle we are introducing a new shim or what we're calling our flow Path routing header in between the layer 2 and IP.

And the reason why we put it here is we do not want to dictate new hardware implementations on the Mac, just take advantage of what you already have, right?

And we don't change the ip, we don't change the transport.

In fact, we don't use the IP or the transport at all as a part of the switching decision making.

On the right hand side, this is the header that we're defining.

As you can see, there are five 16 bits fields which we're calling hop IDs.

Now your hop ID maps to a physical port number on your device.

So this implies that your controller has to have intimate knowledge of not just your fabric and what you're connecting to, but

it also has to have an intimate knowledge of the actual switch itself and what physical ports map to your front panel.

So it's a little bit more involved work onto the controller, but

it's something that we believe is achievable.

So on the left hand side in this diagram you'll see a server connecting through a two tier fabric to another server, sending traffic from point A to point B.

We have defined roles for each port in this fabric.

One role is the access port, the other role is the fabric port.

Your access port's responsibility is to perform a lookup of your flow.

And your flow can be Flow can be determined by an arbitrary set of fields up to you, the one who defines the fabric.

Right now we're assuming source port, sipdip, maybe Q pair source and Q pair destination in the case of Rocky.

Subtitles ListPlay Video

Source Routing for AI Fabrics

assume

approach

process

dedicated