Name: Advanced NIC simulation for AI back end use cases
Uploaded: 2025-05-08T05:10:29.000Z
Duration: 22 min 14 s
Description: Thousands of YouTube videos with English-Chinese subtitles! Now you can learn to understand native speakers, expand your vocabulary, and improve your pronunciation...

Today I'm gonna talk about advanced NIC simulation for AI backend use cases.

So this is a collaborative work with our team and I would like to acknowledge my team, VSR, Pradeep and Radha.

So Sataranth, the co-presenter today along with me, couldn't make it, so I'll handle the whole presentation.

First I'll start with why we are doing this work, like why we need advanced NIC modeling and why is it particularly important for AI workloads.

So what's so different here that we need a very detailed modeling.

It's an open source simulator which is already available.

I'll explain details of it and how it works along with the RDMA protocol, which is like a major use case in AI workloads.

So then I will actually move to the core of the talk, which is the actual models that we have built for PCA and NIC.

We'll provide some insights with the data that we have in the current state of the work.

And then we'll call to action on collaboration.

So if you're also working on building next generation AI infrastructure hardware like us, or really interested in doing a very detailed simulation of these models, do connect with us for that.

So let's start with why this is important work.

So in general, simulation is always good.

You will need, you will have better time to market, avoid bugs in the hardware beforehand and all that.

So if we take for AI wise, it is more important.

So everybody knows from morning we are hearing about the scale of the training workloads and the time they run.

So there are other important aspects also apart from scale.

So here is like, some of the things are the importance of packet latency and to avoid packet drops here.

So if you take traditional data center, so there are like many applications talking with many other applications and each of these flows will have different parameters, right?

They will have different timelines, SLAs, some may be, majority of them would be short loop flows.

Many of them can be also elephant sustained flows.

So the effect of any packet drop here would be actually local.

It will be local to that node, that port, that application.

So there will be logic built in the application to actually handle that and it will not affect your entire data center.

But that's like very different for AI networks.

So when you start your training job, your packet latency or drop increases, it will affect your entire training time, right?

So that's the main thing that why we need to pay close attention at what's happening at NIC and PCI level and try to tune different things or observe different aspects for each of the training loads so that we get better insights.

The other aspect is if you see the data transfer on the PCI, it is remarkably different from a normal TCP application.

So normally the host would actually create, host CPU would create the packets and you would see a lot of PCI writes from the host when it is sending traffic out.

But for AI workloads, there is a, your RDMA is handled on the NIC, your RNIC.

So that is doing the reads from the host memory, right?

So it will also do, update the completion queues.

So the data is like frequent bidirection.

So the pattern would be like very different.

So if you observe some packet drops or some latency spikes here and there, then it would be really difficult if we don't have insights on what's going on here.

So that's why we started this work and try to build a very detailed models of this.

So I'll introduce you to this simulator which we have used.

So there are like so many simulators out there.

So there is a simulator to plan your IoT network which are at the edge of the system, right?

There's another simulator called Riverbed which Telco operators used to.

So they are like custom built for some purpose.

Like that, there is another simulator called NS3.

So that NS3 is a simulator which we all imagine a simulator would be like.

You run it on your normal general purpose system.

Every node is virtually represented in some way and you have logical connections between them and you have packet generation and real packets go and you actually do your whole stack processing at every node.

And so it's like a normal simulation but it doesn't work at scale.

It has like functionally you can do good functionality study like you are developing new protocols, you can use that.

And some places where you cannot simulate like a medium, like a wireless medium, like how many customers would be there, they would be moving.

So those kind of things like noise, so those kind of things that is very good.

But there is this other simulator called HTSim, high throughput simulator.

It borrows the concepts from NS3 but it's really different.

So if we see only the performance modeling, we don't really need to process the packet fully, right?

If you see the packet life, it's series of hops and every hop has some latency.

So that latency could be fixed based on whatever you're processing.

Or it could be like varied a bit based on the size and things like that.

It could be dependent on other things like what other things are happening.

Maybe it's skewed and the scheduler is like picking packets from there.

So it has dependency on other things happening.

So in that sense, if you see, you don't really need real packets also.

So in HTSim, there are like no real nodes, just a series of hops, which are logically grouped into some nodes.

You can build whatever you want with that.

And you associate different aspects to each of the block so that you can get a simulation model of your entire path.

So basically, that's a very high level overview.

There are like some more aspects which I'll cover in the next slide.

So this picture basically shows you the actual simulator.

It is available for everybody to have a look.

So I'll take an example of a Rocky protocol, which is a major protocol, which is used for RDM applications.

But the concept equally applies to even TCP, right?

So basically, you start out doing your simulation and you have your protocol set up and then you want to create a topology.

So it supports many topologies, data center topologies.

You can decide how many nodes you want, how many tires you want.

And then basically, it builds the components in the map, like predetermined fabric completely.

And then you send the notion of a packet.

So each of these building blocks will have sinks.

So I'll not go into very much technical details there, but so the sinks form hops and that will actually form a route for each packet and the packets actually follow that path, right?

So here, even though functionality is not needed, at some level, you need to handle the aspects which are related to performance.

Like for example, you need to maintain little state so that you can actually send back the X because your throughput is actually dependent on that.

So those parts of the protocol you still handle, but you don't have the full stack, right?

So this thing massively scales, but right now, it doesn't have any parts which are related to the PCI or except for basic queuing, it doesn't have anything else on it.

So that's what we have set up to add, right?

So let's see what kind of data it actually gives, right?