Name: Airlift Interview with Chris Decarolis (Full Interview)
Uploaded: 2025-04-17T23:09:20.000Z
Duration: 21 min 56 s
Description: Thousands of YouTube videos with English-Chinese subtitles! Now you can learn to understand native speakers, expand your vocabulary, and improve your pronunciation...

I'm here in San Francisco to talk to Chris about Airlift, a project that he's been working on.

He has a long history at Dijkstra and he's worked on a lot of cool things.

Reflexive pronouns

Would you mind giving yourself an introduction?

So yeah, my name's Chris as Colton knows, but you probably don't know.

Before that I was doing some machine learning related research stuff and then working on big data systems at Airbnb a little bit.

And I, yeah, I joined because the team was really strong and I thought the problem was really interesting.

But in that time I've worked across the stack.

I worked on open source tooling for a while.

And then once our cloud product launched, I worked on our billing platform and our alerting system, built out a few other features like zero downtime deploys, ran our 1.0 release.

More recently I've been focused on this problem of airflow migrations.

I know with Dijkstra you can observe different systems in an organization and Airlift offers a solution to both observe airflow environments and then also migrate for them.

So can you talk a little bit about why we focused on Airlift and the inception of that project?

So I think, so I think there's a few different parts of that question, right?

So to start out, like why are we thinking about airflow specifically, right?

Like why is that an important and interesting thing to integrate to for our business?

So I think most people who are involved in the data engineering space know that airflow is the incumbent orchestrator in a lot of ways, right?

It was one of the first on the scene in many ways, kind of arrived at a time that coincided with a lot of teams scaling out their data infrastructure.

And so I think for a lot of teams it became the default choice for better or for worse.

And so I think by the time Dijkstra had a fully fledged cloud product and a lot of teams became aware of Dijkstra, those same teams might've already had an airflow deployment in place.

Especially because of the power that an orchestrator holds inside the organization, right?

It's basically controlling all of your compute a lot of times.

It leads to a place where orchestration tools are very sticky and hard to remove.

And so basically we run into a situation where there's a lot of teams who really want to switch to Dijkstra, right?

But they feel pretty overwhelmed by this prospect of ripping out this orchestrate of the prospect of ripping out an orchestration system.

That also makes it hard to migrate off of airflow.

So things like poor local development story, hard to test, like Python dependency management, hell, stuff like that.

These are all simultaneous reasons that people want to switch to Dijkstra and all things that also make airflow sticky.

And so we're trying to basically figure out a way to thread that needle of like making those types of migrations easy while also allowing you to get the benefits of Dijkstra earlier in the process.

A really compelling story with Airlift for me is not necessarily migrating everything off of Airflow, but being able to observe multiple Airflow instances and then bring all Greenfield projects in Dijkstra.

Can you tell a little bit about how you might do that with just observing Airflow from Dijkstra and then bringing new jobs in Dijkstra?

So what Airlift really is, is an Airflow integration in a lot of ways, right?

Airflow has a REST API and we're basically using a lot of, we have some advanced state management tooling that allows us to read a bunch of information about the Airflow deployment using the REST API and then ingest it in Dijkstra.

And more recently in the system, we have this like very clear separation of reporting and orchestration.

And that allows us to ingest an external orchestration system and represent it in a clean way that's distinct from representing things that we're orchestrating, right?

And so basically what that allows you to do, right, is write a few lines, like the upshot of that is you can write a few lines of code, right?

And ingest your entire Airflow instance, right?

And have the whole thing show up in Dijkstra.

And because you're just writing net new Dijkstra code, right, you can do that for any number of Airflow instances.

So you can, something we've seen in the past is that a lot of teams might have a few Airflow instances, right?

They have their one Airflow instance that's doing their important stuff.

They have some other one running in MOA or something that's like doing legacy stuff.

They have some upstream team whose data sources they depend on, right?

And they'll bring all of those different Airflow instances into Dijkstra using something like Airlift.

Yeah, and another thing that's really cool is you can then trigger jobs in Airflow from Dijkstra.

So if you have, say, a machine learning team who just wants to stay with Airflow, but you have a lot of upstream dependencies in Dijkstra, you can then just trigger their DAGs or tasks from Dijkstra directly.

So the way it works, right, is, or the way to think about it is typically speaking, especially in larger teams, the data ecosystem, the asset graph that exists, right, when we think about asset graph, what does that mean?

It's like the physical set of things that are being produced by that organization, right?

Like Snowflake Tables, Excel spreadsheets, anything, right, is widely distributed across a large array of orchestration systems.

And out of all those orchestration systems, Dijkstra is kind of the one that has this notion of both orchestration and reporting.

And so it's a natural place for you to consolidate.

And so basically the whole idea is you can, if you can represent the whole asset graph within Dijkstra, right, then you can also orchestrate between those orchestration systems, assuming you have a way to trigger computations, right?

And so we have this notion of declarative automation, right, which allows you to flow computations through the entire graph, right, starting at some source point.

And so the way things become really powerful is instead of, whereas before you might have all these fragmented crons working across like different orchestration teams, right, instead now you just, if you can represent the entire asset graph, right, you can just cron the top of it, right, and then flow orchestration through the whole thing without, only when you need it.  And so that's, I think, the power of something like being able to trigger computations from Dijkstra even though they're running in some external system.

I think we'll go into an example in a little bit, but before that, I know Airlift has three stages to it, the peer, observe, and migrate.

Can you talk a little bit about why we went with three stages and the design decisions around that?

Yeah, so I think it all basically boils down to this idea that, you know, so the founder, Nick, me and him were working pretty closely on this in terms of figuring out the right way to sequence a migration.

I think the largest piece of feedback we were getting from clients who were trying to migrate off of Airflow is we want something opinionated.

We want something that's heavily opinionated that tells us how we are going to do this.

Nick gave this talk at Facebook called Evolutionary Means for Revolutionary Ends.

It's like the art of designing, the art of changing large scale software systems in place.

The kind of bonus of that whole talk is this idea that you should be making as incremental changes as possible that are easily repeatable and you don't have to think about.

And the other aspect of it is that migrations build momentum, right?

Once you see value, people are a lot more likely to do the work.

And so the whole point of Airlift is to do two things, right?

Number one is give you as much value as possible without doing migration so that you can communicate to your stakeholders why Dagster is useful, right?

Which gives you the buying power you need to actually structure a migration, right?

And then the second part is that there's no silver bullet for migration, right?

At some point, you know, there's some DAGs that are easy to migrate.

There's some DAGs that are hard to migrate.

We can help you do things, make certain migrations easier, but in the degenerate case, you're always going to have to move some code over, right?

And so the whole point is then making that process of moving code over as safe and incremental as possible.

And so the peer step, right, is this, okay, let's give you value as quickly as possible.

You can just ingest your code immediately into Dagster, no changes to Airflow, right?

And it's very safe, simple, easy to try out.

It's like people try it out in like 10 minutes, not a big deal, and they get value, right?

You still don't have to make any changes to Airflow, right?

But now you have to do a little bit more thinking about, okay, what does the actual internal asset graph look like?

Subtitles ListPlay Video

Airlift Interview with Chris Decarolis (Full Interview)

stuff

ultimately

entire

assume