Placeholder Image

Subtitles section Play video

  • RICHARD WEI: Good afternoon, everyone.

  • My name's Richard Wei, and I'm from the Google Brain Team.

  • Today, let's talk about Swift.

  • So traditionally, there are two ways

  • of building models in Python.

  • One way is graph building.

  • The users are required to explicitly build a graph

  • and run it and pass it to a session at runtime.

  • And the other way is eager execution.

  • Eager execution allows you to write everything

  • in the source code, and you interpret it line by line.

  • And it's really easy to use.

  • But there's a tension between graph execution

  • and eager execution.

  • Graph execution has great performance,

  • but it's not very easy to use, especially when

  • it comes to control flow.

  • Eager execution is super easy to use

  • but can be challenging to achieve high performance,

  • like graph execution.

  • Well, can we combine usability and performance?

  • Well, what we are building is something

  • that combines the ease of use with high performance

  • for your machine learning code.

  • But to do that, we have to do something more than just

  • writing a new library.

  • We had to innovate in the programming language.

  • Machine learning is so important to us

  • that we're willing to adopt a programming language

  • and entire software stack.

  • Since we need first-class machine

  • learning capabilities and primitives,

  • not only do we need the ease of use and high performance,

  • we also need an open design process

  • driven by an established community for the language.

  • This brings us to the Swift programming language.

  • Swift is a fast, modern programming language

  • and is super easy to use.

  • It's cross-platform, has a clean syntax like Python,

  • and it has all the powerful capabilities,

  • like type inference, optionals, and supports

  • functional programming and all the great features.

  • So what exactly is Swift for TensorFlow?

  • Well, the key components of Swift for TensorFlow

  • is just a magical tensor type.

  • You can write anything using tensors.

  • As you would expect, you can initialize a tensor like this,

  • like Python.

  • Or you can use a full power generics to power

  • your models, libraries, and applications.

  • Well, math operators, like sigmoid and matmul

  • are type saved generic functions in Swift,

  • without the need of name spacing when you use them.

  • When you combine things together, it's nice and clean.

  • The programming style looks just like Python's eager execution.

  • Well, as you see, tensor just works

  • and is part of the TensorFlow library, written in Swift.

  • However, it works very differently from other language

  • bindings because TensorFlow, while tensor

  • is supported as a first-class citizen in the programming

  • language.

  • For this, we built a technology called graph program extraction

  • in the Swift compiler.

  • In addition to the ability to treat tensors

  • as first-class citizens, we also need

  • the ability to train models using

  • automatic differentiation.

  • It's graph program extraction, using automatic differentiation

  • and do interesting things even with Python

  • through interoperability.

  • Well, all of these components are integrated natively

  • in the Swift programming language,

  • providing great productivity for a developer.

  • So let's start with a TensorFlow library.

  • The TensorFlow library has all the common APIs like the Tensor

  • API and data API.

  • In data API, you can use dataset initializer.

  • You can use functional zip and batch operations just

  • like TF data in Python.

  • Well in addition to nice-looking TensorFlow APIs defined

  • in Swift, you also have access to all the raw TensorFlow

  • APIs like this.

  • Tensor code sits alongside the rest of the Swift program,

  • just like eager execution, and it

  • interacts with arbitrary user-defined data structures

  • in algorithms.

  • But will it actually execute line

  • by line, one statement at a time,

  • just like eager execution?

  • Well, the answer is no.

  • The TensorFlow library works quite differently

  • from all the language bindings.

  • Swift for TensorFlow is a full-fledged compiler,

  • and it makes our code run fast without sacrificing

  • the ease of use.

  • The technology is called graph program extraction.

  • So how does it work?

  • Well, we want users to write code like eager execution,

  • but we also want it to graph level performance.

  • And we want to support native language control flow

  • so your control flow like ifs and whiles

  • can be compiled directly to a TensorFlow graph.

  • We want the compiler to handle data transfer

  • between the device or between devices,

  • between the device and the host.

  • So the burden is now on the compiler, not on the developer.

  • Developers don't even have to think about it.

  • So this is how it works.

  • Well, dataset operations such as zip and batch work as well as--

  • well, they work in a graph.

  • Tensor operations such as matmul can also

  • be represented as a graph.

  • When Swift compiler sees the code,

  • the Swift compiler automatically turns this into a graph

  • by identifying all the graph-extractable operations.

  • So Swift compiler compiles this code,

  • compiles the graph-extractable operations

  • into a TensorFlow graph.

  • And that will be part of the binary

  • that Swift compiler produces so it's one graph.

  • At the same time since it's compiling the graph,

  • it produces error messages before even running your code

  • and catches some shape errors and all the type errors.

  • Now with graph program extraction,

  • we can combine performance and usability.

  • But often in machine learning code,

  • we also need to be able to differentiate code

  • in train models.

  • Well, that's one of the key things

  • in machine learning algorithm.

  • A common way to do that is to take advantage

  • of automatic differentiation.

  • Although most automatic differentiation tools

  • are an implement as a library, we

  • want to take full advantage of being able to improve

  • a programming language.

  • So we build automatic differentiation

  • also into the compiler.

  • So automatic differentiating in Swift

  • works on custom data structures.

  • It works on TensorFlow, as well as

  • lots of standard library functions and standard library

  • types.

  • And as a user, you can define your own type

  • to be differentiable.

  • For example, you can define arbitrary structs, enums,

  • and other data types and make them represent a vector space.

  • And they suddenly become differentiable.

  • So the core of the system is a differential operator.

  • This is actually a keyword in Swift

  • because we've built that into the language.

  • For example, we have a 32-bit floating point operation.

  • It's a function.

  • And to differentiate its function,

  • we just throw a differential operator at it.

  • And we get a gradient, and we can call it.

  • It's all functional.

  • The same thing applies to tensors.

  • I have a tensor inference function

  • that computes a prediction.

  • I can throw a differential operator at it.

  • I can get the prediction by calling it,

  • and I can also get the gradient of it and apply the gradient.

  • And I can use a code like this for training.

  • Swift also supports custom gradients.

  • This is a commonly requested feature.

  • Well for example, we have a times operator,

  • and we want to apply some custom gradient on it.

  • All we have to do is to use a differentiable attribute

  • to specify the reverse mode adjoint for this function.

  • So when automatic differentiation sees the code,

  • it'll automatically call into the adjoint

  • when it's differentiating data flow.

  • Well, because automatic differentiation

  • is language-integrated, it shows the errors at compile time

  • when you're differentiating some non-differentiable function