Name: Performant, scalable models in TensorFlow 2 with tf.data, tf.function & tf.distribute (TF World '19)
Uploaded: 2020-03-31T06:57:28.000Z
Duration: 40 min 35 s
Description: Thousands of YouTube videos with English-Chinese subtitles! Now you can learn to understand native speakers, expand your vocabulary, and improve your pronunciation...

I'm also an engineer on that in TensorFlow team.

to be talking talking about writing performant models

Just to give a brief outline of the talk,

we're, first, going to talk about the structure

And then we're going to talk about the different APIs

that you can use to make your training more performant.

The first API that we're be talking about is tf.data.

This is the API for writing high performance pipelines

to avoid various sorts of stalls and make sure

Next we going to be talking about tf.function.

This is how you can formulate your training step

in a way that is efficient so that the runtime can

And, finally, we're going to be talking about tf.distribute

which is how to target your training to specific hardware

In order to demonstrate all of these APIs,

we're going to be walking through a case study

starting with the most naive implementation

and then progressively adding more performant APIs

and looking at how that helps our training.

So we're going to be training a cat classifier.

Because we're an internet company, that's what we do.

And we're going to be training it on MobileNet V2.

just as a representative task, so these lessons

should be broadly applicable to your workflows as well.

And I'll put the link back at the end of the slide

so you're encouraged to take a look afterward and compare it

A brief overview of training loops in TensorFlow 2.

We're going to be writing a custom training loop.

at the mechanics of what the system is actually doing

If you're used to using a higher level API,

like Keras model fit, these ap-- these lessons are still

broadly applicable it's just that some of this

So if we want to show why these APIs are important,

it's useful to really dig into the internals of the training

we've chosen to use the Keras Applications MobileNet

V2 which is just a canned MobileNet V2 implementation.

So we're going to be using sigmoid cross entropy

This is a numerically stable cross entropy function.

And then, finally, we're going to be using the Keras SGD

So if we were to call our model on our features,

in this case it's an image classification

we would get logics which is the predicted class.

we could then use to optimize and update our model.

We want to call all of this under a tf.GradientTape.

What the gradient tape does is it watches the computations

as they're performed, and maintains the metadata,

And, finally, if we wrap all of this into a step

and then iterate over our data in many batches,

this comprises the general structure of our training loop.

On the data side, we're going to start out with just a straw man

Python generator to show the different parts of the data

pipeline and then we'll look at its performance a little bit

the data so that we see a different ordering each epoch.

We'll need to load all of the images from disk into memory

so that they can be consumed by the training process.

So we'll want to resize the images from their native format

down-- back to the format that the model expects.

And then we'll do some task-specific optimization.

So, in this case, we'll randomly flip the images

to make our training a little bit more robust.

Subtitles ListPlay Video

Performant, scalable models in TensorFlow 2 with tf.data, tf.function & tf.distribute (TF World '19)

specific

strategy

multiple

scale