NA LI: Hi.

My name is Na Li, and I'm one of the engineers for TensorFlow.js

Team.

TensorFlow.js is an open source AI platform

for developing, training, and using ML models in browsers

or anywhere JavaScript can run.

Today, I'm going to show you how TensorFlow.js

powers some amazing web applications and its usage

beyond the web.

TensorFlow.js was started in 2017

and launched in 2018, along with a models repo and Node.js

support.

This graph shows the major development milestones

of TensorFlow,js.

In 2019, we launched 1.0.

We added four more models covering more use cases of ML,

including body segmentation, toxicity detection,

and sound recognition.

We launched two exciting features, AutoML and Direct

SaveModel support.

We'll show you how ML is made easy with these new features.

Also in 2019, we started to see adoption

by large users, such as Uber and AirBnB.

We've also seen increasing adoption

by app developers of many platforms,

such as WeChat and TikTok.

In the last few months, we kept expanding the models repo

to include some of the most popular models.

Today, we're going to showcase them to you.

We also kept expanding our platform functionalities.

We launched a fourth backend, WASM, around Christmas.

We also made TF.js' React Native package global available,

last month.

In terms of user adoption, we collaborated

with Glitch and CodePen, the two largest online JS Playgrounds,

to further support JS developers to learn and use machine

learning in their applications.

In 2020, we will add more models.

And we will keep investing in the TF.js platform

to support more production and research use cases.

This includes modernize the library, add TFX support,

and integrate with more products in the TF ecosystem.

We will release 2.0 later this year.

So please stay tuned.

ML is made easy with TensorFlow.js.

We provide multiple starting points for your needs.

You can directly use pretrained models,

you can retrain an existing model in the browser,

or you can build your model from scratch with a [? Keras-like ?]

Layers API.

We tried to make running existing models really

easy for everyone.

We provide a collection of models

suitable for direct use with inputs widely accessible in web

applications, such as image, video, audio, and texts.

If you already have a trained model somewhere,

we provide a converter for you to convert a model

to TF.js format.

And we just launched the converter

wizard, an interactive command line

tool to help you figure out the correct parameters.

If you have a saved model and your model will run in Node.js,

then you can directly run it without the conversion.

So no missing ops headache, anymore.

Now, let's take a look at some of the new applications

and models we and our users built this year.

First, we introduce the FaceMesh model.

This model is able to recognize 600 facial points.

Its size is under 3 megabytes.

We've seen good performance on high-end phones and desktops.

It performs inference at 35 FPS on an iPhone 11.

Next, we introduced the HandPose model.

This model has a weight size of 12 megabytes.

It also shows good performance on modern phones and desktops.

It's worth mentioning that applications

that will be enabled by FaceMesh and HandPose are only possible

when running on device.

Because as you can see here, the experience

needs to be in real time.

And TF.js enables this.

Next, let's look at a language model, MobileBERT.

The model is 100 megabytes.

It's pretty large for the web.

The inference speed is 100 milliseconds.

We made a Chrome extension for smart Q&A. Users

can ask any question on the web page,

and the most relevant answers will be highlighted.

We demonstrate that it's possible to run

a large model in browsers.

This model can also be used in conversational interfaces,

such as CheckBot.

Thus far, we have open sourced 11 models.

Using these models as building blocks,

you can build a lot of applications for use cases,

such as accessibility, VR experiences,

such as virtual try-on, toxicity detection,

and conversational agents, et cetera.

Next, we show some applications that our users built.

We're seeing a vibrant developer community

building many interesting applications and APIs

on top of TensorFlow.js.

There has been 1.4 million NPM downloads and more than 6,000

dependent GitHub repos, so far.

Besides individual efforts, companies also

use TensorFlow.js in their products.

Modiface is an augmented reality technology company

based in Canada.

They have used TensorFlow.js to build

virtual try-on applications for several beauty brands.

Using TensorFlow.js, they were able to provide a smooth UI

at 25 FPS on phones.

They can mitigate users' privacy concerns by restricting

user data to stay on-device.

They were also able to deploy their products

across multiple platforms, including browsers and WeChat

mini program platform.

It won't be possible to support ML models to run on devices

of all kinds of capabilities without the various backends

TensorFlow.js provides.

I will talk about the characteristics of each backend

and their performance.

We currently have four backends, which

includes a CPU backend that is always

available for all devices.

A WebGL backend that is the fastest backend

by running models on GPU.

A node backend that runs models on desktops, servers, and IoT

devices.

This year, we added a fourth backend, WASM,

which allows C++ code to run in browser.

We also invest heavily in future technology.

A web-GPU backend will be coming soon.

We expect to have even better performance than WebGL.

A natural question is, how fast are they?

We compared the performance of three backends

in browsers on two models.

Here, you can see how fast WASM is.

On both models, WASM is 10 times faster than plain JS

across all devices, especially on small models,

like this face detector model on the right, which

is 100 kilobytes.

WASM's performance is comparable to and sometimes

even faster than WebGL.

This is because our WebGL backend has a fixed overhead.

Simply running an NPG or SL shader

costs 0.1 to 0.2 milliseconds.

This is great, because we've seen

a trend of ultra light models being made

for running on edge devices.

The WASM backend provides a nice addition

to the WebGL backend with really great performance.

Especially, WASM is more accessible in lower end

devices.

WASM is available on about 90% of devices.

As a comparison, WebGL is only available on 53% of devices.

On medium to larger models, such as the one on the left, which

is a MobileNet model, 3.5 megabytes,

WebGL is still the fastest.

The inference time can be as small as 20 milliseconds, which

is 50 FPS.

Really good for real time applications,

whereas WASM is about 100 to 200 milliseconds.

But we're looking to further improve WASM.

With technology like Cindy, it will

be two to three times faster.

For node performance, people often

think they need to use the Pyhton-based TensorFlow

for server side solution, because it's fast.

Actually, as we'll show you here,

our Node.js backend provides same performance

as the Python-based version.

Again, we benchmark the MobileNet model,

which is shown on the left in orange bars.

The Node backend has similar performance as the Python-based