Placeholder Image

Subtitles section Play video

  • [TRAIN WHISTLE]

  • Hello and welcome to another video tutorial

  • about working with Runway and running machine learning models

  • in Runway itself.

  • Now, before you watch this video tutorial,

  • if you've never used Runway before,

  • you might want to go back and look at my Introduction

  • to Runway, how to download and install it.

  • But to be honest, you probably can figure that out

  • if you just go to runwayml.com and click on Download Beta.

  • You're going to want to download and open the runway software.

  • You'll go then to Browse Models.

  • I might go here under Motion, and I'm going to click PoseNet.

  • And you'll find yourself right here.

  • So this is where I am.

  • I've installed Runway.

  • I've downloaded it.

  • And I'm on the page in the Runway software for the PoseNet

  • machine learning model.

  • Now, what is PoseNet?

  • PoseNet is a machine learning model

  • that performs real-time skeletal tracking of one or more people.

  • And guess what?

  • I'm a person, and I've got Runway running here

  • with PoseNet.

  • So I'm going to run it.

  • So let's actually first click Add To Workspace.

  • So I already have a workspace that I've

  • made in the previous video called Coding Train Live

  • Stream.

  • I want to choose an input source, which

  • I want it to be my webcam.

  • So, yep, Runway go right ahead, and there I am.

  • And then I want to choose an output source, which eventually

  • I want to be processing, because I want

  • to get the results of running this machine learning model

  • PoseNet into processing itself.

  • But for right now, I'm just going to click on Preview.

  • So I click on Preview.

  • Oh, and I have to run.

  • But guess what?

  • So this is different than what I've showed in previous videos.

  • I've got an option for Run Locally.

  • And, in fact, this model can only be run locally.

  • It would be silly to run this one in the cloud,

  • because I'd have to spend all this time sending

  • the data over the network.

  • And it's very easy for it to run.

  • This is a very small, fast model.

  • It can be run on most modern computers.

  • So I'm going to click Run Locally.

  • So it requires no GPU credits.

  • Absolutely, can be used for free.

  • And we can see there it goes.

  • It's running right now.

  • It is making guesses as to where the various key points

  • of my skeleton are on my body in the output That's. viewable

  • below.

  • So one of the nice things about working

  • with Runway and its models is a lot

  • of times models have different parameters and values

  • and things that you can tweak and change to try running it

  • in different ways.

  • And these are sort of known as hyper parameters

  • to a machine learning model.

  • And so some of them I would actually

  • have to stop running the model and then

  • I can start to play with it.

  • So, for example, this Architecture one

  • is something I can actually make the models smaller.

  • It might be less accurate, but it will run faster.

  • But so, for example, I'm just going

  • to change this to 0.75 instead.

  • I'm going to run it again.

  • But some of these parameters can actually

  • be tweaked in real time.

  • So, for example, I can change the width and height

  • of the image, which is actually changing

  • the resolution of the image from the webcam itself.

  • And I can make it more grayscale if I want.

  • I could do various things to actually tweak

  • the image before it goes in.

  • But this is not the important piece

  • of what I want to do in this video.

  • What I want to do in this video is

  • we have a moment here where I've got a model running in Runway.

  • And I'm able to play with it, tweak it, get it exactly

  • the way I want it to work.

  • And I want to take that next step

  • from having it run here to be able to see the result of it

  • in my own piece of software.

  • So let's make that happen.

  • This software that I'm going to use to attempt this

  • is something called Processing.

  • So since this size here in Runway of the output

  • is 640 by 362, what I'm going to do in my Processing code

  • is set the size of the canvas to 640 by 362.

  • void draw.

  • background 0.

  • So now, I have a Processing sketch,

  • which I am running right here.

  • How do I see the results, the output of the model

  • in my Processing sketch?

  • So there are a variety of different network protocols

  • that Runway supports.

  • And I can find out about them up here

  • by clicking this Network tab.

  • And the one that I want to use for working with Processing

  • is OSC.

  • So there's a variety of reasons why you might

  • pick one protocol over another.

  • It really depends on what you're doing.

  • In the case of where I just want to get a single image,

  • an HTTP request would make the most sense.

  • And I'll do that in another video

  • when I show you how to work with style again in Runway.

  • But right now, I'm going to click

  • on OSC, which works pretty well with Processing.

  • And it's telling me a lot of information here.

  • So it's saying, hey, this is the server address.

  • So this is the most important thing that I need from Runway.

  • It's what I'm going to tell Processing is the unique IP

  • address, which happens to be the local IP

  • address of this computer and the port number

  • from which it can get the OSC messages.

  • I'm going to click here.

  • And I'm going to create a string called like ip.

  • I'm pretty sure I'm probably going

  • to want the port number in a separate variable.

  • And I'm going to create a port number like this.

  • Now, I could sit here and write all the code

  • for this, which is what I usually

  • do in Coding Train videos.

  • But this is a fairly different circumstance.

  • I really just want to get the example up

  • and running and show you how to do this.

  • And one of the nice things about working with Runway

  • is there are a whole bunch of premade examples

  • for you with different platforms and pieces of software,

  • one of which is Processing.

  • So let me show you how you would actually

  • do this in the real world, how I would be doing this, which

  • is the way to do it right now.

  • So if I go to the Runway ML GitHub--

  • and I actually should go back one level

  • and go here under Runway ML.

  • You can see here's the GitHub page for the Runway software.

  • And there's a lot of information.

  • There's some sort of like high-level stuff

  • here about how to port your own machine learning model

  • to Runway itself.

  • So if you've trained your own model or you

  • find a model that's not supported by Runway,

  • how you could add it.

  • But that's not what we're really doing here.

  • What I want to look for is here Processing, Runway

  • and Processing.

  • If I click here, this repository has

  • a whole bunch of examples of using Runway with Processing.

  • So you can see there's a StreetView one, attnGAN,

  • face landmarks, im2txt, and, voila, PoseNet.

  • This is the one that I'm working with.

  • So I'm going to look at this example.

  • I should say that this is an open source project.

  • Processing is an open source project

  • that I'm involved with that I've talked

  • about in a lot of other videos.

  • So maybe this is a place where if you find another model

  • in Runway that you've made work and you want to contribute

  • your Processing example here, I would encourage you to do that.

  • And Chris, one of the founders and creators of Runway,

  • and I have been talking about making a Processing

  • library for Runway.

  • And it just so happens that I made two recent video tutorials