Placeholder Image

Subtitles section Play video

  • Hello and welcome to another beginner's guide

  • to machine learning with ML5.js video on pose estimation

  • and posenet.

  • So this is the third, the last one that I'll do in this series

  • here about posenet.

  • First I looked at just what posenet is and how it works

  • and how you can get the key points of a human skeleton.

  • Then I took the output of the posenet model, all

  • those key points, and fed them into another neural network

  • to do pose classification, to recognize different poses

  • that I made with my body.

  • And in this grand finale pose video,

  • I will do exactly what I did in the previous video

  • with post classification.

  • But perform a regression.

  • So the final output instead of being a classifier,

  • am I making a Y, M, C, or A pose, I will make a regression.

  • What do I mean by that exactly?

  • So to review, the setup I have is as follows.

  • [MUSIC PLAYING]

  • The system starts with an image.

  • It sends that image into the pre-trained posenet

  • machine learning model.

  • That model performs pose estimation and gives as its

  • output 17 x,y pairs.

  • Wrist, elbow, shoulder, shoulder, elbow, wrist,

  • et cetera, et cetera, et cetera.

  • And then I take all of those and feed them

  • into another neural network, an ML5 neural network, which

  • then classifies those key points as Y, M, C, or A.

  • So that's the process that I've built in the first two videos.

  • I want the final output to no longer be categorical.

  • It's not one of four option.

  • The final output is any number.

  • So you could think of it as the final output

  • is going to control a slider.

  • And that slider is going to have some sort of range.

  • So what I did previously in other examples of regression

  • in this full series if you go back,

  • I used a neural network to output a frequency value

  • to play a musical note.

  • So I certainly could do that here.

  • I could train the machine learning model

  • to play the note [SINGING] for this pose

  • and [SINGING] for this pose.

  • And I could actually have something that output like

  • [SINGING].

  • So I could go and do that.

  • And boy wouldn't that be fun to watch?

  • But I want to do something different.

  • That I'll leave as an exercise to you.

  • Make a gesture or posed based musical instrument.

  • I am going to control color.

  • And this comes from a project that I referenced inspired

  • by a viewer, Darshawn, who made a project that does an output.

  • Because specifically what I want to demonstrate

  • here is that the regression output doesn't

  • have to be a single number.

  • In this case, I want to have three values.

  • And I'm going to think of those values as an R for red,

  • a G for green, and a B for blue.

  • So I can say things like, and the training can be,

  • this pose is this particular color.

  • This pose is this particular color.

  • And then this pose is this other particular color.

  • And then as I move, it will interpolate

  • between those colors by trying to guess the value according

  • to the regression.

  • Now I'm ready to start implementing this in code.

  • So I'm not going to write everything again.

  • I'm going to start from the pose classifier.

  • And the first thing that I need to do

  • is adjust the configuration of the neural network.

  • The differences instead of four categorical outputs, Y, M, C,

  • or A, I just need three continuous outputs.

  • So I could actually just change this number to three.

  • Because it's still a number of outputs but the task

  • is now regression.

  • The other thing I really need to do

  • is think about during the training process,

  • how am I going to create these target values?

  • And this is going to be really tricky.

  • So maybe this color scenario isn't the best one.

  • I only was one person here.

  • But I think to demonstrate this idea,

  • the best way would be for me to make these literal sliders.

  • So I'm going to adjust the sliders

  • and make the target outputs based

  • on the position of the sliders.

  • And then when I actually deploy the model,

  • the model will control the sliders themselves

  • and I'll see the color.

  • I think that's going to work.

  • So this target label is no more.

  • I don't have a target label, there's no categorical output.

  • Instead I'm going to have sliders.

  • So let's comment this out and say, three sliders four red,

  • green, and blue.

  • They're all going to have a range between 0 and 255

  • with some default value, in this case 0.

  • And I'll have the sliders start with red at 255 and G

  • and B at 0.

  • So we can see these are the sliders

  • that I'm now going to control.

  • And match their positions with a given pose.

  • Now if you recall, I had this horribly awkward,

  • for a variety of reasons, interface.

  • As in, no interface at all with just key presses

  • to set a label.

  • And then I'd have this, like, callback

  • hell with nested set time outs.

  • Let me improve this for a little bit for this round.

  • So one thing that I can do to improve this,

  • and I haven't been using this throughout this video series,

  • I've been staying away from it.

  • But I'm going to replace this with something called async

  • and await.

  • These are key words that operate in JavaScript.

  • They're part of ES 8 which is a newer version of JavaScript

  • that allows me to have asynchronous events happen much

  • more sequentially in the code.

  • And I've covered this previously in several videos.

  • If you haven't seen that, you'll want

  • to go watch those or read up about promises and async and a

  • await somewhere else.

  • But what I'm actually going to do

  • is I'm just going to go get the code

  • from a very specific video where I wrote this delay function.

  • I'm going to bring that in here.

  • And then I'm going to change key press to use async and await

  • with that delay function.

  • And let me just do that and then explain what I mean.

  • [MUSIC PLAYING]

  • Oh, it is so lovely, look at it.

  • Look at this nice sequential code that's, like,

  • set the target label, console log it, wait 10 seconds,

  • then do this.

  • Then wait 10 more seconds, then do this.

  • Isn't this lovely?

  • It is really worth taking some time

  • to read up and explore async and await so that you can have

  • some much more readable code.

  • This is all still happening asynchronously.

  • JavaScript, everything happens asynchronously.

  • This is just sweet syntactic sugar

  • to make our lives a little bit more joyful today.

  • But, ah, that's not really the content of this video.

  • That's not the topic.

  • The topic is, I don't have a target label anymore.

  • What I have is--

  • and actually, let's just change this to if key equals--

  • like, I'm no longer going to be collecting

  • a particular key press.

  • So let's just have the collection

  • moment happen when I do--

  • so D for data.

  • And then I'm going to have a target color.

  • And it's going to be an array with the values of all

  • the sliders.

  • [MUSIC PLAYING]

  • So the idea is that when I pressed the D key,

  • I'm going to pull the values from the sliders.

  • I'm going to set that to a target color.

  • I'm going to wait 10 seconds so I can get in position.

  • And then start the collecting process,

  • collect for 10 seconds, and then jump out.

  • Now it would be much better interaction wise

  • if I could manipulate the sliders

  • while I'm making the pose.

  • And if I could just, like, open the magic door

  • and have a volunteer come in and help me with this,

  • that might make more sense.

  • But I guess I didn't think of that in advance

  • so I'll do that another time.

  • I also think that I'm going to be able to get

  • into position a little faster.

  • So let me change this to 3,000.

  • But I haven't done the important part.

  • This target color needs to replace the target label

  • when I collect the data.

  • That's happening right here.

  • So previously I had this target label

  • that was a character that I put into an array.

  • And then passed it and add data.

  • I think I can get rid of this now and just say target color.

  • So this should be good.

  • OK, dare I say that I can collect this data now?

  • Oh, the chat thankfully is pointing out

  • that I missed adjusting these to G

  • and B. Oh, that would have really gotten me later,

  • thank you.

  • So I think also I just want to collect

  • data for, like, 3 seconds.

  • Because I'm going to do things like set the color,

  • set the color.

  • I'm going to move my arms maybe like this.

  • And then just set a lot of different colors

  • with lots of in-between states.

  • That'll really show, I think, the regression more clearly.

  • Let me also console log what colors there just so I see it.

  • I'm going to start with the sliders

  • in their original position.

  • And press D. One, two, three.

  • Collecting.

  • OK, I got some data.

  • Now let me adjust the slider a little bit.

  • Let me add some of this color.

  • I really should pick something where I could see what it is.

  • Oh well, next time.

  • Add, press D.

  • Wait, happened to my pose?

  • Uh-oh, I have a bug.

  • Bug, bad bug.

  • Bad, bad, bad bug.

  • I re-declared target color.

  • I'm making it a global variable so that I can use it across.

  • I mean, there's ultimately a nicer way to organize the code.

  • But I want it to be a global variable.

  • So I set it here and then when I'm adding it I get it here.

  • That was the problem, OK.

  • Now, let's collect some data.

  • Collecting, OK.

  • Now, let me move the sliders around.

  • I really should visualize the color.

  • But what are you going to do?

  • I'll just add a little green and take away a little bit of red.

  • I don't know.

  • And press D again.

  • And, where was I?

  • I'll go like this.

  • Really make this pretty arbitrary.

  • Oh, it really would be good for me to see what I'm doing.

  • I'll make this pose.

  • Let's do this.

  • So you, following this along, if you're

  • going to try to build the same thing,

  • think about how you might really thoughtfully make

  • a bunch of colors with a defined set of poses

  • that means something to you.

  • I'm doing this somewhat arbitrarily

  • just to see if we get some results.

  • Now I could hit S to save the data.

  • And I have a nice JSON file, this default name

  • that downloaded.

  • Let me change this to color poses.

  • Let's take a look at in Visual Studio code

  • just to make sure it makes sense.

  • Looks like it does.

  • It's got a bunch of X's, 34.

  • It's got some Y's.

  • The Y's are the outputs, and it's an R, G, and B value.

  • So I could have done the thing where I named the outputs.

  • If I wanted to have names show up in the data

  • I could change this to--

  • [MUSIC PLAYING]

  • So ML5, the neural network is just dealing with numbers.

  • But ML5 will allow you to specify names of the output

  • so that when you get them back later

  • you can figure out which is which.

  • But I'm just going to remember there's three

  • and they're in the order, red is 0, green is 1, blue is 2.

  • That'll be simpler right now.

  • Now I can go to the training the model stage.

  • So the truth of the matter is, I could add

  • and key press another option.

  • I press tree T, it trains the model.

  • But the way I made my classifier,

  • I did those in three separate sketches.

  • Collect data, train the model, deploy the model.

  • So I'm going to keep going in that way.

  • I'm going to open up the model training sketch from before.

  • I duplicated it and renamed it to regression.

  • The only thing that I need to change here

  • is the outputs are three, the task is classification.

  • And then I need a new data file.

  • So I'm going to delete other data files I have

  • and upload my new file.

  • Load that file.

  • And then everything is the same.

  • I'm just running the train function.

  • And then when it's done, save the model.

  • So there's very little that I need to change here, just

  • a different configuration.

  • Load a different data file and train the model.

  • I hope this works.

  • I really hope this works.

  • If you're watching this right now,

  • you don't know how many times I've tried to get this to work

  • where it hasn't.

  • That's promising.

  • A little bit of wonky stuff going on, but it trained.

  • The loss went down.

  • I think I've gotten some results here.

  • And it looks like those files have downloaded.

  • And we could see those files there

  • in my downloads directory, which means

  • I can go to the last sketch, the one where I load the train

  • model and deploy it.

  • So I've opened that sketch, I've duplicated it,

  • and now I just want to delete the model files

  • and upload my new ones.

  • Files are uploaded.

  • Adjust the configuration of the network.

  • I'm going to delete some old code that's

  • no longer being used.

  • And I don't have a label anymore.

  • This shouldn't be called classify pose anymore.

  • Let's just call it predict, predict pose or predict color.

  • Call it predict color.

  • And this should be the brain.

  • Because I'm doing regression I shouldn't

  • call the classify function.

  • I should call the predict function.

  • This changes to predict color.

  • And now the main work here is I need

  • to change what happens when I get the results.

  • So before I was looking at a confidence score

  • and getting a label.

  • Now I just want the raw red, green, and blue values.

  • So this should change to predict color.

  • Let me just console log the results.

  • So let's see what the results look

  • like in the case of a regression.

  • They'll be formatted differently than when

  • it came as the classification.

  • It's no longer a sorted list of labels

  • ordered by confidence score, sorted by confidence score.

  • Let me also make sure to comment out this post label, which

  • no longer gets drawn.

  • We're just going to look at the console now.

  • So in theory the first pose that it sees,

  • I should get an output here that has

  • red, green, and blue values.

  • Uh-oh, I don't see any values.

  • What happened?

  • Oh, for some reason the path says model 2.

  • I've been messing around with this code

  • and that says model 2.

  • Weird that it didn't give me an error like it couldn't find it.

  • Is it in the--

  • oh, yeah.

  • It's saying failed to load here.

  • I don't know whose fault this is, whether the web

  • editor should be showing this or ML5 didn't log it correctly.

  • But that's definitely the problem.

  • Path is model.

  • Let's try this again.

  • No pose, no pose, no pose, no pose, predicting.

  • And I'm seeing, oh, three objects, R, G, and B. Let's

  • take a look inside.

  • An R, G, and a B. An R, G, and a B.

  • So I should be able to use those values now

  • to set the positions of sliders.

  • Oh, I got to put the sliders in.

  • And then also I could just draw the color.

  • I kind of want to see the sliders move, though.

  • I think it would be fun.

  • [MUSIC PLAYING]

  • So I have three sliders there.

  • So now when I get the result, I can assign it to the slider.

  • So I can say R equals results index 0 dot value.

  • Pretty sure if I go back and look

  • at what was in that object, you'll

  • see there was an array of three objects.

  • And the red value was in a property called value.

  • And then there's 1 and 2, so there were three.

  • Then I should be able to set the slider's position

  • to these values.

  • And then I also might as well add something

  • to draw the color.

  • So here before, when I was getting a label,

  • I drew it as text.

  • Let's draw a background overlay on the video

  • with a little bit of alpha.

  • Let's grab the values from the sliders which were set.

  • And set that at the background with some alpha.

  • Did I get this?

  • Let's run it.

  • OK, look at the colors moving.

  • The sliders are sliding based on whatever pose I'm making.

  • If only I could remember what it was that I did.

  • But anything is going to give me a predicted value.

  • I'm controlling sliders with my body.

  • It's all very arbitrary.

  • But hopefully you can see that if you

  • did this in a thoughtful way, maybe color

  • isn't the output that you want.

  • Maybe three isn't the number of regression outputs

  • that you want.

  • Maybe it's music and frequencies or this or that.

  • You must have a creative idea.

  • But you can see that if you can, as you're moving your body,

  • match the position of your body with some set of numbers

  • you could then train a model to learn

  • all of those relationships.

  • And then interpolate between them

  • as you move your body around.

  • So I imagine that there's a very creative, exciting, fun, unique

  • way of doing this.

  • And I hope that you will explore it.

  • So if you do, please share it with me.

  • There's a variety of different ways to do it.

  • You can find the page on TheCodingTrain.com

  • for this particular video.

  • Ask your questions in the chat, on social media,

  • all of the above.

  • We have a new Discord, which I'll just

  • happen to mention in this particular video.

  • Coding Train has a discord, you can find the link

  • to that in this video's description.

  • That's another way you can join the community and share what

  • you've done.

  • So thank you so much for sticking with me.

  • I don't know how easy this was to follow

  • or if this makes sense because I used so much

  • of the previous code in it.

  • So if you didn't watch those previous videos,

  • hopefully those would fill in some gaps for you.

  • But let me know.

  • I can always revisit this in a future video.

  • And thanks for watching goodbye.

  • [MUSIC PLAYING]

Hello and welcome to another beginner's guide

Subtitles and vocabulary

Click the word to look it up Click the word to find further inforamtion about it