Placeholder Image

Subtitles section Play video

  • Okay, good, so welcome to this talk and it's about advance user experience with tensorflow J's and

  • I'm Oliver

  • And this talk really is it is about user experience, but it's much more about

  • Finding good applications for machine learning in the browser and I want to kick this off because I have zero slides

  • So I want to kick this off with a stone joke

  • Hopefully no one knows it already. So this is about user experience. And oh sorry, this is about user experience in the way

  • It actually works. So hope for the internet works here. So this is what you the user do

  • So it doesn't work quite

  • Obviously guys wrong

  • Fixing it. This is I mean

  • Because you're laughing. I know you have the same feeling as me. Yeah, if you have watch people using user

  • So same thing fixing it

  • Again, yes

  • So that's the joke that I stole but obviously it worked

  • Because I think it only works if you have some sort of experience like this you think so hard about you user

  • Interface and then that's what people end up doing

  • They will figure out how to how to deal with it, but maybe not them sense you expected it

  • So this talk is about the same thing

  • Don't blame me for the for the styling

  • This is my weak spot. Oh wait

  • so, um

  • In a typical user interface. So this regard those buttons those are real buttons the the buttons down here. I'm not real buttons

  • so the buttons are just here to

  • make a point so typically

  • In a very classic

  • user interface the buttons they will light up when you hover over them to give people the the

  • certainty when you click now, you will click this button, so

  • What I had in mind was would it be at all possible to highlight the buttons before you click them?

  • So would it be possible that the machine learns? Which button are you going to click now on next and

  • You may ask what could be the applications of this and I've talked to quite some user experience people that said

  • Nothing or less and then they thought for a little bit longer. So maybe I asked you so I get something back here

  • Why would even do that? I mean highlight the button before the user clicks on it

  • Predicting which button is it going to be any ideas any takers? Yes

  • Perfect so about flows so actually I haven't implemented this but with the buttons up there

  • that's a horrible user interface because you need to click those buttons in certain order and

  • Some orders are totally don't make any sense you get weird error messages

  • This would be my next step like learn how to actually use those buttons up there because they're they're complex

  • But I haven't implemented this yet. So what?

  • What would be another reason why would you why would you do that? Yes

  • Accessibility yes, so make it easier. So that was also my fault. Yes. You have another point. Yes in the back

  • Sorry

  • Yes preload resources that was also something that I came up with

  • someone said maybe there so I don't know a server less function behind that button and you want to start it up because you think

  • Oh, the user is going to click on that button soon. You better have this thing ready or

  • Preload the next page or someone

  • I've written down some stuff. Wait. Oh

  • yeah, someone came up with the idea when there's artificial reality and it's always

  • Pretty hard to decide which detail are you going to show and then if you can predict where the user is going to look next?

  • then you can preload that stuff and

  • Again I only implemented the easiest word because I just wanted to know it's this at all possible

  • Could you even do that? And the easiest version that I implemented is just what you said

  • I highlighted the button that I think the users going to click. So, um

  • There could be heuristics to do this you could just measure I don't know

  • It's the user close to a button like this

  • and if the if it's reasonably close then you would highlight that button which I don't do now and

  • You could also check its the mouse closer to this button

  • And so they and it might all work, but it might not match the individual style

  • That someone has that used this user interface

  • for example

  • my I always use the trackpad and it's

  • extremely sensitive because I don't move much and that might mean if I want to hit this button I might

  • Briefly go over it and then go back

  • That means depending on the user that you're having and depending on the machine that you're using and how is it set up?

  • The mouse movements might be completely different. So I'm pretty sure

  • In my I'm not sure that I'm pretty sure a machine might identify me just by my mouse movements

  • Out of all of the people in the on somebody's nodding. I haven't made the experience but my mouse

  • Mouse movements are so weird. Italy makes people really nervous

  • angry

  • so

  • Having said that so what I came up with is that I said, maybe it's possible to learn that

  • Those movements using machine gun and

  • If you want to do that, the the crucial part is you need data

  • So the typical machine learning that we do you need to have data and if you don't have data

  • Probably not going to work, but fortunately I can generate data on the fly and I'm doing it like this

  • So once I hover over button, I generate a data set and typically the user would also need to click that button

  • But it would take me forever to generate just a few data samples if I really clicked on that button

  • So for the sake of this demo, I'm just counting hovering over button is a click

  • this is this is just fake or hacked but that's the only hack part and I'm doing this to

  • quickly

  • Get some data. Yeah, and what kind of data would you I mean, I'm

  • Generating data. What kind of data would you would you collect what would you do?

  • I mean if you were to predict the button I'm going to click you solve those movements. What would you recall?

  • Yes

  • Proximity you could record that I did not do this. What else?

  • Yes

  • Yes, that's what I did so I recorded XY of the mouse I recorded the Delta between the previous

  • Movement because I just get this from the mouse movie event. I just get this for free and

  • delta-t exactly what you say so and

  • There's also something really special about this. I recalled a sequence of

  • So to say time stamps and I can show these

  • If I find them on local storage, there you go, so I stole them local storage and

  • They are they give me a second here

  • And that's how they look like so this why hopefully it's big enough is it big enough for the back

  • Yeah good. So the why it's just an idea of the button and two means this one I

  • Think so. This one should be

  • One and this might be zero or maybe this is one. This is two this is three, but I'm not quite sure but why represents?

  • The button and I

  • Made this already complicated so x and y don't mean positions on the screen, but they mean the input sequence

  • This is x and y is the button that I'm going to click so

  • this is the input sequence of

  • Mouth positions you see this is position X this is position Y

  • Delta X Delta Y that means difference from the previous mouse position and this is the time step or delta T

  • So how much time how many milliseconds past from this event to the other?

  • And sometimes it's very similar and sometimes it's different

  • so that's my input and

  • I

  • Did some pre-processing on that because I was considering

  • how many of them would I really need like maybe 10 maybe 100 because I didn't know I

  • Cut all these sequences into parts and trained all of this into the model

  • So but you could like for example

  • What would he suggest that you could have taken anything else and if it works you find out by experiment?

  • That's how it works in machinery. So you make an assumption you try the model and then you see if it works and

  • well, this might work at see so I have the data now and now I

  • have this

  • This collection of buttons and I'm the only person in the world right now who knows how to

  • Use this thing because it's a horrible user interface giving a talk but user interfaces and then they playing horribly. I

  • Apologize, but that's the way it is. Now I can train on that data

  • So I collected some data and I collected some data before you see I have

  • 369 dataset that means um, I don't know my wave down here from over from outside to down here. I

  • Did some clicks to have some sample data and if you're at all into machine learning having?

  • 370 samples for three categories is nothing

  • Tippity people won't even start they will say this is not going to work. Well

  • But maybe does so I'm clicking this train model now, this is all

  • I've no fall back. That's why I'm having sweaty palms here. So this looks good

  • reasonably good

  • this is using tensorflow, Jess and

  • Tensorflow jess comes with a very nice brutalization library. I did not write this code

  • So this stuff that comes with tensorflow Jess and it shows you something and hopefully something good

  • Oh and shows me something good, but I didn't know before I kicked off this thing. So it looks good

  • What's most interesting is these?

  • Two curves down here. That's why I'm doing it now

  • and

  • These curves tell you something about how well the training process

  • Proceeds I will show you some code later

  • But let's save on the conceptual level for a while

  • And we concentrate on that blue curve and the blue curve describes over time

  • How well is the machine learning a model able to reproduce the data that that I gave to it?

  • so I've had in the mouse movement sequences and

  • as a prediction the button that is going to be clicked and now the machine learning tries to

  • learn which sequence

  • Spits out which bar and

  • the higher the blue line gets the better it is able to do that and this is around 70 percent or

  • 0.67 that means of the 70

  • Sorry off of all the mouse movements that I did 70 percent of them. The model is able to predict correctly

  • so I made some movements to the to the button and

  • That's pretty good 70 percent on that

  • pretty astonishing

  • But what's much more astonishing is the orange line because that is data points that the model has never seen that means

  • 20% which I decided deliberately I've taken off from the

  • Start now it looks good. I've taken off from from the training data, and I'm just using it

  • To

  • Constantly check how well the model is doing on something it has never seen and is anyone at all familiar with machine learning anyone?

  • When we say it doesn't look I mean would it would you have expected that

  • Who would have expected that?

  • Maybe you didn't expect that because I did make myself quite clear, but this is a pretty outstanding

  • Result so it's not something you could expect from very small training data training on this machine is a very powerful

  • GPU but it's still a laptop

  • Something like this. Typically. I don't know. It wouldn't perform that well. I'm not quite sure

  • Why does this is so welcome to the world of machinery, you make an experiment. You can't explain it just works and

  • that's my world at least javascript is also my world but also that so um

  • before we try that and I'm hope you saw I just tried trade this online we can get some more information and

  • This is also built into the visualization tools of tensorflow

  • and I'm always so happy if you guys don't see any red stuff in the console because it's it seems to work quite well and

  • This is a while. I'm hiding this for a second because what you see now, it's a very powerful tool

  • It's called the confusion matrix. I don't think it's only used in machine learning but in

  • Machine learning it actually is so what can you see on?

  • The y-axis that is this axis here

  • you see the real buttons that the model should predict because it knows those are the real buttons that means

  • left button middle button right button and on

  • The x-axis you see the prediction of the model and this is useful

  • Because in the perfect world and in the model that works perfectly you would expect all the clicks to be on that diagonal here

  • But you don't see that you see something else. So what do you see? So this means the real

  • value that the model should predict is left and

  • The prediction actually also is left that's good. But then this here

  • The predictor should be left, but it was middle

  • 22 quite a few and then the prediction should be left but it is right and

  • Then for all the three buttons so for the middle button

  • we see something that at least from my perspective was expected because the middle button is so close to the other buttons that

  • You get a lot of confusion

  • Even though it is the middle button you get a lot of confusion for either left or the right button

  • so that's also totally expected and

  • Finally for the right button that's also something that's slowly expected and also something that I would have expected for the left button

  • So it's actually the right button, but sometimes you confuse it with the middle button, but only rarely with the left

  • And this is because of the way I created those silly mouse movements

  • But it's actually pretty good even without trying the model. I now know that this model might be good. Um,

  • There are some problems with it first problem

  • This is only on the data the model had we still have no idea

  • is it going to perform well in real life I mean and

  • will I do the same mouse movements I make a little bit interesting because I'm still I still also don't know but it could it's

  • It looks good now

  • If you think you would never show this to the end user something like this and the end user would never use something like this

  • this is totally true and you can at any time heighten sting and

  • You can toggle it back in later

  • this is just tools for developers like me and

  • I also haven't shown this but while I was training it was totally usable

  • So the whole thing was was trained in real time, and the user would not have noticed that I'm training

  • Something I also did not show him. But maybe it's also too late now the hit on my GPU was immense and

  • Also the the fan of my laptop

  • Started to blow because this takes a lot of compute power on your GPU

  • Uses WebGL, so that means this this is your installation

  • You could load this in your browser right now

  • But you wouldn't know which buttons to click so don't do it because it has horrible user experience

  • Um, so that means the idea is the user clicks something you record at a certain point thing

  • this might be enough data you train the model you get all the

  • Evaluation results and then decide am I going to use it for a real prediction or not?

  • And this thing might be okay. We have one more metric and this is accuracy per button

  • That means this is this is just a number and this number there's an API for it

  • I'll just question that number and then I could write some code if the accuracy is above say 70% for all the buttons

  • I will enable that model and I can just do that

  • How shall I do that?

  • Yes, I'm gonna bit scared. I'm really so this is no show that I'm scared

  • so I can toggle the prediction and

  • I need to open the console because all the interesting stuff

  • Is still locked here. This is very early example, so now

  • Let's see. So what you see

  • I'm doing this thing where I'm pretty sure that that it will be. Okay. So just see oh it looks okay

  • So now it will predict I'm going to click the right button and there's no explanation for that

  • Yes, there might be an intuition. It changes a little bit still right button, but

  • This thing basically is a black box. So you have no idea. Why does it you just happy that it does

  • but you get some information you get some information down in the console and this is

  • probability per button

  • so these

  • So the microphone is here

  • I could have used the mouse, but then I would change this so

  • let's stick to this one, right so

  • That means the probability for the right button

  • 75%

  • the probability for the middle button is 10 and the probability for the left button is 15% that

  • Means at a certain threshold that I decided to be 50%

  • I will highlight that button and say the user is going to click that button and let's see now so

  • Middle button, you see I'm not hovering over it. I'm them just close to it

  • Left button in works, and I had sweaty hands it sort of works

  • You wouldn't probably say it's perfect. And it also seems to work on proximity. Not not of mouse path, right?

  • Maybe limit on Mouse path, but mostly on proximity

  • At least that's my impression. Would you agree?

  • Is it Mouse path?

  • It's not perfect. You see that it's not perfect, but it might be good enough

  • For anything reasonable and we just did this on the fly right here. So it's it's not something that I prepared

  • Um, but you might ask it's this maybe you asked that I don't know if you do. I'm just saying you ask

  • Is this the right way to do or would you rather?

  • Not use proximity so much but do you really the movement of the mouse is that more interesting and

  • if you value proximity or the the past of the mouse movement over a proximity and

  • That is something that you model into your model and I will show the model now

  • Hopefully

  • Sorry just enough space

  • So, um, then it's the tensor flow test part and all the other part

  • I'm pretty sure all of you could also write this is just

  • clicking and then have a mouse missed 9 then have a quick list not all that stuff and prepare the data has had a little

  • Bit but this is the part where you only have very few lines of code

  • But you need to think about those few guys of course and have variations of this so

  • This tensorflow thing is it's the tensorflow namespace

  • I'm saying I want to a sequential model. That means we have layers they follow each other and

  • The first and this now it gets interesting. So this stuff before maybe not so much

  • This says it's a simple RNN and that is the sort of neural network that can learn from a sequence of things

  • And there's a lot of stuff that you don't need to know about

  • but this will say I

  • have

  • quite sure how many maybe it's 25 I cut the the

  • mouse movements into into chunks and then have five features that means how does the input to this thing look like and

  • Then this says how many neurons do I want? And this is tony hand-wavy you just try that out

  • and typically you don't want to use too many because it needs to be trainable on the browser and

  • You saw it is if you take less it goes much faster, if you take more it might not work. So well and and

  • It also takes a lot of time and you saw the predictions were actually real time online

  • so the interesting part is

  • The evaluation of this thing was is much worse

  • Than on the other things that I will show in a minute

  • But in real life it showed it showed the behavior that people actually expected that when I moved close to a button

  • It should be highlighted

  • That's the that's a result people expected. But actually maybe you rather want something where the whole path the whole movement

  • Falls much mentor into into account and there are other

  • styles of neural networks that are better at this so you decide how much of the history do you want to keep and how much

  • interesting how interesting is their history and I have

  • Wait I have this

  • Pre-recorded because I didn't know if I would be able to make this run in return. So this thing here

  • In the middle you see it reacts much faster or much

  • Very much earlier. It was it was an earlier version of this thing

  • But it only reacts not by proximity but of the path that you did that mouse describes

  • so it's

  • Much more expensive it took much more resources to train and it's still not

  • Intuitive because if you if you just move close to the button

  • It will do won't do anything because that's not how you click on a button

  • You don't slow you go to the button and then you eventually click. This is much more

  • Much closer to what you actually do and if you want to do that

  • you will use a different type of machine learning model and

  • this different type of machine learning model is

  • Commended out here

  • it's called an LS TM which means long short-term memory network and

  • You might see that apart from this thing here

  • Where's this? It's same thing and the name is different. But the name is just name of variable. Then there's a

  • cheaper version of this which is called GRU, but

  • You could just exchange that and do different experiments see and I made some comments

  • slow to train

  • Evaluation is bad, but it actually performed really well and in its it's the network that I used for the for the second example

  • Good five minutes left so I will show some more code unless

  • They're questions right now questions

  • Yes

  • Yes, so the question is

  • Have I made something like a baseline? Where you?

  • Exit and no but exactly what you propose also was in my head

  • I'm just using the vector and I'm seeing what is the the mouse curve pointing?

  • It might be it works better, but I haven't done this and also not proximity

  • I think it would be it would be pretty easy to implement, but I didn't want to spoil it for myself. I think

  • Maybe it works better. If you do that, please contribute. It's on github make a pull request

  • I would be maybe I would not be so happy, but please still do

  • It could be better. So but the point being here, maybe that's maybe the use case is silly

  • I'm not even sure myself, but the point here is so this data it never left the browser

  • So this stays in the browser, you can download it yourself

  • But it will never reach the server of someone you don't trust in terms of privacy

  • For example, you don't want that maybe and also the model that you're training is also completely it stays on your machine

  • It will not leave your machine and the prediction will also be on your machine and it will be in real time

  • That means you do something and and the model will predict you didn't see any at least I didn't see any delay

  • And you didn't see any installation so you can start this maybe not on your phone because you don't have the mouse

  • But you couldn't start this on any laptop and it would still work if you know, which part is a click

  • And that's the point. So if you if you have anything you think that might be a good idea

  • This might be the right thing. Otherwise, it's pretty hard to explain. Why would you even do machine learning in the browser? Because

  • for example

  • the browser doesn't

  • doesn't train at least in Chrome when when the

  • The window is not upfront when it's not high light or when you look at something else. The browser will restrict your resources

  • So it's not the ideal platform for training, but it might be if you after

  • previously or

  • high activity

  • so

  • just

  • Well, hopefully to show you that it's not very hard to get this done so this is something you could copy from pretty much any

  • Tutorial and that's the whole model so model definition

  • this part

  • And this part and the other part is commented out so it doesn't count I guess and

  • the most

  • Experienced that have put into it was to make it very general with very few data sets

  • Which is typically extremely hard and this involves

  • tweaking with strange parameters where there's barely any theory to it and one of the theory sorry one of the

  • parameters that was tweaking was this drop out thing and then this patch normalization and and

  • this is just

  • experimental you've

  • You have some intuition and then you just play with it and you see when it performs well and what you see as a result

  • Of this very strong. Well sort of strong

  • Regularization so it still works with not many data sets. Sorry

  • so

  • preparation boring

  • training

  • This is training, so you have the model you call fit that means you stick in

  • the input that is the mouse movements and you get back the

  • button, but in this case you just train the model tell it this is the true data that I have learned from it and then

  • There's still too many parameters that I can't explain now, but I would be able to do that in five minutes

  • Okay, then prediction so done after that

  • This was triggered by one of the buttons. So after that you can evaluate that

  • Somewhere and then eventually you call predict on that and this is again model stick in the X data set get back

  • the Y that means the bar or

  • In this case to be more accurate the percentages of probability for each bug being picked

  • so

  • You can save that to local

  • Storage you can download that

  • And then you have two

  • files that you can store you can use the data to train that on just normal tensorflow, which also did

  • Where I exported the data I did the training

  • and then you can also

  • Export the model that you trained on a real server and load it into the browser if you really want to and the idea

  • Behind that is maybe you want to share

  • Some training experience with other people who are willing to give the data to I don't know someone else

  • Uploaded it to it

  • To a server and then you could train the general model

  • Upload it to this thing here. I could do this. It's loading remote model now and you can see I

  • Don't know if it's performing well

  • Not so much, but also not so bad. So this model that I just loaded it has been trained on a server

  • Probably more data and still works

  • So with that, I'm

  • Closing it and what I would like you to give to me is is ideas

  • So I don't with our I don't on person

  • is there anything any inspiration you got from this what you could do in the browser didn't tell me and

  • If I mean if it's great that you keep it to yourself just implement and make a lot of money

  • But maybe maybe if you need I don't know

  • Some going back and forth talk to me. I would be really happy. So thank you and have a nice conference

Okay, good, so welcome to this talk and it's about advance user experience with tensorflow J's and

Subtitles and vocabulary

Click the word to look it up Click the word to find further inforamtion about it