Placeholder Image

Subtitles section Play video

  • All right.

  • Hello, world.

  • This is CS 50 on Twitch Mining This Colton Ogden And today I'm joined by Sees fifties Connor Doyle.

  • Connor, Thank you for coming into the street.

  • Thank you.

  • What do we know what we're talking about today?

  • Yeah.

  • So welcome, everyone excited to be here on the first time on the street on today.

  • We're gonna be looking at augmented reality development specifically in Spark a R, which is a tall released by Facebook to develop argumentative Aleke applications on Android and I Let's the voices on messenger instagram.

  • So I'm pretty excited.

  • Argument really is definitely trending.

  • You and I were just talking about the suite for the stream.

  • It's been around for a few years, but I think given hardware limitations probably wasn't, you know, in its initial phase probably wasn't as popular as it's surely going to be soon.

  • Tools like spark they are, or what you're gonna talk about today are, you know, sort of what we need to be able to progress in that sense.

  • Yeah, I think right now we're even augmented Reality.

  • Is that really nebulous time?

  • This mixed reality, this virtual reality there's augmented reality.

  • There's X are all these different, kind of like a vocabulary for describing this kind of movement into the virtual round, where we're going to experiencing sort of the physical world in this kind of digital way on that kind of hybrid is, um, between the digital on the sort of physical.

  • It's kind of what they are is exploring thing on.

  • Snapchat obviously have been a big proponent.

  • They've released a similar software, the Spark I R and a lot of different unity.

  • Lot has loads of our applications not just for a mobile device, but also like hollow lens magically different things like that.

  • So there's definitely a lot of excitement about a all right now and its application or M on its application in the world.

  • But I think you're right.

  • I think, like simple tools like Spark A R, which will allow you to simply simple in the terms.

  • It's really easy to get into that quite sophisticated but simple to solve, get grips with, allow you to sort of explore the kind of story telling on dhe sort of possibilities of this media for future interaction, because I think that's what he's gonna say.

  • I think right now, like one of these devices going, It's really hard, and I was really difficult.

  • But we've got to try and envision a way in which it's gonna become very seamless with the interaction with the world.

  • And, like one weighs.

  • Let's begin to visualize how we're gonna interrupt.

  • Like you.

  • I design.

  • I know people you spark out for, like you.

  • I do things like that.

  • Like one way toe start to explore play those possibilities is using a tool like this site, right?

  • Yeah.

  • We're only as powerful as the tools that we have.

  • Yeah, And you've sort of been at the forefront of, you know, ever since the d R.

  • Face started, I was 2015.

  • You've sort of been kind of passionate about this sort of new progression that we're taking in the world of tech.

  • Yeah, and this is sort of how you came onto CS 50.

  • Would you mind talking?

  • Maybe a little bit about how you got introduces suits?

  • 50 your history with us so far?

  • Sure.

  • Yeah.

  • So, uh, so back when I was a freshman, McAnally and student at Harvard College, I'm studying Chinese literature, a senior on back when we Waas, a freshman.

  • I was really interested in like storytelling, specifically looking at kind of a theater on how theater could scale to different places around the world, or how we could give a sense of, like, different environments and bitter, like No, innit, not in an actual traditional setting.

  • So that led me into a virtual reality.

  • Actually, I went along to event in Boston that kaleidoscope held with Renee on, and it's kind of like a film best the war.

  • But virtuality experiences and I tried on a headset was immediately like, well blown away.

  • I was like, This is definitely gonna be the future of like how we experience the world, how we relate to it.

  • There's something there's a seed in it.

  • It's kind of special.

  • So that began my journey into hell.

  • And then I I was looking at sort of coding things on campus, and I can't quite see It's 50 and I reached out to David and we began talking about sort of some of the video aspects of of what's been happening.

  • It's it's 50 honestly, incredible video production team and 50 production and everyone, But I was like, Yeah, what about via What about lunch reality in education?

  • What about votes?

  • Reality and see It's 50.

  • And that's when the conversation started and we did a lot of really cool projects and developments.

  • We did tears 50 around the world, which was way sent sort of V R cameras small be our cameras to different people around the world who takes it to the online.

  • We got to see that world in environment on with films saw 22 lectures in virtual reality as well, when alongside the actual traditional capture, so that people could put a headset and be at Harvard and see David and see elections.

  • So and that's really where my journey we see a 50 started and been developing evidence is very normal product.

  • I think we may have been the first university to film their way.

  • Pretty amazing.

  • Pretty amazing accomplishment.

  • Um, but yeah, I totally agree.

  • I think that v r.

  • And they are have a lot of potential on, especially once they proliferated.

  • I think because right now it's not common.

  • I think that you see people with with the actual hardware.

  • Yeah, um, but you know, more much like computers themselves back in the eighties nineties.

  • As soon as we actually proliferate this hardware, we make this a commercialized thing.

  • Yeah, to the extent that computers are smartphones, I think you're right.

  • I think we have a lot of crazy, awesome stuff, but sort of the near future.

  • In this today, we're talking about Spike.

  • There's kind of a step in that direction.

  • I'm gonna actually switch over to your screen here.

  • Show all set here on also, why don't we pluck up a question?

  • We had also sense the rules.

  • That's a great accent.

  • Shadows to Connor for being our first UK representative on street.

  • But Andre asked x r what does that stand for?

  • Yes, So ex Aw, that's a great question, excellence.

  • The coin of So because we've got augmented reality, virtual reality, mixed reality sort of X r is kind of sort of this space between like, Okay, we don't really know what the reality is yet.

  • It's like a lit.

  • It's like all of them combined.

  • It's sort of like all these kind of ex nebulous stuff reality.

  • It's like an umbrella term umbrella returned, correct.

  • Peaky.

  • Blinders is also quite funny that that's a great reference.

  • I have probably about to get out.

  • Good accent.

  • Pick up there.

  • Yes.

  • Oh, but no.

  • So what you're saying, I think that, um, in terms for X are at least because of the definitional difficulties with this technology.

  • I mean, like, it's moving so fast and different.

  • Hardware providers are trying describable to try and create these kind of lightweight glasses, which enables sort of this kind of mixed reality experience always augmented reality.

  • Always holograms.

  • Or is it?

  • Projections is really kind of difficult to kind of disown, actually what this stuff is, because in the popular language, people might say a hologram is I thing that is actually projected into space.

  • There's materialized in space, and that's like what some definitions prescribe it is that and then other people, the hardware manufacturers will say that actually, it's projecting into the retina.

  • Then maybe it is just that is a hologram as well.

  • It appears to be in space, and it's a hologram.

  • So get into difficulties, different interpretations, a lot of different versions of the lot of deployments of this idea of accentuating the real world with things that are not physically there correct would have a visual sort of represented show or not just visual.

  • I mean augmented reality can can be basically anything that's adding to the reality, so it can be sort of haptic.

  • It can be auditory.

  • It can be sort of taste, smell all that sort of stuff and is really cool developments actually happening in, like, unique sense that being generated.

  • And I are like, Wait stuff that's funny.

  • I remember back in the nineties, I think there was a smell, a vision eighties with this was the thing that people tried out.

  • Obviously, the technology back there was a lot worse, probably than it is now.

  • So I'm serious.

  • I would be very curious thing that would really take people in different, especially with games.

  • Yeah, imagine going into like the first thing that popped into my mind was playing reasonable to like.

  • There's a sewer level going into a sewer level, going and immigrant Absolutely disgusting.

  • I think about Charlie and the chocolate factory where you can like every year, might take a chocolate bar out of the screen, actually smell it.

  • Yeah, and I don't know how feasible I mean on olfactory and sort of tastes are related.

  • I'm curious.

  • Like how that would evolve into, like, taste feedback.

  • Yeah, if that's even a possibility.

  • Yeah.

  • Look, taste is linked to smell quite a lot as well, so that's kind of these interesting questions.

  • And these are all gonna be potential realms.

  • I mean, it kind of seems far right now like, Oh, I'm gonna take I want to be, like, smelling self.

  • People are exploring it and, you know, maybe it's kind of it's gonna come in on grass.

  • Interesting question.

  • The actual reality has problems causing motion sickness.

  • How does that work with a R?

  • That's a great question.

  • So one of the reasons virtuality causes motion sickness or has been like said to courts motion sickness was that because basically wearing of your headset, it includes it stops you from seeing the world at all so that if your body's moving on, you're not moving in are I've been in virtual reality.

  • Sorry, fuel.

  • So if you were walking around, but the headsets not tracking properly, then you have this weird out of body experience where, like, the track kind of froze ear, and you kind of feel nauseous.

  • Where is a R?

  • Because our is projecting onto the world.

  • So it's taking the sort of given sort of visual plane as a sort of a base level and then just adding certain things on top of it.

  • Then you're not gonna get the same kind of total out of body experience because you always have a reference point in the world.

  • So even if the track doesn't track promptly, it's just gonna be like jumping around.

  • They're not gonna be like, actually throwing your kind of whole body around in the same way the others.

  • So that's one way in which motion sickness is not as much of a problem in augmented reality.

  • Which one coming over here just a little better You're getting cut off.

  • Their apologize for that are awesome.

  • So a are will make millionaires.

  • So on that note, eso spark a are So this is what you have loaded up.

  • Where can folks folks want to get Sparky first?

  • Or is it free?

  • Yeah.

  • And where can people downloaded if it is?

  • Yeah.

  • So Facebook releases completely for free.

  • Andi can download it right now and I can get up.

  • Get it up.

  • It's Barclay.

  • All that come just download.

  • I think it's about 250 megabytes or something on dhe.

  • You signing with Facebook on essentially allows you to deploy thes whatever you developed in the development environment directly to Facebook on you can share with your friends on Facebook.

  • And there's a closed instagram bater as well now as well that you can apply for.

  • So you can also release filters directly to instagram on one of the coolest things I think about this and one of the reasons why I really enjoy using this tall and advising others on using this told is that the scene was quality of it for user experience.

  • So So you built this incredible augmented of an application in immunity on real on you like, Hey, I want to show you this Really share this experience, download this app.

  • I'm like that's been one of the biggest problems, like the friction barrier there.

  • Like having people having fun like go on down there.

  • The app just experienced, like a like a simple filter or something like they're not gonna do.

  • It works well for games, I'm guessing, but not sure something that somebody could spend 10 seconds.

  • Look, don't delete the app innocently.

  • So why spot they always interesting?

  • Because it's literally just a link to if you have messenger installed, which quite a few people do.

  • Instagram.

  • Then it just loads the camera in that in that service and just allow you to use this box.

  • They are off creations directly in the device with that up already loaded.

  • So that's one of the best things about anything.

  • Is there?

  • Uh, is there the ability to make a stand alone APS with spark Air Studios was intended strictly for its pretending to shoot me for Facebook.

  • Yeah, you deployed directly to Instagram or fighting makes sense.

  • I mean, plenty of people use it, so it doesn't matter too much.

  • Cool.

  • All right, well, I'm excited to learn a little bit about it, cause I don't know almost anything about this.

  • Great.

  • Okay, so should we, don't you think?

  • Yeah.

  • Don't leave it.

  • Folks have any questions?

  • What?

  • We're doing this definitely post with the chat.

  • We'll take a look smelling games.

  • A terrible idea of it.

  • It will be killed by pull my finger jokes in a fortnight.

  • Pull my finger jokers.

  • Importance is Andre.

  • Maybe I wouldn't like that.

  • Be kinda green.

  • Yeah, So let's Yeah, let's let do something.

  • It's fine.

  • Just put me up.

  • So I'm just gonna quickly explain like what we're looking at here before before I start building anything in technical human features.

  • So this video always makes me laugh.

  • So there's a video is a stock video.

  • You see why the head's tilting and stuff in a minute?

  • But so what we're going to be focusing on is this spot, okay?

  • Are basically allows you to use both the front and rear facing camera in on a phone on.

  • Basically, how it does that is, um we use a message out, but for the front facing camera, or you're the one that's facing a face like the one that's being shown in this sword portal view window right now is this different things that it sort of has built in.

  • It has, like a face tracker toe like, perfectly sort of track the face in three D space.

  • It has a hand tracker on, then for the reverse camera.

  • So for the rear facing camera that faces off into the world, if you're gonna take a picture off something, it has a plane tracker so plain is basically any kind of surface.

  • On what?

  • Why that's important is because if you were to place an object into the world or at least make it appear to be in the world, then there needs to be a sort of stable tracking surface that the phone can sort of know the position off in orderto accurately place that digital asset into the physical environment.

  • So that's what really facing camera does.

  • Front facing camera is what we're gonna be looking at primarily today on specifically, we're gonna be looking at some of the face tracking and Facemash tools that built directly into spark.

  • Cool, Cool.

  • Somebody actually, a pretty good question.

  • I kind of suspected the Brit accent, thanks to the Siri's slasher movies.

  • But I can't distinguish between Scottish accent.

  • Do you have any tips for distinguishing between a Scottish in a British accent?

  • Um, no one gets a box in question.

  • You do.

  • What can you do?

  • It's gotta fact calling this one sort of.

  • I wonder if it's a stereotype.

  • Maybe the UK people can do each other's accents.

  • Yeah, we call it because the disguise accent is like it's definitely distinctive Internet It's kind of distinctive.

  • We have loads of different dialects in England that vary on region, even like Birmingham verses like Newcastle, which is that's a major.

  • That's a major one.

  • Does the book was Lucas Newcastle?

  • Yeah.

  • Uh, yeah.

  • So this allows a different accent.

  • I don't know.

  • Maybe which break up, If that's true, that's true.

  • Um okay, so great.

  • So right.

  • So we have different couple of panels here.

  • Let me just talk me through.

  • So we have the inspector panel on the right hand side, which has any item that you could you click on.

  • It brings up the sort of features that you can change or different parameters on the way.

  • And so I should do I should probably get rid of our child.

  • Get a chapter that's going a little a little bit hard to see.

  • Yeah, I'll just get rid of it all together in a way that for a second Yes, on the right hand side.

  • Here you have this kind of inspector which has all the values that you can change depending on what you're clicking on in your scene.

  • Over here.

  • Esteemed panel.

  • You have the device, then you have the camera of the voice.

  • And here you've got the front and back camera, but you can select.

  • We're currently selecting on the front camera, and we was having the microphone as well.

  • You do like sonic like you could do things based on audio.

  • Yeah, as well.

  • Yeah, you can play sounds, but you can also distort sounds coming in as well.

  • Yell and make something happen.

  • Yeah, that's pretty cool.

  • Yeah, that that's feasible.

  • So in the camera, I said.

  • So if we look in this scene right here, this is kind of a three d sort of unity.

  • Almost.

  • It's almost like it's that looks a lot like unity.

  • Be very much like unity in terms of this.

  • And there's loads of different views as well.

  • You see, you've got, like, that was perspective, but you've got off a graphic you've got, but I like, right sort of front.

  • There's all different views of the three D c.

  • That's cool.

  • Yeah, yeah, very much like a blender or unity interface.

  • Correct?

  • Yeah, we're company from the front side.

  • Give me a second.

  • How did we just I always think it happens.

  • They're gonna see it.

  • So Hey, we've got the face, and this is just looking directly down the camera.

  • So the first thing we do is going to show you the face tracker innocent.

  • So if we insert here with quickness, plus button up here Is that is that coffin?

  • Oh, no, that's just happens to me because our windows moved over it.

  • So we insert face tracker.

  • It looks like nothing happened because we don't have a mess to render that face track of the right.

  • Now, the face truck at is the sort of thing that is tracking the face in the sea.

  • But all to represent that track, you might want to insert sort of face mesh.

  • So by hitting face mesh.

  • Whoa!

  • Now, we can say, um, how accurately is tracking the face in the same?

  • And some people are asking, why is the woman in the shot making all those facial expressions?

  • Yeah, but presumably to test, I'm assuming to test different things, right?

  • Yeah.

  • So the reason why that this kind of this weird, looping video looking left and right smiling is because you got it.

  • Now look at that mess.

  • We can see how well the face measure is actually performing on and tracking on the face.

  • You could just especially see it when she opens her mouth when she just like the shocked look.

  • Yeah, And also, if we look up here on the inspector panel, we have different thing.

  • It's for this mess so we can actually disable the eyes Underestimation.

  • We can also disable the mouth as well.

  • So that's kind of weird.

  • Why might someone want to do something like that?

  • So safe?

  • We society, for example, like we wanna crack mask on the face, which I'm going to show you a minute by applying some sort of material to this Facemash, if you want me to the eyes to be visible in the material or if you wanted the mouth to be visible invisible.

  • This is just a very easy way city got.

  • It makes us Yeah, so this is pretty simple.

  • This is just a phase tracker, but it's quite a sophisticated.

  • It's built straight in.

  • Another thing on a point off the back is that this'll person is doing all this look weird funny faces.

  • But if I go to video up here, I can change.

  • The person doing the video is different couple ones that Facebook gives.

  • It will give you here all we can actually hit the fights.

  • I'm HD camera on pointed.

  • It may surprise.

  • We have a green screen behind us.

  • Take it out.

  • I need you can see tracks, pretty trucks.

  • Bree.

  • And this is all just happening in real time.

  • This is over.

  • Lt's a pre recorded video.

  • This is you.

  • Actively right now making me my conceded.

  • Yeah, Andrea's asking won't be the primary device for using an air gap Oculus or a mobile phone tablet.

  • Yes, again, just to reiterate that spark I all is Sparky are built for Facebook application.

  • So it's built by Facebook.

  • It's meant to be deployed on Facebook.

  • Service is so right now.

  • It's deployed on messenger on instagram in closed beta.

  • So that's what this kind of applications for so particularly mobile.

  • But it's could be on IOS and Android.

  • Sure.

  • Yeah, works really well, People saying it works, it doesn't take elected really well, if we're going to assets.

  • So now we've got Facemash and we've got If I put my hand, you can see the kind of distorts a bit.

  • Kinda needs to detective face.

  • Uh we've got this face match going on.

  • If I want to apply material to mesh instead of this kind of canvas, if we're gonna assets and hit, create new material and then I on this under the Facemash there's a material box here.

  • If I click at, I hate my default material.

  • Now I've got the material on my face.

  • So I've got this kind of very simple, very plain, white sort of default material on I can disable my eyes, Which is kind of really weird, As I was saying, right?

  • Yeah, So this would be a guess If you had a texture of model that you didn't want to see your eyes through.

  • Yeah, that would make complete sense, Disciple my mouth, which I just did as well.

  • But we can disable whole thing way.

  • Get the real deal.

  • So I'm gonna keep the toys for now in the mouth.

  • I mean, this is a different.

  • Why this is so cool is that you have this kind of assets.

  • And the device tracker is that under the material shader under the material Sorry.

  • There was loads different.

  • Like things you can apply to the material is sharing options so under defuse, you can apply sort of different color, different colors to the to the mask.

  • But he simply by just like positioning it in different ways, it's pretty simple.

  • You can then just get off that to get back to white.

  • Everything will pick the robot guy.

  • I'm assuming you're talking about Mr Robot.

  • I may be in luck.

  • All right, there's Ah, there's a show called Mr Robot.

  • Oh, yeah, you're right.

  • I know about that.

  • Yeah, I think that's what they are talking about this really cool stuff, like speculum, which could introduce the kind of shyness to mask in different ways.

  • And then if I were to like, this is why it's so good you're just playing with this sort of difficult to take down the capacity of it.

  • He started getting these kind of really kind of interesting, weird effects on the mask.

  • Right now, there's, like a hot I immediately.

  • What?

  • I want to sell this right now.

  • There's, like, a hard line on this mascot.

  • The top.

  • All right, on DSS.

  • I wanted to get rid of that hard line going to make it look like tomorrow.

  • Probably actual food blends it a bit more.

  • There's different couple ways You could do that.

  • But one way, actually, it would be to enable this thing called Alfa down here on dhe under the under the texture part of the Alfa Spark.

  • I'll give you face reference assets appear in a photo.

  • I just looked watching the mask like actually pretty well, lip sync to what you're saying.

  • Yeah, it's real time.

  • If I go into texture and if I choose now under the reference face assets, which I just pointed out, it's under the help section of Sparkle.

  • And then if we go into face a mesh mask and if I open that you can see now this is a blend right on top of the mask.

  • The color is a little bit so right here.

  • You would maybe want to change the color of it to to match your skin tone.

  • I'm guessing.

  • Yeah, I was going to actually show something even more interesting in in a minute that could apply.

  • This is just again.

  • This is just a default material.

  • There's also you can employ image textures.

  • Here you can apply external textures, which are kind of interesting, so you can pull textures from a year old that's cool.

  • That's hosted online overnight changing.

  • Or you can also host videos, a CZ external textures, and you could actually video put a mask video directly onto this mask.

  • I'm sure Minnelli was asking what Web camera using this is just a front vice and webcam on Mac.

  • Just a Maxi Mac book.

  • Webcam was asking, Can we have a different voice?

  • I'm guessing software, but that would be kind of hard to, I think, Show on camera.

  • Yeah, I don't even think he's on camera, But yeah, you can.

  • I think you can definitely distort the voice in in that.

  • Okay, I think there are ways of data for a member.

  • You know, we don't have an audio set up for his computer currently at the moment, but great.

  • So we've got this weird mask going on.

  • Let's kind of let's get a bit more into the features of this tall set now, and think about you mentioned like, not having my face, that's kind of explore deeper into, like, potentially getting my face onto my face in this kind of a weird way.

  • Oh, right.

  • I'm sorry if I don't get rid of this material for a second.

  • Well, back to you.

  • Our favorite spinal leakage.

  • Undistracted exactly.

  • Under face tracker.

  • Actually, it was really cool tool called texture extraction.

  • If you hit texture extraction, it creates and material down here called face tracker the number of the vice record company Fire truck is there, Right face, Jack, is there a texture?

  • So if I go and create new material now, um, under the material under texture I apply the face trackers every texture just made.

  • And then if I goto, this is just for the material now and we'll call this material face off my face if I go into face Tracker interface mesh, which is the sort of chemistry sets highlighted in the window now on under material.

  • If I add my face bank, here is my face.

  • It's if Pastor being generated of your face that's getting applied to my face is a map to the mesh.

  • And you can kind of see this a bit better if I would like to insert maybe, um, like I say, a rectangle of something were quick.

  • And if I were to like time for me to bring, we're going to the face tracker and I bring the face mesh.

  • Maybe off my face.

  • Slightly Look forward.

  • Oh, right.

  • Yeah.

  • Okay.

  • Um, I don't even need a rectangle.

  • Can you see that?

  • You can call you.

  • Oh, there it is.

  • You calling to see how that's like, how That's kind of crazy.

  • So it's following your face motion, and it's applying this transformation to three object that now has got to see it now has a texture map of your face onto it.

  • Yeah, and like right now, it's a bit dark so far under the ambient lighting here.

  • Forward to like, huh?

  • A ploy.

  • Another ambient light into the same.

  • You can see it gets a bit light enough to kind of bit more matched.

  • And it may be a wonder.

  • Like maybe I want to bring down maybe the my face material eyes a bit too visible right now.

  • Maybe want to change the opacity of the material of it.

  • Oh, into practice.

  • Cool.

  • Like, um, power thing so you can see already, like, just about playing with this is some really interesting things you can do.

  • We bring the past back up here like we go into the face track.

  • We're going to the Facemash.

  • We can see we've got no always mouth.

  • But now it will enable the eyes.

  • You can see here my eyes back on the detection, his one and there's my mouth gone and there's more always gone.

  • So you put that back onto your face.

  • Now I noticed that there was, like, a so if a difference in skin tone between the texture.

  • And we've got something that might be a lighting function that is a light in Virginia.

  • And you can if I have any drink screws, it doesn't.

  • It doesn't recognize that that's funny.

  • Yeah, but already you can do pretty cool stuff.

  • And, Mike, why a simple talk?

  • A simple thing that I really like starting today from this was like, Okay, so if you just pull is attracting for a minute, we've got one Facemash, right?

  • But there's no reason that only needs to be one.

  • Wait for it to duplicate the fates tracker on the face.

  • Mesh on like may be offset down into different in a different plane.

  • So if we right now, actually, this is kind of annoying.

  • Is there, like a like a right click to rotate the scene.

  • Or so Yeah, it should pay.

  • I don't know what Eisen.

  • Hey, Nice.

  • All right.

  • If I go back to respect your head again So why we focus on the face, Mitch, You can see that one is here, and the other one is for next week.

  • But I want to move this back now.

  • Wait, Fly!

  • Thanks to our could DeRay Zeal for the fall.

  • Appreciate it.

  • Great.

  • Now you can see the book too, Tiu, Um, two faces during it.

  • No, this one doesn't have my face on it, but I can pretty easily under face.

  • Just You just duplicated the mess that was already there.

  • Just moved it further.

  • Why?

  • Just t the mesh.

  • And now I've got to, Like many Mieze operating as you can see relative position to fight.

  • So you could have, like, a scene with, like, five different people, but with the same face on all of them.

  • That's one potential thing you could do.

  • Great.

  • Um, this is called looking pretty crazy there right now.

  • Um, have we got out for on this?

  • So maybe if I just put if I go back to face Mitch Mask and I've got this kind of interesting Alfa Blah.

  • Remember the thing we did out here on the blood?

  • Oh, right, yeah, to create that.

  • Make more seen before you see, this is hard line right on.

  • If I read you that now, we're depending again.

  • So it's kind of really fun.

  • Unlike this is kind of like kind of a quite advanced, I say feature of it.

  • But track faces?

  • It's something that I really like because I think that how it's put is pulling one thing.

  • You think it would do what you think it's doing, So it's kind of really the hypnotic saying, Do this just like for you to say one thing you think it's doing is you think it's just pulling the camera data on deploying directly onto the face on Detroit.

  • They're in there because you could also pull the the actual camera feed and put it into objects in the sea.

  • And it's not doing that because if you think about it, if it takes a flat, if you take a flap, something of her face and put it onto a mask of a face, it's goingto distort and not be mapped correctly to the face.

  • It's actually working at the geometry is using the three days is a kind of reverse engineer.

  • You have an image of what the fixed look around and I can show you.

  • So under reference, face assets here under textures.

  • You can see that like this is weird if you consider that this is what a texture might be that you would apply to the face for this face, right?

  • This is what it essentially has to generate and then wrap around.

  • This is what he's generating on what is wrapping back onto your face from a standard feed, which is kind of his certainly more muscular version.

  • Muscular sort of fate.

  • Masculine, it says here face.

  • And then if we go into this, you can see this whole geo off of her face smashed.

  • And then that was the face mention mask, which creates a kind of Yeah, it's like even Andi.

  • Here's the track on points on it.

  • So if I said I wanted to create like a really cool like face pain like this, lots of different people of the needs effectively, they've got, like face paint or things that, like animate around face and stuff.

  • They'll use kind of something like faces, a reference set on bringing into the shop and now sort of distorted or, you know, graphic image minute program or whatever.

  • Um, and they'll distort it in whatever way they want.

  • And then, when it wraps onto the face, will wrap correctly onto the face.

  • I'm just asking a top left is that the lights were for the shading back in the studio.

  • This Yeah, this is the directional light so that when you move that around, you'll see changes in what the model is excited.

  • Let's just get rid of it's just actually just go back Thio based on this and we'll just a reply, a face track and a face measure of quick, because I also want to see it with the blending two and see how convincing we could make it look so hot.

  • So if I just bring this back on, I think I need to reset.

  • It says it right.

  • So this is on my face.

  • What what we say?

  • Well, a couple things that I would like to see first off used the blend thing that you talked about now, while it's on your feet like, first of all, get the image back onto it, use it More fights back.

  • Yeah, So that that would be going to my material on my face.

  • Right?

  • And then I don't think that, uh is that it's tracking correctly here.

  • Kind of interesting one second.

  • Because if that was actually the match, that it's why is it because of the material, my face and they never go to my face and I hit vice truck is a protection kind of interesting on moment.

  • I think it's because I didn't get too many of these.

  • I kind of just run the deleted stuff.

  • So if I read, I recreated So just to quickly go back go back over this and speed face tracker Face mesh Going to face tracker texture extract that face tracking texture here Credit new material.

  • Call it whatever you want.

  • We'll call it like my face or something.

  • Going to my face under texture, click face track zero texture down into Facemash material.

  • We're back.

  • Take a while.

  • Here we go.

  • So now we have You know what we were at before now?

  • What if we applied the blend operation to that?

  • What?

  • The Alfa thing.

  • Yeah.

  • Yeah, I know.

  • We did.

  • Yeah, we did.

  • Did, uh, earlier.

  • But we'll do it again.

  • How far?

  • Um, under choose file.

  • We go to face mesh mask on there.

  • That's that.

  • That's pretty out.

  • Nice.

  • Yes, you there.

  • Little starting to look more convincing, actually matches the skin tone pretty well.

  • Yeah, it's a bit dark and one of the things you can do.

  • This isn't the only way you can do this, but you can start if you've got your own face.

  • Uh, and if a tight knit me a pass, there we go.

  • That's pretty good.

  • Take down the capacity.

  • You can see there's a bit of fighting here, but you can start doing cool stuff like there are other ways to do this.

  • But you can start when it fired.

  • Speculate now had already before I You can see that it's not applying to its shape deploying to the actual skin tone.

  • You just got to a festival or something.

  • Here we go.

  • Moisture.

  • Anyway, it moist couldn't love this kind of work.

  • Do you mind if I tested on my Well, Let's see what you look like This.

  • Uh, well, it work.

  • Yeah, Well.

  • Oh, wow.

  • Look at that.

  • What gets put on some makeup or something?

  • Uh huh.

  • That's funny.

  • It's pretty crazy How that were accepted on.

  • Then it screws it up without, huh?

  • I have it.

  • Maybe they got used to yours or something.

  • Uh huh.

  • And then obviously you can.

  • What's cool is that you can start deploying like, um Collis and stuff like, because he's got the face now and it's got the job of it.

  • If you change.

  • This looks like a blooming blue man group.

  • I mean, uh, also, David popped in the chat saying nature straining to see Connor, Connor Connor and cold in the reference to your three free.

  • But call it kind of recall, So this is kind of weird, but you gotta get the gist.

  • You start playing this tall and you start going.

  • There's so many options here t things.

  • This is tracking on my face.

  • It doesn't even look like tracking on my face.

  • But you can see here in this panel, but he's doing a pretty pretty good job of doing it.

  • Another thing is that went under face tracker.

  • Right now we have Facemash.

  • But that is not the only object that can be sort of truck attract to a face.

  • If I wanted Thio sort of like for important asset right now, like there's a project I did.

  • I really liked the design of Virtual Abalo.

  • Actually, Andi as a sort of funny thing I thought would be great if I sort of took the famous speech marks of Pablo on Track that to my face.

  • So why'd created was in front Shot was a three day I'm extruded some texts and corrected three D object.

  • You can also download through the orbit online visitors quotation Mark, These are just quotation marks, right, with some space in the middle and then in spark.

  • I are This was kind of the first thing I did actually was to test.

  • It was just drop these in and if I get rid of that Oh, yeah, I get rid of the face mission now, and just have these you can kind of say that he's a kind of traps to my head, okay?

  • And it's kind of funny sort of way.

  • So if you start thinking through the possibilities of what this means is like, you can you can re correct.

  • Like artistic, you can re create any sort of artistic sort of portray or anything using the face.

  • Not you don't have to, like, just use the face as a face.

  • You can use the face as a sort of reference point for other objectivity.

  • You just at this point, you're just taking three objects in manipulating their position.

  • Orientation.

  • Yeah, you manipulate the position orientation relative to the space.

  • There's another feature that I wanted into before we go deep into the face track.

  • Although we've done quite well in the face, tracking now is a background segmentation.

  • So another big another big on I'll take another big feature are funny seeing you here, actually another big feature off spot.

  • They are one of the We haven't even scratched the surface, really, and this is meant to be a sort of introduction to spark here, not by any means and in depth exploration.

  • And their two tours of resource is online of really, really good.

  • So I definitely check them out, too.

  • Interested in any aspect of this that spark dot com, um, but here writes to say we want to do a background segmentation.

  • What is that me.

  • So right now we're actually being green screened as we saw sucrose Surprise.

  • We're being keyed on a green screen.

  • And that means that the cameras looking for green pixel on that it's taking those green pixel was out of the image and replacing them with whatever background we want in this case, my laptop screen.

  • Um, well, you cannot do that without green screen anywhere in the world using, like using your front facing camera and sparkle.

  • It isn't all the tools do this and I'm gonna show you how we can do this on Sparky are specifically so in Sparky are under camera which is taught the top left the camera, Lee.

  • That's sort built into the seat we have We have segmentation here now if we hit Plus and we hit a person on the segmentation we create down here on the textures We've created a segmentation mask detection.

  • So if I go into the camera and I insert like a background so like a rectangle say And then I make that the side of the screen by hitting Phil parent.

  • We now have this sort of canvas as the background.

  • But the problem is what How does he know what to cut around the because right now it's just overlay directly in front of the camera we want to create.

  • We want to cut around the person on what might do that is using the segmentation MascoTech chur that the camera generates automatically.

  • For you, missy, on the way to do that is to under the rectangle we need to have a material.

  • So we're gonna create the material.

  • Um, I'm gonna call it, uh, segmentation, segmentation, background under segmentation background.

  • If we then and we're gonna go Alfa um, Alfa on we go segmentation mask, texture.

  • And then we get involved.

  • Don't go back to that.

  • Was just setting on material itself.

  • Then we need to employ the material to the rectangle we hit rectangle.

  • We selected material mistake segmentation background here.

  • You can now say Not too clearly yet, but if I apply color, you're about to see it.

  • The background could be anything.

  • Lee.

  • What now he's being segmented.

  • Awesome.

  • We'll have some sort of algorithm.

  • I'm guessing that, like can detect which you know what part is that person front Would parts the background?

  • Yes, and it doesn't It doesn't pretty good job.

  • Like, if you see that that's King.

  • May.

  • Yeah.

  • It's not bad.

  • No, no, pretty well.

  • And, you know, if we changed this just approved the point here.

  • Like this isn't the color of the wall.

  • This is like the color, the way selecting on the color wheel.

  • How old does it do with, like, background objects?

  • Like if I had, if I could just pretty I mean, it does pretty well.

  • I stood behind you.

  • Wouldn't still stand behind me.

  • Yeah.

  • Oh, so it actually okay, So it takes people, It takes people.

  • Yeah, I disappeared to dissipate that for a sec, so yeah, like as he concedes, calling if you look closely, it's frank.

  • My head's has always difficult to k.

  • Right?

  • Color's gonna seeps through them a little bit.

  • Yeah, but if we go into segmentation going to the segmentation, masks texture on we go here, we can see that we can increase the mask side weekend D.

  • We can contract it as well.

  • And we can also smooth out here, make it less smooth.

  • Uh, we'll make it was soft.

  • And this means that obviously, right now there's a blue background.

  • But if I wanted to put anything behind.

  • I could if I go into the segmentation background and if I select texture on hit new image texture again, this could be an external texture.

  • It could be a video.

  • It's hosted a line.

  • Or it could be an image that's hosted online online.

  • Or we can go into town and I have a picture to see.

  • It's fair.

  • This isn't gonna wrap correct place.

  • Just it is a wrapped properly, but you can sort of see if you were to stretch distractions out properly and map it to it, it would it would make it here.

  • Is it always This is pretty fair.

  • And you said you could do this videos to you could do a video.

  • I mean, yeah, you could do a video as well as a host of CIA City Fair every day.

  • Yeah, I wanted to write, um, so far just older for me, that for a second.

  • Very good.

  • So you can see that this is interesting.

  • This is an interesting another tool that could be used to do really cool stuff.

  • Get directly in spot.

  • This introduced me to Serena actually wanted to talk about which is like a powerful part of sparking are, which is the package editor.

  • We're not gonna get too much into the patch editor today, but I want to talk through some of the things that it that it has a building for free.

  • So if I hit view on, if I hit a patch editor, you can see this is kind of bright space opens up on the bottom of the screen.

  • If I right click on that space, I get all these kind of interesting options.

  • And they have tun Side Basically what the patch it it allows you to do is create triggers and correct different actions that can be applied to any parameter.

  • Pretty much that you saw in the Inspector.

  • So that means site.

  • We look down here, it has loads of different What?

  • The court kind of like notes you could manipulate so faith landmarks.

  • So I've got cheek chin, Bible eyebrow interruption.

  • We've got, like, a blink hair.

  • You've got eyebrows lowered, eyebrows raised, head nod, head rotation.

  • So, big head.

  • Not for a second.

  • So it creates kind of like noted here.

  • It takes in the face and outputs a sort of trip an impulse is, has the head be nodded?

  • So we need to input the face, actually into his head, not tall on the way to do that is by typing and face on hit.

  • The first sort of thing is Face Finder, which finds all the faces in the current camera frame.

  • So we want to find a face.

  • And right now there's a one count.

  • If we sort of see that there's one face in the scene.

  • Um, and if I hit face select, I want to select that face, uh, of index era and put that face into the head.

  • Not this kind of a visual programming environments is a visual programming environment, and this is kind of like what sees footage would scratch and also what some other service's unrealized and unity with their third party plug and sort of offer.

  • Yeah, but why, like this?

  • Is it so easy?

  • Andi, the difficulty becomes if you stop doing more comple

All right.

Subtitles and vocabulary

Click the word to look it up Click the word to find further inforamtion about it