Placeholder Image

Subtitles section Play video

  • Thank you. It's a pleasure to be here.

  • When something really matters

  • it elicits emotion

  • and how do we help people communicate emotion,

  • to make life better, whether at home,

  • at work, for your customers, for your kids at school?

  • Our challenge at The MIT Media Lab and at Affectiva,

  • has been to come up with better ways

  • to measure and help people communicate emotion.

  • I'll start with an example that we did years ago

  • which was a skin conducting sensor

  • that we gave to an audience like you, worn on the palm of the hand

  • and it glowed brightly every time

  • the audience got excited. Every time they were engaged.

  • We learned a lot that day.

  • We learned for example that every time a new speaker

  • came out on stage, the audience glowed.

  • Every time there was a live demonstration,

  • whether it worked or not, the audience glowed.

  • Every time there was live Q&A or laughter,

  • the audience glowed but unfortunately, every single time

  • there was a PowerPoint presentation

  • there was a decaying exponential in brightness.

  • (Laughter)

  • Let's, let's look more closely

  • at an example of the signal changing.

  • Here, for a nine year old boy, watching a movie trailer,

  • the signal will go up about a second after each of the parts of the trailer

  • that cause him to engage.

  • Starting with the music.

  • (Music)

  • The three biggest peaks.

  • (Laughter)

  • See this is where it kind of falls of here though.

  • (Music)

  • He peaks for the name of the movie and the date of it,

  • so you can see why people in Market Research and Advertising

  • are extremely interested in this information

  • or I as an educator, I'm extremely interested too.

  • 'Cause I want to know what content

  • is connecting with my students and with my loved ones.

  • Here's an example of the signal changing with a Patriots fan,

  • back when the Patriots were in the Superbowl.

  • And this one illustrates an important point about skin conductance

  • and that is even as we measure the auto-immunic arousal

  • of a person and their excitement.

  • This goes up with things that are good or bad.

  • So we see it go up here when the guy has a --

  • when his favorite team scores a touchdown,

  • it also goes up during an endzone overthrow.

  • So it goes up with good and bad.

  • Interestingly, it also goes up during a Dorito's Mouse T.V. commercial,

  • when the mouse comes out and catches the man.

  • (Laughter)

  • It's interesting now that we can get this information in real life,

  • to see how relatively important different events are in life

  • and where the significance of something that you share with somebody,

  • compares with this response to some other things that are going on with their life.

  • What drives this signal?

  • This skin conductance is driven by the autonomic nervous system,

  • sympathetic nervous system response.

  • Your autonomic nervous system has two main branches,

  • the sympathetic and the parasympathetic.

  • The sympathetic can be thought of kind of like stepping on the gas, in the car,

  • it revs your heart up, it causes the skin conductance to increase.

  • The parasympathetic causes the heart to decrease.

  • The sympathetic is thought of as fight or flight,

  • the parasympathetic is rest & digest.

  • Almost all the organs are innervated by both the sympathetic and the parasympathetic,

  • except the skin, which gets the pure sympathetic,

  • which makes it ideal for us measuring this fight or flight response.

  • Now, I became interested in measuring this response when I started working a lot

  • with people in the autism spectrum, who are non-speaking.

  • The most heart-breaking stories, that I hear from these --

  • from the families and the individuals themselves,

  • when they later acquire the ability to speak,

  • is that, the changes that were worst, were when they got kicked out of school

  • or were shipped off to an institution,

  • removed from learning opportunities and social opportunities.

  • What caused that to happen?

  • Usually it was a meltdown or a series of meltdowns

  • and the meltdowns, like adult tantrums, appear to come from nowhere.

  • One moment the person seemed calm and relaxed,

  • the next moment they might have become injurious

  • to themselves or to others.

  • So, they seemed very unpredictable and uncontrollable.

  • Now, as we undertook the privilege of getting to know some individuals

  • who could later communicate, they told us that these meltdowns

  • never came from nowhere that, in fact, they were always preceeded

  • by increasing stress and frustration.

  • They couldn't understand why people couldn't see what was bugging them.

  • So this made me think, "Wow, what if we could enable them

  • even though they can't communicate,

  • to signal out their increasing stress and frustration?"

  • But to do that required a device that is shown on the left here,

  • a classic, galvanic skin response,

  • skin conductance sympathetic nervous system measure,

  • that involved a lot of wires and boxes,

  • and that was uncomfortable to wear in daily life.

  • So my team at MIT went through lots of iterations

  • until we came up with a device that could be worn robustly

  • on the wrist and gather data in real life.

  • This has just been commercialized by Affectiva

  • and I'm pleased to say is finally available as a product

  • for people to gather and communicate this information

  • in the real world, outside the lab.

  • Here's the first example of seven days of data that I saw from a student at MIT

  • wearing the skin conductance sensor on his wrist.

  • And the first thing that jumped out here was there's a lot of studying and it's very activating.

  • Studying and lab work, in fact, sort of to the discouragement of we, professors,

  • is much more arousing that any of the classroom activity. (Laughter)

  • We also saw these huge peaks that were happening during sleep which puzzled us in the first place.

  • We just expected sleep to be flat.

  • Generally, as we did further polysonography,

  • and teamed up with some expert sleep researchers on this,

  • we found that these peaks tend to occur

  • during slow wave sleep or deep sleep, and we're now looking at the relation

  • of those to sleep quality and memory and learning.

  • Here's an example, of a girl on the autism spectrum,

  • where she's able to communicate out her arousal level, using this sensor.

  • Here she's wearing one on each ankle so you'll see two signals streaming

  • across the bottom.

  • She's wearing it while she's undergoing an ordinary occupational therapy session.

  • This is 45 minutes of an ordinary session here,

  • and you'll see, you know, it peaked here

  • and she had a little bit of a meltdown here, got in a ball-pit,

  • calmed down, peaked here, calmed down.

  • What you see up here, in blue, is right now, this window here

  • the right edge is what you're seeing in the video and we see that it peeked

  • as she climbed on this swing, but as she starts swinging,

  • as that rhythmic activity kicks in, it has this nice, decaying exponential.

  • We see for many people on the autisim spectrum, when they rock, or do these repetitive movements

  • they can be very calming for them.

  • They're not just doing that to be difficult or something.

  • It didn't have a very important, regulatory effect.

  • In fact, I've noticed an increasing number of rocking chairs, cropping up in airports

  • since 9/11 and they're almost always taken, at least at Boston Logan.

  • I think that's a healthy sign.

  • Once you get the ability to make a new measure in the real world

  • and have people wear it, unobtrusively in daily life,

  • you learn a lot of things that are surprising and this is a really special surprise we ran into.

  • One of our kids on the autism spectrum was wearing this over Christmas.

  • I was looking at the data and I saw this enormous peak on one side

  • not the other side, I thought: "Oh dear, something is going wrong with the sensor!"

  • I looked more closely and it looked like it was fine before the peak and after.

  • I asked his brother, "Any idea what could have happened here?"

  • and he said, "I was with him and he had a seizure right after."

  • Wow! I got on the phone with Joe Madson, Boston Children's Hospital,

  • "Is there any way somebody could have a huge auto-immunic surge

  • on one side of the body with a big seizure?"

  • He said: "Yes"

  • Fast forward about 8,000 hours of data and extensive analysis,

  • led by my student, Ming-Zher Poh for his doctoral work

  • and we have now shown that measuring signals from the wrist, reflects signals

  • measured from the brain and the seizures are labeled

  • by the epileptologists reading the EEG only, they coincide exactly with these peaks,

  • measured just from this electrical signal, from the surface of the skin, on the wrist.

  • So this was really mind blowing to me!

  • That a change in our brain could show up as these events, picked up through a simple wrist band.

  • We're hoping this now can lead to better detection and treatment in real time

  • of people who suffer from seizures, of the seizure events themselves.

  • Now another thing that kind of surprised me in doing this, was --

  • the events show up, sometimes, on just one side.

  • Usually the two sides are showing the same information.

  • We where just measuring two sides for redundancy, to cancel noise,

  • but we found that -- we learned something that was already in the literature

  • the right hemisphere of the brain controls the electrodermal activity of the right palmar surface,

  • and the left, the left.

  • I sort of rediscovered this when a tragic event happened in my family.

  • I lost a loved one and that day I was at work, having trouble speaking.

  • Those who know me, know I never have trouble speaking.

  • So, one of my colleagues, Ron Elco, said:

  • "Put a sensor on both sides!"

  • I'm strongly right-handed and sure enough, the left side was incredibly suppressed.

  • That night I was, sitting in the kitchen, in this total stupor,

  • just kind a stirring a pot of something

  • and one of my sons came up and started rubbing my back

  • which, I felt really blessed, this is a very special moment,

  • I almost felt hope restored and then, his little brother came up and did the same thing to him.

  • I had one of those rare moments of parental bliss, where you kind of tingle down to your toes

  • especially as a mom of sons.

  • And -- (Laughter)

  • at that moment, my little one looks up at the display in the kitchen, 'cause of course

  • I'm streaming my data wirelessly to this monitor,

  • (Laughter)

  • and he says: "Mommy, you're lines crossed!"

  • My right signal that was way above my left, had come together

  • during that moment of parental bliss.

  • Now I thought: "Wow! I've heard about positive and negative stuff for the brain

  • you now might show up but I never thought this could show up on the wrist.

  • Now, that's just a few data points, but I monitored myself overtime,

  • with sleep and sure enough, the signal, as I recovered,

  • went back to the normal sort of switching, left and right.

  • Now this is just a few data points but it shows that there's this whole new potential to explore

  • what's going on in the brain, even by measuring some of these peripheral signals downstream.

  • There are many positive moments as well, that people have shared with us,

  • using our emotion technology.

  • Here's one from a bride in India, who, naturaly, was sort of stressed out before her wedding

  • you see lots of peaks.

  • Her friend loved seeing this part where she was chilling with her friends

  • and they helped calm her down before and then the four major peaks here,

  • during the wedding ceremony.

  • The first, where the vows were exchanged,

  • the second, third and fourth in the Hindu ceremony

  • where the bride and groom exchange symbolic gestures

  • that they are now husband and wife.

  • Here's an example where a mother and a daughter wanted to share

  • the daughter's experience at school.

  • The daughter had her first, big concert, in front of school that day.

  • She was playing a harp and singing in front of the entire school

  • and she expected it would be kind of peeked there, and it was.

  • Also it was very peeked during physical education, to be expected

  • anything that gives physical exertion, makes you sweat, can be very arousing.

  • Her mom was very disappointed to see that math lessons,

  • (Laughter)

  • was the low point of the day.

  • But there was this other, huge peak there between Reading and PE, and --

  • What was that?

  • Turned out the daughter had a talk with her mom, later that day.

  • In fact as she recalled this experience, we see the emotions coming back up again.

  • This was this horrible, bullying event that she was being dragged into by her friends.

  • Wouldn't it be amazing if the people we care most about

  • were given the opportunity to share, in an externalized way

  • some of what was going in their day that was so difficult.

  • Whether they can speak or not, this can be a powerful way to take your emotions outside of you

  • and begin to reflect on them and share them with others.

  • I have one last example with the skin conductance sensor, a personal one.

  • I discovered one day while taking my son

  • ona birthday outing to Six Flags, that I indeed had big peaks

  • with the Roller Coasters, which I love,

  • and a really awful one with a ride that I won't tell you about, that was very unpleasant.

  • (Laughter)

  • But the surprise to me, when I looked at the data at the end of the day,

  • was not that I had this huge peak when I'm on the fastest, highest, most intense roller coaster

  • in all of New England, but it was actually earlier in the day

  • when we were just trying to get out the door.

  • (Laughter)

  • Many people on the autism spectrum who are conversant,

  • tell us that they have difficulty reading facial expressions,

  • they are trying to pay attention

  • to what you're saying and at the same time,

  • they can't tell if you're looking pleased or displeased, interested or bored.

  • In particular, many of them have asked us if we could please help them