Is it possible to use only these images to

reconstruct the sound? that is can you hear pictures?

in this video I'm gonna

try to demonstrate that it's possible to get sound from pictures but it's not

gonna be easy so I'm gonna need some help.

this episode was sponsored by

LastPass which allowed me to fly to the Bay Area to meet with one of my science

heroes so now on with the video.

How're you doing?

I'm sorry my place of my concern is like threw all

this crap over here.

So this is the experiment that I came up with. -It's like

a crumpled up ball of tinfoil? -yeah like if I had a more powerful camera and I

had like the right lens then you know we could do something that looks more like

you're spying on somebody. But we should be able to demonstrate that you can

recover like a rhythm or a sound from you know whatever camera you have.

Now you might think it would be easy to record sound in video because after all

sound is just vibrations so the air is vibrating back and forth and everything

it hits should vibrate back and forth too. So you'd think all we need to do is

video that motion and plot displacement versus time and then recover the sound.

But it's not that simple because for one thing I mean these sound vibrations are

incredibly tiny. They move objects only about one micrometer and even if you're

super zoomed in that is way less than a pixel we're talking a hundredth or a

thousandth of a pixel. -We're not seeing something that is at one pixel move to

an adjacent pixel. You're seeing one pixel get slightly darker and the

next pixel get slightly brighter. -What objects work best for recovering sound? -So

the things that work best are things that have a lot of damping but are also

very light so that they move very readily with changes in the air pressure

so what are some good examples? -well like a bag of chips.

you know the initial

experiments were very like contrived in a way. You know we had these objects on

optical benches we were blasting light at them

the sounds were like as loud as we could make them.

Mary had a little lamb, little lamb, little lamb

I figured we'd try to do a rhythm.

This is...

shave and a haircut

let's put that camera on a tripod

- oh yeah that's I mean that'd be great

All right, let's give it a shot,

this is the actual clip I recorded and I want you to

notice two things: first, you can't really see much motion; and second, there are

plenty of pixels getting dimmer and brighter because of image noise. I mean

it's not a pristine, perfect image so how do you tell the difference between

pixels getting brighter and dimmer due to tiny movements versus noise?

Essentially you look for edges in the image and then you say well if the object

moves by some fraction of a fraction of a pixel in one particular direction

then pixels on one side of that edge will get a little bit brighter pixels

the other side of that edge will get a little bit darker. And so basically what

we do is we sum together all the ones that are supposed to get brighter and

subtract all the ones that are supposed to get darker and then that gives us sort of

one number right and if you track that number over time then that gives you an

estimate of the displacement over time.

This is time but in samples and

then this is position.

-hmm so what do you do now?

well we're gonna try to do some

filtering on that.

it's clipping. You don't say

I mean it's not much... it's not much but you can

recognize two of the beats.

This is what we've recovered from a hundred and

eighty frames per second, which isn't really a whole lot within the range of

like audible frequencies so that's why you know kind of the most we can hope

for here is a bit of a rhythm

Now of course the main limiting factor

is framerate because we can hear sounds from 20 Hertz to 20,000 Hertz but most

cameras only shoot 30 frames per second so they miss virtually all of these sound frequencies.

Imagine this is the motion created by a 30 Hertz sound if

you try to capture this with a camera at 30 frames per second you would end up

seeing the object always in the same position because it's at the same point

in the wave cycle. So your perception would be that the object is not moving at all

So in order to measure a frequency of sound you actually need to

sample at least twice that frequency which is why a lot of music is sampled

at 44 or 48 kilohertz - that's more than twice as much as the highest sound we can hear.

At any rate if you want to get a something more intelligible you're

going to need some higher frame rate camera.

so we just went to the camera store and picked up a new camera that should be

able to shoot a thousand frames per second or thereabouts.

Is that gonna be enough?

it'll be enough for something.

Hehehe I love that confidence

This is one modulation away from dubstep right here.

Just a little bit more of a wump wump

and then we have the next big track.

I've set it to a thousand frames per second

now we're talking -yeah

okay

okay so you have the footage there and

you're cropping in a bit, tell me about that.

well we're running this on my

laptop as opposed to the servers that I had back at MIT and that is gonna mean

that if we run it on the full video of my laptop will crash -okay

so we're gonna crop it and try it on that - I can see a little bit of motion

-yeah I mean I think

that one question is whether that's like -resonant motion

yeah well in this case

would be kind of like the equivalent of a rocking chair like if the if the foil

has a like a rocking mode then that's actually not gonna give us a lot of

sound information -mm-hmm

Can you tell whether this is gonna work or not?

I am optimistic.

I think because I know what I'm listening for I can hear it in there but

yeah you've gotta be careful though really?

that you're well just gonna be careful

that you're not like confirmation bias sure.

let's try that

that's about 60 Hertz that seems a little much

okay we're gonna try one more time. We have basically put the piece of foil on

top of the speaker, we're dialing up the volume to... 11. -Well I mean it's a shower

speaker so

Cool So what do you think of that image there?

it's beautiful, it's gorgeous.

it has been like a couple years since I've looked at this code so I suspect I

might have forgotten something and I'm using it wrong but I want to point

something out here, which is that here's what we recovered and here is here is

the piano roll of the signal that we sent.

This looks like duh duh to me and

that's the duhhh yeah yeah

that's it that is the...

---dih duh dud din duhhhh din dih - yeah okay so let's

see if we can actually get it to hear that

-We can see it, we just can't hear it

I think I know what it is.

What is it?

I don't think my laptop can play those frequencies.

hold on a second let me get some headphones

We need real speakers or something.

What do you hear Abe?

hold on.

I hear shave and a haircut two bits

here, listen this.

All right. -here, I'll hold it.

thank you.

yeah.

and... pitch shift

[recovered sound of Shave and a haircut, two bits]

yep there you go, visual microphone. [laughter]

this was a basic proof

of concept but Abe showed that with more powerful equipment he could recover

human speech from outside soundproof glass.

Have you ever considered that your

computer is a physical system that gives off vibrations each key on the keyboard

produces a unique sound due to its unique location. In fact, research has

shown that audio recordings of typing reveal 96% of keystrokes accurately.

Now this portion of the video was sponsored by LastPass, an app that stores all your

usernames and passwords so you never have to type them in. And this prevents

people from stealing your passwords by say recording audio of your keystrokes

or just looking over your shoulder.

What's even better is the convenience of

never having to fill in usernames and passwords or getting locked out again

With LastPass you don't have to write remember or reset passwords.

You get unlimited password storage and free cross-device sync.

When you open an app

or site on your computer or on an iOS or Android device LastPass fills in your

username and password this saves you valuable brain space so put your

passwords on autopilot with LastPass.

Now something I particularly like is if you

upgrade to LastPass premium you get advanced multi-factor authentication and

it works like magic.

So click the link below to find out more about LastPass

and thanks to LastPass for sponsoring today's show