Placeholder Image

Subtitles section Play video

  • MALE SPEAKER: Welcome.

  • It's my pleasure to introduce Gil Weinberg.

  • He's a professor at Georgia Tech for the Center of Music

  • and Technology.

  • And I went to Georgia Tech a long, long time ago

  • and we didn't have a music department back then.

  • So when I visited recently, the folks I was meeting with in ECE

  • sent me over to meet Gil and he's

  • doing some really cool stuff with interactive computing

  • and musicianship.

  • So building machines that you can interact with,

  • not just from a dialogue perspective,

  • like the kind of thing I'm thinking about,

  • but more from a music perspective

  • where you want to jam and do some jazz

  • improvisation with a robot.

  • And he's going to talk about that in many other things

  • today.

  • So welcome.

  • GIL WEINBERG: Thank you.

  • [APPLAUSE]

  • Thank you.

  • Thank you for coming.

  • So I'm going to talk about three or four main projects.

  • Only three of them I showed in these slides.

  • The last one is a surprise.

  • We did the project which we just finished and I have some very,

  • very fresh new slides.

  • The reason I started to be interested

  • in robotic musicianship is that I'm

  • a musician before I became interested

  • in computation or in robotics, I was a musician and still am.

  • And I was always fascinated by playing in a group.

  • By being constantly ready to change what I'm doing.

  • By trying to build on what other people

  • are playing in real time.

  • Improvising in a group, a visual cues, auditory cues, of course.

  • So when I started to be interested in robotics,

  • I wanted to capture this experience.

  • Before I show you some of the efforts that I did,

  • maybe I'll show a short clip of me playing with a trumpet

  • player and trying to see the kind of experiences

  • that I was trying to recreate.

  • [VIDEO PLAYBACK]

  • [JAZZ MUSIC]

  • [END PLAYBACK]

  • So I think you've seen a lot of eye contact,

  • trying to build a motif that I'm not sure what is going to be

  • and trying to create something interesting with them

  • back and forth.

  • And the first robots that I tried to develop it

  • was building on these ideas.

  • But what I wanted to do is to have the robot understand music

  • like humans do.

  • The big idea that I started with,

  • was to create robots that listen like humans,

  • but improvise like machines.

  • Because I felt that if I want to push what music

  • is about through new, novel improvisation

  • algorithms and new acoustic playing,

  • I first have to have connection between humans and robots,

  • and that's why there is the listening like human part.

  • Only then would I be able to start to make the robot

  • play like a machine in order to create this kind of connection

  • and relationship.

  • I'll start with something very simple

  • that probably many of you are familiar with.

  • If I want a robot to understand me,

  • maybe the first simple thing that I can make him do

  • is understand the beat of the music that I play.

  • And here we use auto-correlation and self-similarity algorithms.

  • This is a piece from Bach.

  • You see the time is both on the x and on the y.

  • And you see that it's symmetric and by comparing

  • the [INAUDIBLE] to the algorithm you

  • can try to find sections that are similar

  • and detect the beat from that.

  • But what we try to do is to actually have it in real time.

  • And you see here my student Scott Driscoll

  • used this algorithm based on Davies and Plumbley

  • from Queen Mary.

  • And you see that's it becomes much more

  • sophisticated because it's not just analyzing the beat in Bach

  • or in the Beatles-- Scott is playing in real-time so he

  • is trying to get the beat, but then with Haile,

  • start to play with it.

  • Scott is trying to fit what he is

  • doing to what Haile-- Haile is a robot-- and back and forth.

  • So it's a little more complicated you see.

  • Sometimes they escape from the beat and get back to the beat.

  • And here's a short example.

  • Got the beat.

  • Now Scott will start faster and slower.

  • Faster , got it.

  • So as you can see, it loses it, it gets it back.

  • I think in a second, it will play slower which

  • shows how the system fails.

  • The next thing that I was trying to get

  • is to have the robot understand other things at a high level

  • musically.

  • Not just the beat but concepts that we humans

  • understand such as stability and similarity.

  • And basically we had a huge database

  • of rhythm generated almost randomly-- with some rules,

  • some stochastic rules.

  • And then we had a coefficient for stability and similarity.

  • Whenever Haile listened to a particular beat

  • there's some settings and coefficient

  • for stability and similarities.

  • At some point the robot actually decided by itself.

  • First the human on the side can change the similarity

  • and stability and create some output to bring rhythm back.

  • And this similarity is based on Tanguiane from 1993,

  • basically looking at the overlapping

  • onset between beats.

  • I can get more into this if you want, maybe later.

  • And this is as similarity algorithm

  • based on Desain and Honing.

  • [INAUDIBLE] between adjunct intervals

  • is what set how stable the rhythm is.

  • This is based on music perception studies

  • that I've been doing.

  • And basically, there is a mathematical procedure

  • where you compare each one of the notes

  • to the note that comes after it.

  • And at some point, after giving preference to one and two,

  • which are stable, you can get for every particular rhythem--

  • for example, this one, the quarter quarter,

  • two-eighths quarter-- a particular onset stability

  • by combining all of the ratios and getting something stable.

  • And here's a short example of Scott

  • playing with Haile and Haile trying

  • to understand the stability of Scott's rhythms.

  • And then based on a curve of similarity,

  • starting most similar then going less similar.

  • And basically a human put a curve.

  • But I can see a scenario where Haile could come up

  • with a curve by itself, trying to start with something

  • that Scott understands.

  • Scott is playing seven quarters, and then slowly

  • introduce new ideas.

  • And you can see how Scott actually listens to the robot

  • and at some point, building on what the robot is doing.

  • So Haile is building on what Scott is doing, obviously,

  • by looking at the stability and similarity.

  • But at some point, Scott is almost inspired.

  • Maybe inspired is too big of a word,

  • but that's a goal that Scott will come up with an idea

  • that he would come up with if he played with humans.

  • [VIDEO PLAYBACK]

  • [DRUMMING]

  • That's almost a new idea.

  • Scott is building on it.

  • [END PLAYBACK]

  • And this is a darbuka drum player concert.

  • You will see how the professional darbuka

  • player, actually from Israel, is kind of surprised.

  • But I think his facial gestures were interesting for me

  • because I think he was surprised for the better.

  • And at some point you'll see how all of us are playing

  • and Haile tries to get to beat.

  • So we combine the stability similarity

  • and beat detection into a drum circle.

  • [VIDEO PLAYBACK]

  • [DRUMMING]

  • This is call and response.

  • Later it will be simultaneously.

  • [DRUMMING]

  • And now, what it does, it listened to these two drummers,

  • and tricked the pitch from one drummer,

  • and the rhythm from another.

  • And the other arm, most from pitch

  • and the timbre of the two players.

  • [END PLAYBACK]

  • So we played the rhythm that one player played,

  • and the pitch-- well, it's not really pitch, it's in the drum,

  • but this is lower and this is higher next to the rim,

  • and tried to create something that is really

  • morphing between these two.

  • Again, things that humans cannot do and maybe shouldn't do,

  • but here something interesting can come up.

  • Another thing that I was interested in is polyrhythm.

  • It's very easy for a robot to do things that humans cannot.

  • Sometimes I'll ask my student to clap.

  • I will not ask you to clap.

  • I'll give you an example.

  • I think there is two main reasons here.

  • [SPEAKING RHYTHM]

  • This is nine.

  • [SPEAKING RHYTHM]

  • I don't ask you to clap but sometimes I would.

  • It was [SPEAKING RHYTHM] seven.

  • And then I asked my students to do

  • the nine in one hand and the seven

  • in another hand which I will definitely not ask you.

  • But see how Haile here captured--

  • decided to record the rhythm.

  • So it records the nine.

  • He choose the nine and the seven, and at some point

  • he introduced them in polyrhythmic, interesting

  • rhythms.

  • [VIDEO PLAYBACK]

  • [DRUMMING]

  • And we add more and more rhythms over it.

  • [END PLAYBACK]

  • And I don't know if know, but Pat Metheny had a project

  • that he used robots in.

  • He came to our lab and I explained it to him,

  • I showed it to him.

  • I said, this is something no humans can do.

  • And he said, sure, my drummers can do it.

  • And he can also do the four with his leg, and another three

  • with another leg.

  • So I think everyone can do it except maybe Pat Metheny's

  • drummer.

  • And here is something that's at the end of concert,

  • it's obviously, you know, fishing for cheers.

  • We just have a little short MIDI file where

  • we playing with it together.

  • Unison always works, so I'll just play this.

  • We grabbed the nine and seven.

  • [VIDEO PLAYBACK]

  • [DRUMMING]

  • And that's from a performance in Odense in Denmark.

  • They had a robot festival.

  • [END PLAYBACK]

  • So the next project was Shimon.

  • And actually I have an story about this

  • because I put a video of Haile and Scott playing,

  • the first one, on my website.

  • It was before YouTube, or before I knew about YouTube.

  • And someone grabbed it and put it on YouTube,

  • and then CNN saw it and they asked to come and do a piece.

  • And when they put a piece, the next day

  • I got an email from the NSF, from the NSF director

  • who said we saw your piece, please submit a proposal

  • and continue to develop that.

  • Rarely happens.

  • Never happened since.

  • I tried.

  • I put so many videos--

  • And this is the next robot that we came up

  • which adds multiple things.

  • The main thing that it adds is the concept of pitch.

  • It plays a marimba.

  • And the second aspect it adds-- we're

  • talking about the personal connection, gestures,

  • visual cues-- is the head.

  • And many people ask me, why the head?

  • It doesn't play any music.

  • In a second I'll show some of the utilization

  • of what the head does.

  • And of course it has a camera.

  • That's the first question you get

  • about any head, robotic head.

  • So the name is Shimon, and here is the first example

  • of my post-doc, Guy Hoffman.

  • He's now a professor in Israel.

  • [VIDEO PLAYBACK]

  • [PLAYING PIANO]

  • It shows how a headless Shimon responds

  • to dynamics, to expression, and to improvise.

  • Look at Guy, how he looks at him as

  • if his kid did something nice.

  • [END PLAYBACK]

  • [LAUGHTER]

  • So when Guy played fast, the robot responded.

  • Simple, just tempo detection.

  • And then loud and louder, and tried

  • to get some understanding of tonality

  • and of giving the right codes.

  • And we had some studies done about it.

  • If you remember the first code was [SPEAKING CODE]

  • Three codes.

  • And we tried to see if, by looking at the robot,

  • at the gestures, with the arms before the head,

  • if the synchronization between the human and the robot

  • is better.

  • And you see that the first code is maybe easier.

  • The second code, very difficult to synchronize

  • in terms of milliseconds.

  • The third code, you kind of adjust yourself.

  • So it's a little better.

  • And we have multiple conditions, one only visual, one audio,

  • and one with synthesized sound.

  • And you can see that the visual is better

  • because it's less milliseconds of delay

  • between the human and the robot in both of the conditions,

  • precise and imprecise.

  • Especially in imprecise, you'll see it's better.

  • For imprecise, the robot could play 50 milliseconds,

  • before or after.

  • And just looking at the robot helped

  • it become a much more synchronized,

  • tight performance.

  • And it works much better at slower tempos, less than 100

  • BPM as in more than that.

  • Then we added some other aspects.

  • We said, OK, we have a head, let's do what the head can do.

  • Actually we didn't think of this way.

  • We thought of what the head can do and then built the head,

  • but for the story.

  • And what it can do is multi-modal detection.

  • We had a camera there and we tried

  • to detect the beats of the hip hop artist

  • to coordinate with the beat detection of the music.

  • And you see how the multi-modal detection work.

  • And also just to help them feel together--

  • and this is Guy Hoffman's work-- and develop gestures

  • that will make everyone feel that they are

  • synchronized with each other.

  • [VIDEO PLAYBACK]

  • [MUSIC PLAYING]

  • So that's just the audio.

  • [MUSIC PLAYING]

  • Now double-checking with the beat of the head.

  • And of course blinking never hurts.

  • The blinking is random.

  • [END PLAYBACK]

  • All right.

  • OK.

  • So that was a little bit about listening like a human

  • and maybe grooving like a human.

  • And then I started to think more about creativity, or machine

  • creativity and what it means.

  • And actually, this is the first time I've showed it.

  • I hope it's not streamed, I think it's not streamed.

  • This is a TED Ed video that I just finished doing.

  • I didn't put in online yet, so it's a part of it.

  • I didn't show it yet.

  • But it shows one of the ideas I had

  • for creating machine creativity using genetic algorithms.

  • And the idea was that if have a population of motifs, maybe

  • random motifs, and then I have a set of mutations

  • and cross-breeding between these motifs,

  • and have a fitness function that is based on human's melody,

  • then I can create new, interesting music

  • that has some aesthetics for humans in them.

  • Because it's easy to create melodies

  • that were never created before.

  • Most of them will not be listenable.

  • The question is, how do you make meaningful and valuable

  • melodies?

  • And in the video, you'll see a little reference

  • to the Lovelace Test, which probably some of you

  • know about maybe I'll just say that the Lovelace Test is based

  • on Ad Lovelace, the first programmer,

  • or is considered to be the first programmer,

  • trying to-- instead of intelligence like the Turing

  • test, try to determine when the robot is creative.

  • And she said something like, a robot will never be creative

  • before it generates ideas that its own programmers cannot

  • understand how they came to be.

  • And then in 2001 there was a group of scientists,

  • some of them continued to develop [INAUDIBLE] that

  • came up with this Lovelace test, which is mostly

  • a theoretical thought experiment of creating something

  • like that.

  • But even if we do something like that,

  • I didn't think that the music that would come be valuable.

  • And I'll show this little TED Ed clip

  • and see how I went about it.

  • [VIDEO PLAYBACK]

  • -So how can evolution make a machine musically creative?

  • Well, instead of organisms, can start

  • with an initial population of musical phrases.

  • [MUSIC PLAYING]

  • GIL WEINBERG: They put the drum beat.

  • It's not mine.

  • -And a basic algorithm that mimics reproduction

  • and random mutations by switching some parts,

  • combining others, and replacing random notes.

  • Now that we have a new generation of phrases,

  • we can apply selection using an operation called a fitness

  • function.

  • Just as biological fitness is determined

  • by external, environmental pressures,

  • our fitness function can be determined

  • by an external melody, chosen by human musicians or music fans

  • to represent the ultimate beautiful melody.

  • [MUSIC PLAYING]

  • GIL WEINBERG: Chopin.

  • -The algorithm can then compare between our musical phrases

  • and that beautiful melody, and select only the phrases

  • that are most similar to it.

  • Once the least similar sequences are weeded out,

  • the algorithm can re-apply mutation and recombination

  • to what's left, select the most similar or fitted ones, again

  • from the new generation and repeat for many generations.

  • [MUSIC PLAYING]

  • The process that got us there has so much randomness

  • and complexity built in, that the result

  • might pass the Lovelace Test.

  • More importantly, thanks to the presence of human aesthetic

  • in the process, we'll theoretically

  • generate melodies we would consider beautiful.

  • GIL WEINBERG: All right.

  • [END PLAYBACK] So here's an example

  • of how this is used in real-time.

  • Chopin is not [INAUDIBLE] with fitness function

  • as you see here.

  • But actually, the saxophone player and myself

  • playing piano-- I programmed Haile with multiple motifs

  • and every time it listened to one of our musical motifs,

  • it used dynamic time warping to see

  • what's most similar and come up with a variation.

  • And it stopped after something like 50 generations

  • because, of course, if it ran more than 50

  • it would become exactly what we played.

  • So we just stopped one time to create something

  • that is similar to what we did, but a little different

  • and based on the, let's call it, original DNA of the population

  • that we started with.

  • And [INAUDIBLE] Haile will never use it with Shimon.

  • This is Haile when we first tried to have Haile with pitch.

  • So the whole idea of creating rich, acoustic sound

  • is not there.

  • We put the little toy vibraphone below Haile.

  • So excuse the quality of the sound.

  • [VIDEO PLAYBACK]

  • [MUSIC PLAYING]

  • And listen to how all three of us

  • are building on each other's ideas.

  • Saxophone, piano, and robot.

  • [MUSIC PLAYING]

  • [END PLAYBACK]

  • Yeah.

  • It's difficult to evaluate it because of quality of sound.

  • But that was the idea of trying to put it

  • in a real-time scenario.

  • Another idea where I tried to have

  • it play more like a machine was style morphing.

  • And as you can understand by now, I play jazz, I like jazz,

  • I like Monk on the left and Coltrane on the right.

  • And I thought, wouldn't it be awesome

  • if we tried to morph between Coltrane and Monk

  • and create something that they themselves will never

  • be able to create.

  • They cannot morph into each other.

  • And maybe something interesting will come up

  • that is just a combination of both their styles.

  • And we used HMMs here, basically analyzing

  • a lot of improvisation by both of these musicians

  • and trying to see what the likelihood is that at any given

  • note, what the next note would be, what the next rhythm would

  • be, just a second order.

  • And then with an iPhone I can play a little melody.

  • [VIDEO PLAYBACK]

  • And this is a little app that we built regardless.

  • And at some point when the melodies that I play,

  • of course it can be piano too, is ready,

  • I will send it to Shimon, and then I

  • will be able to morph between my own melody, Monk's

  • style, Coltrane's style and with two styles with saw

  • a different kind of balances between these two

  • improvisations and myself.

  • [MUSIC PLAYING]

  • If you know Monk's syncopation in clusters.

  • If you know Coltrane, you know the crazy, fast licks.

  • And this is somewhere in between.

  • And of course, one doesn't play saxophone over piano,

  • so it doesn't sound like any of them.

  • [END PLAYBACK]

  • But at least the notes are based on the improvisation.

  • Some other quick examples of things that we tried.

  • Here we had the connect because we

  • wanted to get more than just what

  • this simple camera in the head had.

  • And we tried to have motion-based improvisation.

  • You see the Atlanta symphony percussionist, Thomas Sherwood,

  • doing some tai chi-like movement and trying

  • to have Shimon improvise.

  • And actually Tom is playing an electronic marimba

  • and Shimon has a system in which sometimes it will

  • repeat what is listened to.

  • Sometimes it's [INAUDIBLE], sometimes

  • it will introduce new ideas, a stochastic system that's

  • supposed to inspire Tom to play differently

  • than he would otherwise.

  • [VIDEO PLAYBACK] So that's the tai chi part.

  • [MUSIC PLAYING]

  • And it wasn't easy for it to repeat, exactly this pattern.

  • It's very limited with the four arms and where they can be.

  • I'll show some path planning strategies later.

  • So not every motif was captured perfectly.

  • And now it starts to introduce new motifs.

  • It will repeat and then create some kind

  • of a musical interaction.

  • [END PLAYBACK]

  • This is my student, Ryan Nikolaidis, the entire motif.

  • [VIDEO PLAYBACK]

  • And Shimon is developing this motif.

  • This is part of the US Science and Technology Fair

  • in Washington.

  • [MUSIC PLAYING]

  • [END PLAYBACK]

  • And for something else, some of them just, you know,

  • they love dubsteps.

  • And the machine learning maybe interesting or not,

  • but dubsteps are there.

  • So it's just a dubstep piece that one of our students

  • developed for Shimon.

  • [VIDEO PLAYBACK]

  • [MUSIC PLAYING]

  • So it can do all kinds of stuff.

  • [END PLAYBACK]

  • Here we used vision-- and you see this green ball

  • on the conductor's baton-- and tried

  • to combine the input from conductor

  • and from the audio analysis from the orchestra to improvise,

  • based on the chords that the orchestra is playing.

  • [VIDEO PLAYBACK] That's the Georgia Tech student orchestra.

  • [MUSIC PLAYING]

  • [END PLAYBACK]

  • All right.

  • And I think this is my favorite clip.

  • This actually was a PSA of Georgia Tech for three or four

  • years.

  • You know that we come from Georgia

  • where dueling banjos are a big thing, at least in some movies.

  • So we tried to do dueling marine buzz if you like,

  • and present kind of the dueling aspect and Shimon

  • with the gestures-- and this is again,

  • Guy Hoffman's work-- project some personality.

  • Hopefully better than the movie.

  • [VIDEO PLAYBACK]

  • [CALL AND RESPONSE MUSIC PLAYING]

  • [END PLAYBACK]

  • I think I have two more clips before we

  • go to the next project.

  • I just got the request, how about having a robot band?

  • They listen to each other and create something.

  • I'm sure robotics people here are being asked about,

  • will robots replace humans often?

  • And in my case it's really painful sometimes

  • because they come to me and tell me, how dare you?

  • Music, really?

  • Like, OK cars, OK cleaning carpets,

  • but you're going to take human from us

  • and let the robots play it.

  • So I never tried, I'm not interested in having robots

  • playing together by themselves.

  • It's only and always have been about inspiring human

  • and creating something interesting for humans.

  • And here you can see a quick clip

  • where you see bit detection and [INAUDIBLE] improvisation

  • between two students and two robots.

  • [VIDEO PLAYBACK]

  • [MUSIC PLAYING]

  • He's not so good with the beat, by the way,

  • so-- strictly for Haile to detect it.

  • But it got it.

  • And now it start to improvise with the other arm.

  • What you hear here is a student that we don't

  • see, playing the synthesizer.

  • [MUSIC PLAYING]

  • And now Shimon, before we had a head,

  • actually we used someone else's CGI head.

  • Andrea Tomaz from Georgia Tech.

  • I'm dropping back and forth in time here.

  • [END PLAYBACK]

  • All right.

  • So that's a robot-robot interaction for you.

  • I promised to talk a little bit about path playing.

  • There are multiple ways in which these forums

  • can be next to the piano or next to the marimba.

  • In this case I would just put a piano.

  • And we have state machines that represent each one

  • off these states.

  • Given any melody, what would be the most reasonable path

  • to take?

  • And the inputs that we have is a curve

  • that humans play for tension-- melodic tension

  • or harmonic tension, or rhythmic stability.

  • And I'll show it here.

  • This is some the pitch or note density.

  • And using the Viterbi algorithm, actually [INAUDIBLE] Viterbi,

  • we're trying reduce the space and try

  • to come up a reasonable path to create improvisation,

  • following these curves.

  • So it's basically developing, as I say,

  • an embodied musical mind, where not only of the music

  • it has to play but also it's own body.

  • The whole idea of embodiment is a bit part

  • that robot musicians have and that software musicians

  • or machine musicians don't have.

  • Only robotic musicianship has.

  • So we are trying to come up with these algorithms, and actually,

  • my student, Mason Bretan programmed really extensive

  • jazz theory into Shimon.

  • So now it can listen to chords that Mason, in a second

  • will play, and then try to improvise,

  • using this both music and physical limitations

  • that he has using Viterbi.

  • He tried some other, but the Viterbi worked pretty well

  • and this is his improvisation.

  • This clip actually went viral kind of for our fill.

  • There were around 50k viewers last month.

  • [VIDEO PLAYBACK]

  • [MUSIC PLAYING]

  • Mason fed all the chords offline.

  • All the calculation of what's going to be play

  • was done offline.

  • [MUSIC PLAYING]

  • [END PLAYBACK]

  • So I don't know if it's got [INAUDIBLE] quality

  • but it's not bad.

  • I was happy.

  • All right, so we all think it's very important, the path

  • planning and creativity and so on.

  • But it's important to realize that this can

  • be taken in a different light.

  • And I think if you're being made fun of for what you do,

  • this guy is probably the best guy to make fun of you.

  • Some people tell me, don't show it,

  • but I think-- I'm proud of being made fun of by Stephen Colbert.

  • [VIDEO PLAYBACK]

  • -Jazz robots, which combines two of the biggest

  • threats to our nation-- jazz and robots.

  • He's been programmed to dig it too.

  • [LAUGHTER]

  • Bobbing his head to the beat and checking in

  • with this fellow players.

  • Meaning that long after the robot uprising kills

  • all humans on earth, there will still

  • be someone on the planet pretending to enjoy jazz.

  • [LAUGHTER]

  • [END PLAYBACK]

  • OK.

  • Shimi was an effort to take Shimon and make

  • it a little smaller.

  • Actually it almost became a product.

  • There was a failed Kickstarter there.

  • And it focused on the gestures, not so much

  • from the acoustic reproduction of sound,

  • which is part of the reason that I went into robotics because I

  • kind of grew tired by sound from speakers, no matter

  • how great your hifi system is.

  • I was missing the acoustic sound which

  • is one of the reasons I went to robotics.

  • So here we don't have acoustic sound,

  • but we have a bunch of other applications.

  • We call it a robot companion that can understand music.

  • It can connect to your Spotify.

  • It can dance, it can do the bit detection.

  • It has a set of algorithms to get data from the audio

  • and respond accordingly.

  • And here's a clip.

  • [VIDEO PLAYBACK]

  • [MUSIC COLDPLAY, "VIVA LA VIDA"]

  • GIL WEINBERG: In this case it takes the music from the phone,

  • but we also have an application that takes it from Spotify.

  • And the camera and the phone can see if you don't like it.

  • And better to say no, because the music is loud.

  • [TAPPING "DROP IT LIKE IT'S HOT"]

  • [MUSIC SNOOP DOGG, "DROP IT LIKE IT'S HOT"]

  • GIL WEINBERG: [INAUDIBLE] speaker dock.

  • You can see the dancing speaker dock.

  • [MUSIC SNOOP DOGG, "DROP IT LIKE IT'S HOT"]

  • [END PLAYBACK]

  • Here is a couple of other applications, some using

  • speech, and some using audio analysis.

  • That stuff is audio analysis.

  • Looking for a beat in what Mason is playing.

  • [VIDEO PLAYING]

  • [PLAYING GUITAR]

  • GIL WEINBERG: No, there wasn't anything there.

  • [PLAYING GUITAR]

  • You get it.

  • It has a system of gestures.

  • [END PLAYBACK]

  • GIL WEINBERG: And here, using Google Voice.

  • Mason is connecting to Spotify.

  • [VIDEO PLAYBACK]

  • -Look at me.

  • Can you play something happy?

  • GIL WIENBERG: And we have some algorithm

  • to determine happy, disco, rock.

  • [HAPPY MUSIC PLAYING]

  • -Can you play disco?

  • [MUSIC BEE GEES, "SATURDAY NIGHT LIVE"]

  • GIL WIENBERG: Of course this.

  • So a different set gestures for different genres.

  • All right.

  • [END PLAYBACK]

  • Annie Zhang, another student, worked on the Shimi band,

  • trying to create collaborations between three Shimis.

  • And she looked at a lot of [INAUDIBLE], music

  • [INAUDIBLE] elements you would get from the audio.

  • Anything for the beat detection to the beat to the arm

  • is for the energy.

  • MFCC for vocalist to see if there's vocals there or melody.

  • And then this is her interface.

  • And she then came up from the low level features,

  • detecting some high level features,

  • beats, energetics, melodics, vocalness,

  • percussiveness of the sound.

  • And then she actually went and studied [INAUDIBLE] system

  • and tried to create a dancing grammar

  • for all of these gestures that will connect

  • between how Shimi is dancing.

  • And here are some of her ideas for that.

  • And a whole system of different ways in which three Shimis can

  • dance together-- solo, having the wave, two

  • in synchronicity from both sides, and the main one

  • is different.

  • So a full system that later allowed her to have this.

  • [VIDEO PLAYBACK]

  • [MUSIC PLAYING]

  • And every time it moved different, the gestures.

  • [END PLAYBACK]

  • And even in a very noisy environment in a demo--

  • [VIDEO PLAYBACK]

  • [MUSIC PLAYING]

  • [END PLAYBACK]

  • Here too, we try to have an embodied musical mind

  • and have an understanding of what are the important music

  • notes in a piece.

  • Tonally, maybe they're more stable,

  • maybe it's the end of a phrase that goes up and so on.

  • And then combine it with some understanding

  • of what are the positions in which what

  • can move to particular places, being

  • aware of its own limitation in terms

  • of the time of the gestures.

  • So here you see the red dots that

  • represents a particular end point

  • in gestures that try to be based on the important notes

  • melodically.

  • Again, the combination of being aware of the embodiment

  • and of musical theory and creating a system

  • that, I think Mason here put some titles

  • that you can read as he plays.

  • [VIDEO PLAYBACK]

  • [PLAYING GUITAR]

  • [END PLAYBACK]

  • And as you can see, Mason is extremely talented,

  • and he is looking for an internship this summer.

  • [LAUGHTER]

  • Just to end the Shimi part-- maybe

  • I'll just play this little clip.

  • I couldn't make a Shimi without letting it dance to this song.

  • [VIDEO PLAYBACK]

  • [MUSIC STAR WARS, "CANTINA SONG"]

  • [END PLAYBACK]

  • And that's the last project that we just finished last year.

  • I actually took a break.

  • I took a sabbatical year off.

  • And just when I was about to go to the sabbatical

  • I got an email from Jason Barnes.

  • He's an amputee drummer.

  • He lost his arm, now it would be like three years ago.

  • When we met it was two.

  • And he saw some of the videos online

  • and he was really devastated.

  • There's a clip online.

  • You can see him talking about how he thought he had nothing

  • to live for if he cannot drum when he lost his arm when he

  • was electrocuted in some weird accidents on a roof.

  • And then he saw these video and he emailed me and asked me,

  • can you use some of the technology

  • to build me a robotic arm?

  • And, of course, that was very intriguing for me.

  • What he really wanted was to be able to control the grip.

  • Because he had the arm up all the way up until here so

  • he can still hit.

  • But for a drummer, it's very important for them

  • to have control of the grip, tight or loose

  • to create different kinds of drum strokes.

  • And that was very interesting for me.

  • And the people I usually collaborate

  • with couldn't come through, so I-- I

  • don't know if anyone here is from Meka,

  • but I did contact Meka who built this arm.

  • And that was great but because I'm interested in improvisation

  • and creativity, I really wanted to have this aspect

  • of my research in this arm.

  • So I asked Jason if he minded that we had two

  • different sticks, not only one.

  • One was all about grip, so you can have your expression

  • and play it as he wants.

  • The other one actually would listen and improvise

  • and will listen to the music and listen to other musicians

  • and try to maybe inspire you to create something else

  • and become better or try to create music that he

  • would never create otherwise.

  • Let me show you a couple of clips

  • first of the basic interaction of just controlling the group.

  • And what we discovered after Phillip from Meka

  • built the arm is that we don't need just to hold the stick,

  • we can actually use it to generate hits.

  • And after some work on noise reduction,

  • we were able to some pretty quick responses,

  • very low latency so Jason could just

  • flex his muscle, in the muscle and actually create

  • a beat, in addition to using his own arm.

  • So there's some aspect of a cyborg

  • there because part of the gesture

  • is his own body and part of it is electronically

  • based on his muscle and maybe other muscles,

  • not only the forearm muscle.

  • So actually, this took eight months.

  • It was amazingly fast, again, thanks to Meka.

  • And this is the first time Jason tried it.

  • MALE SPEAKER: Next.

  • Next.

  • Next.

  • GIL WIENBERG: And that's where we really

  • got-- I didn't expect that, that we can create 20 hertz.

  • And you know, my favorite polyrhythm games.

  • I'll show more of the musical context later.

  • But let's first-- see, here we didn't

  • have the EMG working yet.

  • It's a little knob that changes between tight and loose,

  • but just to see the effect.

  • This was a tight hit.

  • [VIDEO PLAYBACK]

  • [DRUMMING]

  • Assuming the holding of the stick tight,

  • this is a loose hit.

  • [DRUMMING]

  • And this is a role tight.

  • [DRUMMING]

  • [END PLAYBACK]

  • Many more expression than he had before.

  • He couldn't do all of this.

  • And actually, he was in tears the first time.

  • He said, I didn't play this well since I lost my arm.

  • Which was the first time I actually worked with people.

  • And it was pretty powerful.

  • But, here, you can see the second stick.

  • The kick is very short.

  • Look here, we have an accelerometer

  • and when he wants-- only when he wants,

  • he can lift the arm quickly and the second stick will pop up.

  • It will be really fast.

  • OK.

  • And then we can start with the improvisation

  • and the creativity, the machine creativity.

  • Before we do that we can just have a computer

  • generate strikes.

  • So nothing to do with him.

  • Even though it's part of his body,

  • suddenly someone else is generating hits for him.

  • And even Pat Metheny's drummer cannot do.

  • Not with one arm.

  • And this is, again, playing with roll,

  • with tumble, because it's so fast it becomes actually color

  • and not even rhythm.

  • And this is what I call cyborg drumming where

  • you combine some of the computer generated hits

  • with his own hits.

  • He decides where to put the arm, where not, what to choose.

  • Create a human machine interaction

  • and musical collaboration.

  • And at some point-- actually I wrote an NSF proposal

  • about it-- we want to create some human brain machine

  • interaction.

  • But now some of the most sophisticated

  • stuff we're doing with pedals.

  • With his leg, it controls dynamics and expressive

  • expression elements.

  • And when this first clip went online,

  • it also kind of became viral.

  • And I got a lot of emails and one of them

  • was from a guy called Tom Leighton who actually

  • played with Richard Feynman.

  • I didn't know that Richard Feynman was a drummer,

  • playing hand drums.

  • And he said, I have a bunch of recordings from Richard Feynman

  • and I would love for you to play with it

  • and see if you could do something with it.

  • And of course I jumped on this opportunity.

  • And what we did here is we put some of the rhythm

  • that Richard Feynman played in his arm,

  • so he had a chance to play with Richard Feynman in a weird way.

  • And first you'll see him hovering over the drum,

  • just listening to the rhythm, and then he

  • starts to decide what drum, when he whats to lift the symbol,

  • and so on so on.

  • So it created all kinds of interaction

  • with someone who is not with us anymore.

  • And I think it also was a powerful experience for him.

  • [VIDEO PLAYBACK]

  • [DRUMMING]

  • That's the rhythm by itself, just hovering.

  • And now he adds to it with the other arm.

  • [DRUMMING]

  • Now he starts to decide where to put it.

  • He's hovering but now we drew this

  • as part of his gestures and movements.

  • [END PLAYBACK]

  • Here we have analysis of the chords

  • and we try to create similar connections,

  • a very intimate connection between the guitar player,

  • Andrew, and Jason.

  • Different chord that he plays.

  • We analyze the chord and change the rhythms

  • that Jason is playing.

  • So suddenly he has to respond, not only

  • to the arm that is generated by the computer

  • or generated by his muscle, but generated or controlled

  • by a different musician.

  • And you see the whole eye contact and the whole,

  • trying to create something together,

  • happening between them two.

  • [VIDEO PLAYBACK]

  • [DRUMMING AND GUITAR]

  • It was one of the first times that he

  • did it and it's not tight, but it becomes tighter

  • with every rehearsal and every performance.

  • [DRUMMING AND GUITAR]

  • [END PLAYBACK]

  • And here he is, in context, trading four for those of you

  • who play jazz.

  • You play four bars and then someone else

  • improvises four bars and back and forth.

  • Here, the arm listens to the improvisation for four bars

  • and then Jason gets to improvise.

  • And the arm is responding, improvising, [INAUDIBLE]

  • classic manipulation of what it heard in the four bars before.

  • [VIDEO PLAYBACK]

  • [JAZZ MUSIC]

  • [END PLAYBACK]

  • And I have longer clips for all of this online.

  • This is the latest project.

  • I just received this videos from Deepak, my student.

  • Just yesterday he just finished doing that.

  • And if you may have noticed, a lot

  • of the expressions that if you are a drummer

  • you probably know that you can hit in much more versatile ways

  • than just a hit or not a hit.

  • There's a lot of multiple bounces

  • and a lot of expression in it.

  • Here is Deepak who is a drummer.

  • My students need to be musicians,

  • and in this case roboticist.

  • Actually he was just accepted to a robotics program

  • at Northwestern.

  • The first master student from music technologies

  • that actually got into a robotics program.

  • It shows the different kind of drumming techniques

  • that he has.

  • [VIDEO PLAYBACK]

  • So all of this richness and expression, we couldn't have.

  • So Deepak had to go and learn PID control, which

  • again, is not one of the classes that we

  • offered in music technology.

  • [END PLAYBACK]

  • And he tried to come up with a model for the talk,

  • taking into account gravity.

  • And something he had to know, it's a very complex system.

  • The bounciness of the drum, for example,

  • is not being taken into account.

  • But he tried to put everything in simulation.

  • And with some coefficient of restitution

  • here is 0.85 just as a constant.

  • And he tried to create all kind of bounce behaviors.

  • Some of them are just based on humans and bouncing balls kind

  • of techniques.

  • But some of them are actually bounces that humans will not

  • create or are not likely to.

  • You see this is much faster.

  • At its end, it's lifting for the next hit.

  • And here you have a high to low force profile you see.

  • This hit is softer than the next one.

  • So you really have to change-- it's time varying talk changes.

  • And he put it into a model and created a PID control.

  • And this is the first clip.

  • He just did it yesterday morning and sent it to me

  • just before I left.

  • [VIDEO PLAYBACK]

  • So hopefully this will be embedded into some

  • of the new performances.

  • And we're going to France to perform in May.

  • Hopefully Jason can play a much more wide range of expression.

  • This is some of the future works that the NSF proposal that I

  • have wrote is going to address.

  • Am I'm working with a neuroscientist

  • from Northwestern, [INAUDIBLE].

  • And we are trying to get some understanding

  • about anticipation.

  • Because there are latencies in the system,

  • and no one isn't focusing on volition detection, which

  • is basically-- your brain is shooting,

  • sometimes seconds before your motor cortex

  • actually knows that you're about to hit.

  • In a way, you know you're going to do something

  • before you know you're going to do something,

  • or a part of your brain knows about it.

  • I tried to get this data.

  • We did some experiments with 36 EEG electrodes,

  • and there maybe some way to get information

  • before the hit is going to be made and then

  • better anticipation algorithms.

  • So if he hits with both hands, or hits

  • with both times-- this hand hits by itself,

  • it will be on time because we have some information.

  • We did some machinery to detect different kind of strokes

  • and we saw that the sum-- it's promising.

  • It might be tricky and we have some outliers,

  • but we can pretty much detect single strokes

  • versus double strokes, which is something else that we

  • can try to implement.

  • And this is one way it will probably never look like.

  • But this picture is in the proposal.

  • Justin is an example.

  • The trick here, when you create such cyborgs

  • is how not to interfere with what you're doing anyhow.

  • But as you can see, I'm jumping into third arm scenarios.

  • So half of the proposal would be working with Jason

  • on his own arm, creating this.

  • But the second part is, let's try, available to anyone,

  • whether you are disabled or abled, and have a third arm.

  • And the promise, the spiel is that if we

  • have anticipation algorithms and timing and synchronization,

  • that works in music-- and music is a very, very time demanding

  • and medium.

  • You know, you hear five milliseconds

  • that what you heard is two.

  • So if we can pass synchronization in music,

  • we can anticipate and help in all kind of scenarios--

  • on operation rooms or fixing a space station with a third arm.

  • So how to put it on the body in a way

  • that you can be completely-- first,

  • the cognitive load, how to control it.

  • So we thought about the brain, the organ of muscles

  • that we can still try to hit.

  • But the muscles are used for legs for the drums

  • and for arms for the drums.

  • This is a different kind of design for the future brain

  • machine control.

  • And I'll be happy to go for questions right now.

  • Thank you very much.

  • [APPLAUSE]

  • AUDIENCE: I think you alluded to this,

  • but can you say more about why you

  • think the physical manifestation of a striking thing

  • is important.

  • Are there other artists actually getting information

  • from the motion of the robot or is it just for looks.

  • GIL WIENBERG: So one study showed that it

  • helped with synchronization.

  • That's the graphs that I showed when you see the arm moving,

  • when you see the beat.

  • You'll better synchronize your gestures with it.

  • I'm a big believer in embodiment.

  • I believe that that's why we go to concerts as opposed

  • to listening to music at home.

  • We want to see how people interact, not only by music

  • but by gestures.

  • A drummer and a guitar will end the song together.

  • And there's a lot about interacting with gestures

  • and being aware of your body as a musician that I think you

  • cannot do without it.

  • Of course, all of it can two iPhones that play this music.

  • But I think we're going to miss the whole musical experience.

  • It's something really deep and strong in embodiment

  • in the musical experience.

  • AUDIENCE: So two sort of related questions, if you don't mind.

  • The first is, have you tried sort

  • of persisting some information across several sessions for any

  • of these to try to have the robot develop

  • some kind of style more incrementally,

  • and sort of lock into certain patterns

  • that it prefers systematically over longer periods of time?

  • GIL WIENBERG: I thought about it.

  • We didn't get to it.

  • It's a great idea because musicians--

  • that's what they do.

  • There's length, there's lifelong learning.

  • And we're doing just small episodes.

  • There was a student that was interested in starting to do

  • but then he decided that a master's is enough.

  • He doesn't need a PhD.

  • But it's a great suggestion and it's definitely

  • something important for music.

  • AUDIENCE: And then related to that,

  • some composers or musicians have--

  • you talked about embodiment-- certain sort

  • of different bodily features like, I don't know,

  • someone like Django Reinhardt in jazz had only two fingers

  • to use on the guitar.

  • Rachmaninoff had very big hands.

  • So have you thought about sort of creating robots

  • that have different variations on what kind of motion

  • is available to them and seeing how that guides things, or sort

  • of, how more constraint can change?

  • GIL WIENBERG: So that's the latest work by Mason

  • with the path planning.

  • And he tried to come up with a system.

  • His PhD thesis will come up with a model of other musicians.

  • That's one of his main goals to continue

  • to develop the field of robotic musicianship.

  • Other people who build robots will

  • be able to use this kind of path planning ideas

  • and put as a coefficient the different limitations

  • that they have.

  • AUDIENCE: OK.

  • Thank you.

  • GIL WIENBERG: Yeah.

  • AUDIENCE: There was only one instance

  • where you mentioned a robot knowing

  • anything context-specific or genre-specific.

  • For most of the examples, it was instead,

  • everything that the robot should know

  • about how to generate the content

  • comes from the content it's receiving.

  • But then there was one where one of your students

  • programmed all of these jazz theories into it.

  • Do you have an aversion to telling

  • a robot genre-specific or context-specific information?

  • Or do you want it to be as blank of a slate as possible?

  • GIL WIENBERG: I don't have aversion for anything.

  • I come from jazz.

  • Jazz is about improvisation.

  • A lot of what I'm trying to do is [INAUDIBLE] improvisation.

  • So it doesn't necessarily fit maybe other genres.

  • Like with classical, yes, Bach can be improvised

  • and you can have all kind of counterpoints

  • based on improvisation.

  • But I can see a scenario when you're trying

  • to put classical theory in it.

  • I less see how it's going to be used for improvisation.

  • I think that it should be very interesting, yes.

  • AUDIENCE: Do you think you would ever do it negatively and say,

  • OK, well, these things, if you ever come across this,

  • this is not idiomatic?

  • You should avoid it?

  • I'm thinking, for example, in jazz of course,

  • but also for the hand drums, like, I don't know anything

  • about that style of music but would

  • you ever want to say like, in whatever mutation

  • or combination that comes up, this particular pattern

  • is seen as ugly?

  • GIL WIENBERG: I don't know if ugly, but--

  • AUDIENCE: Or not--

  • GIL WIENBERG: Not part of the grammar.

  • Yes, we have it in the jazz system.

  • AUDIENCE: OK.

  • Cool.

  • GIL WIENBERG: And I think it will

  • be interesting to try to see what happens when

  • we go to other genres, with maybe a different kind

  • of tonal theory.

  • AUDIENCE: I have a question back to the pathing analysis.

  • I'm curious if you have a measure of how much less you

  • are able to do when you do have these physical constraints,

  • versus if you could have the machine improvise it and watch

  • other people, and it would just have to produce tones.

  • Is there some sort of measure as to the limitations,

  • qualitatively versus what it could do, if it didn't have

  • the physical limitations of forearms that

  • can't cross over one another?

  • GIL WIENBERG: I think we can deliver--

  • I don't have a measure.

  • But I think a measure can be developed

  • and I think a measure can also be used for future robot

  • designers to build robots that have less of these limitations.

  • And when we came up with the four arms,

  • that this arm can never play this octave, we knew that.

  • And we knew that, in a way, it might play faster than human

  • and create 16 chords simultaneously which humans

  • cannot do.

  • But it will also be more limited than humans.

  • And I think when I play piano, I do path planning in a way.

  • If I don't have the fingers, if I just

  • to look at notes with the fingers,

  • I constantly, probably, do something

  • similar to Viterbi's beam search job algorithm or something.

  • Creating a measure in terms of coming up with different kind

  • of chords that you can or cannot play--

  • different kind of speed of phrases that you can or cannot,

  • you know.

  • They're what, if you noticed, one for the black keys,

  • for the accidentals and one line for-- so the limitation just

  • there and see if you can play clusters.

  • Can it play a cluster of C C sharp D E flat E?

  • It can't.

  • We don't have a measure to really come up

  • with a coefficient, but I think it's a good idea.

  • I think all of you have a lot of good ideas.

  • AUDIENCE: Thank you.

  • AUDIENCE: First, awesome talk.

  • Thank you very much.

  • The question I have has to do with,

  • kind of, wanting to play with these robots myself.

  • There's a bit of a bottleneck in that

  • your team is doing the hardware and also all the software.

  • Have you thought about moving towards a more

  • platform-based approach where other labs could write code

  • against your robot and be able to try it out?

  • GIL WIENBERG: That's a good one.

  • And by the way I'd like to give credit when credit is due.

  • My students are mostly computer science students.

  • The platform itself-- I have colleagues

  • come from the MIT Media Lab.

  • Guy Hoffman designed and Rob [INAUDIBLE]

  • designed the robots and Meka designed the latest one.

  • So this is not our research.

  • Probably the Meka robot is the best candidate

  • to become a platform.

  • The others are too flaky, and too specific.

  • But part of what I'm interested in

  • is also the art and the design and the coolness

  • of creating a new robot.

  • So it's just because I love it.

  • But I think it can be a good idea to try to, for the arm,

  • for example, to try to make a platform,

  • create code, create, I don't know, an API for other people

  • to try to play with.

  • Definitely.

  • I think this field is pretty new and there's

  • all kinds of [INAUDIBLE].

  • At some point, if enough people think it's important,

  • maybe we will use a robotic operating system.

  • But for now, we just focus on creating cool musical one-offs.

  • AUDIENCE: Thanks.

  • AUDIENCE: In the physical gesture robot,

  • you showed a picture of a score where

  • you had analyzed important points in the melody where

  • you might end a gesture.

  • But it was taking as input, raw audio.

  • Is that right?

  • As opposed to a MIDI file or a digital encoded score.

  • GIL WIENBERG: The input was a chord

  • that Mason and played from a MIDI piano.

  • So it got the symbolic notes from the MIDI.

  • AUDIENCE: Oh, OK.

  • And he was playing a guitar I thought at one point as well?

  • GIL WIENBERG: Yes, that was different.

  • The guitar was only played with shimmies.

  • Piano plays with Shimon.

  • The guitar for Shimi to depict the right--

  • and yes, it did repeat-- actually, as I mentioned,

  • this is offline.

  • Mason played this melody, it did it offline

  • and then it recreated it.

  • AUDIENCE: Oh, OK.

  • OK.

  • Thank you.

  • And one other thing.

  • About the auto-correlation beat detection.

  • Does that infer any time signature information?

  • Or how good is that at detecting strange time signatures?

  • If at all?

  • GIL WIENBERG: It's not as good.

  • Four quarters is much easier.

  • A big estimation, so it pretty much

  • focused on regular time, three, four.

  • Others would be more difficult.

  • AUDIENCE: Thank you.

  • MALE SPEAKER: I had one question.

  • I'm a classical musician and this is about online learning.

  • I mean, yeah, online real-time learning from the instructor.

  • So as a classical musician, you sit down, you have a lesson

  • and you have a instructor that teaches you,

  • based on what they're hearing, giving you feedback

  • and then they correct or improve your playing right then.

  • I don't know anything about instruction

  • for jazz improvisation.

  • I guess I've seen a bit of that.

  • Wynton Marsalis has a program right now

  • that you can watch where he's working with some young jazz

  • musicians and he's giving them feedback.

  • A lot of it's about communication and listening.

  • Have you thought about that sort of thing?

  • Not so much-- what you're showing here

  • is the actual improvisation, but if you wanted to give feedback

  • to the robot and say, you know, hey, why don't you try this.

  • Or that's a little bit too busy.

  • You maybe should be kind of simpler at this time

  • and then add some more color later or something.

  • I don't know.

  • GIL WIENBERG: The same students that

  • decided that MS is fine were thinking about learning.

  • Starting from nothing and learning getting some input

  • and reinforced.

  • And then develop a style that will develop over time.

  • We were thinking about.

  • We didn't get into it.

  • But yes, in jazz too, I think similar to classical music,

  • that's how it works.

  • First production of the sound.

  • You know, when I first played piano,

  • it was a lot of hits on my fingers and holding turtles.

  • If I have some more students using

  • learning-- not in the machinery applications of it,

  • but actually your robot that knows nothing and starting

  • to teach it slowly.

  • It can be very interesting.

  • Maybe very interesting new music perhaps that

  • wouldn't come this way.

  • MALE SPEAKER: OK, thank you very much.

  • That was awesome.

  • GIL WIENBERG: All right.

  • Thank you.

  • [APPLAUSE]

MALE SPEAKER: Welcome.

Subtitles and vocabulary

Click the word to look it up Click the word to find further inforamtion about it