Placeholder Image

Subtitles section Play video

  • [MUSIC PLAYING]

  • CHRIS KELLEY: Thank you so much for joining us.

  • My name is Chris.

  • I'm a designer and prototyper working

  • on immersive prototyping at Google,

  • and I'm joined by Ellie and Luca.

  • And today, we're going to talk about exploring AR interaction.

  • It's really awesome to be here.

  • We explore immersive computing through rapid prototyping

  • of AR and VR experiments.

  • Often, that's focused on use case exploration or app ideas.

  • We work fast, which means we fail fast,

  • but that means that we learn fast.

  • We spend a week or two on each prototyping sprint,

  • and at the end of the sprint, we end

  • with a functional prototype starting

  • from a tightly scoped question.

  • And then we put that prototype in people's hands

  • and we see what we can learn.

  • So this talk is going to be about takeaways we have

  • from those AR explorations.

  • But first, I want to set the table a little bit

  • and talk about what we mean when we say augmented reality.

  • When a lot of people think about AR,

  • the first thing they think about is bringing virtual objects

  • to users in the world.

  • And it is that.

  • That's part of it.

  • We call this the out of AR.

  • But AR also means more than that.

  • It means being able to understand the world visually

  • to bring information to users, and we call this understanding

  • the in of AR.

  • Many of the tools and techniques that

  • were created for computer vision and machine learning perfectly

  • complement tools like ARCore, which is Google's AR

  • development platform.

  • So when we explore AR, we build experiences

  • that include one of these approaches or both.

  • So this talk is going to be about three magic powers

  • that we've found for AR.

  • We think that these magic powers can help you build better AR

  • experiences for your users.

  • So we're going to talk about some prototypes that we've

  • built and share our learnings with you

  • during each of these three magic power areas during the talk.

  • First, I'll talk to you about context-driven superpowers.

  • That's about how we can combine visual and physical

  • understanding of the world to make magical AR experiences.

  • Then Ellie will talk to you about shared augmentations.

  • And this is really all about the different ways

  • that we can connect people together in AR,

  • and how we can empower them just by putting them together.

  • And then Luca will cover expressive inputs.

  • This is about how AR can help unlock

  • authentic and natural understanding for our users.

  • So let's start about context-driven superpowers.

  • What this really means is using AR technologies

  • to deeply understand the context of a device,

  • and then build experiences that directly leverage that context.

  • And there's two parts to an AR context.

  • One is visual understanding, and the other

  • is physical understanding.

  • With ARCore, this gives your phone

  • the ability to understand and sense

  • its environment physically.

  • But through computer vision and machine learning,

  • we can make sense of the world visually.

  • And by combining these results, we

  • get an authentic understanding of the scene,

  • which is a natural building block of magical AR.

  • So let's start with visual understanding.

  • The prototyping community has done some awesome explorations

  • here, and we've done a few of our own

  • that we're excited to share.

  • To start, we wondered if we could

  • trigger custom experiences from visual signals in the world.

  • Traditional apps today leverage all kinds of device

  • signals to trigger experiences.

  • GPS, the IMU, et cetera.

  • So could we use visual input as a signal as well?

  • We built a really basic implementation of this concept.

  • This uses ARCore and the Google Cloud Vision

  • API that detects any kind of snowman

  • in the scene, which triggers a particle system that

  • starts to snow.

  • So through visual understanding, we

  • were able to tailor an experience to specific cues

  • in the environment for users.

  • This enables adaptable and context aware applications.

  • So even though this example is a simple one,

  • the concept can be extended so much further.

  • For example, yesterday we announced the augmented images

  • API for ARCore.

  • So if you use this, you can make something

  • like an experience that reacts relative to device movement

  • around an image in the scene, or even

  • from a known distance to an object in the world.

  • If you think this concept is interesting,

  • I highly recommend checking out the AR VR demo tent.

  • They have some amazing augmented images demos there.

  • The next thing we wanted to know is

  • if we could bridge the gap between digital and physical,

  • and, for example, bring some of the most delightful features

  • of e-readers to physical books.

  • The digital age has brought all kinds of improvements

  • to some traditional human behaviors,

  • and e-readers have brought lots of cool new things to reading.

  • But if you're like me, sometimes you just

  • missed the tactility in holding a great book in your hands.

  • So we wanted to know if we could bridge that gap.

  • In this prototype, users highlight a passage or word

  • with their finger and they instantly

  • get back a definition.

  • This is a great example of a short-form-focused interaction

  • that required no setup for users.

  • It was an easy win only made possible

  • by visual understanding.

  • But as soon as we tried this prototype,

  • there were two downfalls that we noticed,

  • and they became immediately apparent when we used it.

  • The first is that it was really difficult to aim your finger

  • at a small moving target on a phone,

  • and maybe the page is moving as well,

  • and you're trying to target this little word.

  • That was really hard.

  • And the second was that when you're highlighting a word,

  • your finger is blocking the exact thing

  • that you're trying to see.

  • Now, these are easily solvable with a follow-up UX iteration,

  • but they illustrate a larger lesson.

  • And that's that with any kind of immersive computing,

  • you really have to try it before you can judge it.

  • An interaction might sound great when you talk about it

  • and it might even look good in a visual mock,

  • but until you have it in your hand

  • and you can feel it and try it, you're

  • not going to know if it works or not.

  • You really have to put it in a prototype

  • so you can create your own facts.

  • Another thing we think about a lot

  • is, can we help people learn more effectively?

  • Could we use AR to make learning better?

  • There's many styles of learning, and if you

  • combine these styles of learning,

  • it often results in faster and higher-quality learning.

  • In this prototype, we combined visual, oral, verbal,

  • and kinesthetic learning to teach people how

  • to make the perfect espresso.

  • The videos explain--

  • I'm sorry.

  • We placed videos around the espresso machine

  • in the physical locations where that step occurs.

  • So if you were learning how to use the grinder,

  • the video for the grinder is right next to it.

  • Now, for users to trigger that video,

  • they move their phone to the area

  • and then they can watch the lesson.

  • That added physical component of the physical proximity

  • of the video and the actual device

  • made a huge difference in general understanding.

  • In our studies, users who had never used an espresso machine

  • before easily made an espresso after using this prototype.

  • So for some kinds of learning, this

  • can be really beneficial for users.

  • Now, unfortunately for our prototype,

  • one thing that we learned here was

  • that it's actually really hard to hold your phone

  • and make an espresso at the same time.

  • So you need to be really mindful of the fact

  • that your users might be splitting

  • their physical resources between the phone and the world.

  • And so as it applies to your use case,

  • try building experiences that are really

  • snackable and hands-free.

  • Speaking of combining learning and superpowers together,

  • we wondered if AR could help us learn

  • from hidden information that's layered in the world

  • all around us.

  • This is a prototype that we built

  • that's an immersive language learning app.

  • We showed translations roughly next to objects of interest

  • and positioned these labels by taking a point cloud

  • sample from around the object and putting the label sort

  • of in the middle of the points.

  • Users found this kind of immersive learning really fun,

  • and we saw users freely exploring

  • the world looking for other things to learn about.

  • So we found that if you give people

  • the freedom to roam and tools that are simple and flexible,

  • the experiences that you build for them

  • can create immense value.

  • We now have physical understanding.

  • This is AR's ability to extract and infer

  • information and meaning from the world around you.

  • When a device knows exactly where it is, not only in space,

  • but also relative to other devices,

  • we can start to do things that really

  • feel like you have superpowers.

  • For example, we can start to make

  • interactions that are extremely physical, natural,

  • and delightful.

  • Humans have been physically interacting

  • with each other for a really long time,

  • but digital life has abstracted some of those interactions.

  • We wondered if we could swing the pendulum

  • back the other direction a little bit using AR.

  • So in this prototype, much like a carnival milk bottle game,

  • you fling a baseball out of the top of your phone

  • and it hits milk bottles that are shown on other devices.

  • You just point the ball where you want to go, and it goes.

  • We did this by putting multiple devices

  • in a shared coordinate system, which

  • you could do using the new Google Cloud Anchors API

  • that we announced for ARCore yesterday.

  • And one thing you'll notice here is

  • that we aren't even showing users past their camera.

  • Now, we did that deliberately because we really

  • wanted to stretch and see how far we

  • could take this concept of physical interaction.

  • And one thing we learned was that once people learned

  • to do it, they found it really natural

  • and actually had a lot of fun with it.

  • But almost every user that tried it had to be not only

  • told how to do it, but shown how to do it.

  • People actually had to flip this mental switch

  • of the expectations they have for how a 2D smartphone

  • interaction works.

  • So you really need to be mindful of the context that people

  • are bringing in and the mental models they have

  • for 2D smartphone interactions.

  • We also wanted to know if we could help someone visualize

  • the future in a way that would let them make better decisions.

  • Humans pay attention to the things that matter to us.

  • And in a literal sense, the imagery

  • that appears in our peripheral vision

  • takes a lower cognitive priority than the things

  • we're focused on.

  • Would smartphone AR be any different?

  • In this experiment, we overlaid the architectural mesh

  • of the homeowner's remodel on top of the active construction

  • project.

  • The homeowner could visualize in context

  • what the changes to their home was going to look like.

  • Now, at the time that this prototype was created,

  • we had to do actual manual alignment of this model

  • on top of the house.

  • You could do it today.

  • If I rebuilt it, I would use the augmented images API

  • that we announced yesterday.

  • It would be much easier to put a fixed

  • image in a location, the house, and sync them together.

  • But even with that initial friction for the UX,

  • the homeowner got tremendous value out of this.

  • In fact, they went back to their architect