I'm on the Android team.

And I'm the technical lead for the multimedia framework.

I've been working on Android since October of 2007.

But actually, technically, I started before that,

because I worked on the MIDI engine that we're using.

So I kind of have a long, vested interest

in the project.

So today, we have kind of an ambitious title,

called "Mastering the Media Framework."

I think the reality is that if you believe that--

that we're going to do that in an hour,

it's probably pretty ambitious.

And if you do believe that,

I have a bridge just north of here

that you might be interested in.

But I think we actually will be able to cover

a few kind of interesting things.

In thinking about this topic,

I wanted to cover stuff that wasn't really available

in the SDK, so we're really going to del--

delve into the lower parts of the framework,

the infrastructure that basically

everything's built on.

Kind of explain some of the design philosophy.

So...

Oh, I guess I should have put that up first.

Here we go.

So on the agenda,

in the cutesy fashion of the thing,

we're talking about the architecture--

Frank Lloyd Android.

What's new in our Cupcake release,

which just came out recently.

And those of you who have the phone,

you're running that on your device today.

And then a few common problems that people run into

when they're writing applications for the framework.

And then there probably will be

a little bit of time left over at the end

for anybody who has questions.

So moving along,

we'll start with the architecture.

So when we first started designing the architecture,

we had some goals in mind.

One of the things was to make development

of applications that use media, rich media applications,

very easy to develop.

And so that was one of the key goals

that we wanted to accomplish in this.

And I think you'll see it as we look at the framework.

It's really simple to play audio,

to display a video, and things like that.

One of the key things, because this is

a multi-tasking operating system,

is we have-- you could potentially have

things happening in the background.

For example, you could have a music player

playing in the background.

We need the ability to share resources

among all these applications,

and so that's one of the key things,

was to design an architecture

that could easily share resources.

And the other thing is, you know,

paramount in Android is the security model.

And if you've looked over the security stuff--

I'm not sure we had a talk today on security.

But security is really important to us.

And so we needed a way to be able to sandbox

parts of the application that are--

that are particularly vulnerable,

and I think you'll see as we look at the--

the framework, that it's designed

to isolate parts of the system

that are particularly vulnerable to hacking.

And then, you know, providing a way

to add features in the future

that are backwards compatible.

So that's the-- the room for future growth.

So here's kind of a 30,000-foot view

of the way the media framework works.

So on the left side, you'll notice

that there is the application.

And the red line-- red dashed line there--

is denoting the process boundary.

So applications run in one process.

And the media server actually runs

in its own process that's actually booted up--

brought up during boot time.

And so the codecs

and the file parsers and the network stack

and everything that has to do with playing media

is actually sitting in a separate process.

And then underneath that are the hardware abstractions

for the audio and video pass.

So Surface Flingers are an abstraction for video

and graphics.

And Audio Flinger's the abstraction for audio.

So looking at a typical media function,

there's a lot of stuff--

because of this inner process communication that's going on,

there's a lot of things that are involved

in moving a call down the stack.

So I wanted to give you an idea--

for those of you who've looked at the source code,

it's sometimes hard to follow, you know, how is a call--

A question that comes up quite frequently

is how does a function call, like, you know,

prepare or make its way

all the way down to the framework

and into the-- the media engine?

So this is kind of a top-level view

of what a stack might look like.

At the very top is the Dalvik VM proxy.

So that's the Java object that you're actually talking to.

So, for example, for a media player,

there's a media player object.

If you look at the media player definition,

it's a pretty-- I mean,

there's not a lot of code in Java.

It's pretty simple.

And basically, it's a proxy for--

in this case, actually, the native proxy,

which it's underneath, and then eventually,

the actual implementation.

So from that, we go through JNI,

which is the Java Native Interface.

And that is just a little shim layer

that's static bindings

to an actual MediaPlayer object.

So when you create a MediaPlayer in Java,

what you're actually doing is making a call

through this JNI layer

to instantiate a C++ object.

That's actually the MediaPlayer.

And there's a reference to that that's held in the Java object.

And then some tricky stuff--

weak references to garbage collection

and stuff like that, which is a little bit too deep

for the talk today.

Like I said, you're not going to master the framework today,

but at least get an idea of what's there.

So in the native proxy,

this is actually a proxy object for the service.

So there is a little bit of code in the native code.

You know, a little bit of logic in the native code.

But primarily, most of the implementation

is actually sitting down in this media server process.

So the native proxy is actually the C++ object

that talks through this binder interface.

The reason we have a native proxy

instead of going directly through JNI

is a lot of the other pieces of the framework does.

So we wanted to be able to provide

access to native applications in the future

to use MediaPlayer objects.

So it makes it relatively easy,

because that's something you'd probably want to do

with games and things like that

that are kind of more natural to write in native code.

We wanted to provide the ability to do that.

So that's why the native proxy sits there

and then the Java layer just sits on top of that.

So the binder proxy and the binder native piece--

Binder is our abstraction for inter-process communication.

Binder, basically, what it does,

is it marshals objects across this process boundary

through a special kernel driver.

And through that, we can do things like move data,

move file descriptors that are duped across processes

so that they can be accessed by different processes.

And we can also do something which--we can share memory

between processes.

And this is a really efficient way of moving data

back and forth between the application

and the media server.

And this is used extensively

in Audio Flinger and Surface Flinger.

So the binder proxy is basically the marshalling code

on the applications side.

And the binder native code is the marshalling code

for the server side of the process.

And if you're looking at all the pieces

of the framework-- they start with

mediaplayer.java, for example--

there's an android_media...

_mediaplayer.cpp,

which is the JNI piece.

There's a mediaplayer.cpp,

which is the native proxy object.

Then there's an imediaplayer.cpp,

which is actually a-- a binder proxy

and the binder native code in one chunk.

So you actually see the marshalling code

for both pieces in that one file.

And one is called bpmediaplayer.cpp--

or, sorry, BP MediaPlayer object.

And a BN MediaPlayer object.

So when you're looking at that code,

you can see the piece that's on the native side--

the server side and the proxy.

And then the final piece of the puzzle

is the actual implementation itself.

So in the case of the media server--

sorry, the MediaPlayer-- there's a MediaPlayer service

which instantiates a MediaPlayer object

in the service that's, you know, proxied

in the application by this other MediaPlayer object.

That's basically-- each one of the calls

goes through this stack.

Now, because the stack is, you know, fairly lightweight

in terms of we don't make a lot of calls through it,

we can afford a little bit of overhead here.

So there's a bit of code that you go through

to get to this place, but once you've started playing,

and you'll see this later in the slides,

you don't have to do a lot of calls

to maintain the application playing.

So this is actually kind of a top-level diagram

of what the media server process looks like.

So I've got this media player service.

And it can instantiate a number of different players.

So on the left-hand side, you'll see, bottom,

we have OpenCORE, Vorbis, and MIDI.

And these are three different media player types.

So going from the simplest one, which is the Vorbis player--

Vorbis basically just plays Ogg Vorbis files,

which is a-- we'll get into the specifics

of the codec, but it's a psycho-acoustic codec

that's open sourced.

We use this for a lot of our internal sounds,

because it's very lightweight.

It's pretty efficient.

And so we use that for our ringtones

and for our application sounds.

The MIDI player, a little more complex.

But basically, it's just another instantiation

of a media player.

These all share a common interface,

so if you look at the MediaPlayer.java interface,

there's almost, you know, one-for-one correspondence

between what you see there and what's actually happening

in the players themselves.

And then the final one is OpenCORE.

So anything that isn't an Ogg file or a MIDI file

is routed over to OpenCORE.

And OpenCORE is basically the-- the bulk of the framework.

It consists of all of the major codecs,

like, you know, MP3 and AAC and AMR

and the video codecs, H.263 and H.264 and AVC.

So any file that's not specifically one of those two

ends up going to OpenCORE to be played.

Now, this provides some extensibility.

The media player service is smart enough