on the long term future of artificial intelligence.

In particular, I want to tell you

about a very important phenomenon called "Intelligence Explosion."

There are two reasons that I work on intelligence explosion

and that I think it's worth sharing.

The first is that it's a phenomenon of immense theoretical interest

for those who want to understand intelligence on a fundamental level.

The second reason is practical.

It has to do with the effects that intelligence explosion could have.

Depending on the conditions

under which an intelligence explosion could arise

and on the dynamics that it exhibits

it could mean that AI changes very rapidly

from a safe technology, relatively easy to handle,

to a volatile technology that is difficult to handle safely.

In order to navigate this hazard,

we need to understand intelligence explosion.

Intelligence explosion is a theoretical phenomenon.

In that sense, it's a bit

like a hypothetical particle in particle physics.

There are arguments that explain why it should exist,

but we have not been able to experimentally confirm it yet.

Nevertheless, the thought experiment

that explains what intelligence explosion would look like

is relatively simple.

And it goes like this.

Suppose we had a machine

that was much more capable than today's computers.

This machine, given a task,

could form hypotheses from observations,

use those hypotheses to make plans, execute the plans,

and observe the outcomes relative to the task,

and do it all efficiently within a reasonable amount of time.

This kind of machine could be given science and engineering tasks

to do on its own, autonomously.

And this is the key step in the thought experiment:

this machine could even be tasked with performing AI research,

designing faster and better machines.

Let's say our machine goes to work, and after a while,

produces blueprints for a second generation of AI,

that's more efficient, more capable, and more general than the first.

The second generation can be tasked once again

with designing improved machines,

leading to a third generation, a fourth, a fifth, and so on.

An outside observer would see

a very large and very rapid increase in the abilities of these machines,

and it's this large and rapid increase

that we call Intelligence Explosion.

Now if it's the case

that in order to undergo an intelligence explosion

many new pieces of hardware need to be build,

or new manufacturing technologies,

then an explosion will be more slow

- although still quite fast by historical standards.

However, looking at the history of algorithmic improvement

it turns out that just as much improvement

tends to come from new software as from new hardware.

This is true in areas like physics simulation, game playing,

image recognition, and many parts of machine learning.

What this means is that our outside observer may not see physical changes

in the machines that are undergoing an intelligence explosion.

They may just see a series of programs

writing successively more capable programs.

It stands to reason that this process could give rise to programs

that are much more capable at any number of intellectual tasks than any human is.

Just as we now build machines that are much stronger, faster, and more precise

at all kinds of physical tasks,

it's certainly possible to build machines

that are more efficient at intellectual tasks.

The human brain is not at the upper end of computational efficiency.

And it goes further than this.

There is no particular reason

to define our scale by the abilities of a single human or a single brain.

The largest thermonuclear bombs release more energy

in less than a second

than the human population of Earth does in a day.

It's not out of the question to think

that machines designed to perform intellectual tasks

and then honed over many generations of improvement

could similarly outperfom

the productive thinking of the human race.

This is the theoretical phenomenon called Intelligence Explosion.

We don't have a good theory of intelligence explosion yet,

but there is reason to think that it could happen at software speed

and could reach a level of capability

that's far greater than any human or group of humans

at any number of intellectual tasks.

The first time I encountered this argument,

I more or less ignored it.

Looking back it seems crazy for me, someone who takes AI seriously,

to walk away from intelligence explosion.

And I'll give you two reasons for that.

The first reason is a theorist's reason.

A theorist should be interested in the large-scale features of their field

in the contours of their phenomena of choice as determined by

the fundamental forces, or interactions, or building blocks of their subject.

As someone who aspires to be a good theorist of intelligence,

I can't, in good faith, ignore intelligence explosion

as a major feature

of many simple straightforward theories of intelligence.

What intelligence explosion means

is that intelligence improvement is not uniform.

There is a threshold below which improvements tend to peter out,

but above that threshold,

intelligence grows like compound interest increasing more and more.

This threshold would have to emerge from

any successful theory of intelligence.

The way phase transitions emerge from thermodynamics,

intelligence would effectively have a boiling point.

Seeing this way,

exploring intelligence explosion is exactly the kind of thing

a theorist wants to do, especially in a field like AI,

where we are trying to move from our current state

,partial theories, pseudotheories, arguments, and thought experiments,

toward a fully-fledged predictive theory of intelligence.

This is the intelligence explosion.

In its most basic form,

it relies on a simple premise

that AI research is not so different from other intellectual tasks

but can be performed by machines.

We don't have a good understanding yet,

but there's reason to think that it can happen at software speed

and reach levels of capability

far exceeding any human or group of humans.

The second reason which I alluded to at the start of the talk

is that intelligence explosion could change AI very suddenly

from being a benign technology to being a volatile technology

that requires significant thought into safety

before use or even development.

Today's AI, by contrast, is not volatile.

I don't mean that AI systems can't cause harm.

Weaponization of AI is ongoing, and accidental harms can arise

from unanticipated systemic effects or from faulty assumptions.

But on the whole, these sorts of harms should be manageable.

Today's AI is not so different from today's other technologies.

Intelligence explosion, however highlights an important fact:

AI will become more general, more capable, and more efficient

perhaps very quickly

and could become more so than any human or group of humans.

This kind of AI will require

a radically different approach to be used safely.

And small incidents could plausibly escalate to cause large amounts of harm.

To understand how AI could be hazardous,

let's consider an analogy to microorganisms.

There are two traits

that make microorganisms more difficult to handle safely than a simple toxin.

Microorganisms are goal-oriented,

and they are, what I'm going to call, chain reactive.

Goal-oriented means

that a microorganisms behaviors

tend to push towards some certain result.

In their case that's more copies of themselves.

Chain reactive means

that we don't expect a group of microorganisms to stay put.

We expect their zone of influence to grow,

and we expect their population to spread.

Hazards can arise, because a microorganisms

values don't often align with human goals and values.

I don't have particular use

for an infinite number of clones of this guy.

Chain reactivity can make this problem worse.

Since, small releases of a microorganism can balloon

into large population spending pandemics.

Very advanced AI, such as could arise from intelligence explosion,

could be quite similar in some ways to a microorganism.

Most AI systems are task-oriented.

They are designed by humans to complete a task.

Capable AIs will use many different kinds of actions

and many types of plans to accomplish their tasks.

And flexible AIs will be able to learn to thrive,

that is to make accurate predictions and effective plans

in a wide variaty of environments.

Since AIs will act to accomplish their tasks as well as possible,

they will also be chain reactive.

They'll have use for more resources, they'll want to improve themselves,

to spread to other computer systems, to make backup copies of themselves

in order to make sure that their task gets done.

Because of their task orientation and chain reactivity,

sharing an environment with this kind of AI would be hazardous.

They may use some of the things we care about,

our raw materials, and our stuff to accomplish their ends.

And there is no task that has yet been devised

that is compatible with human safety under these circumstances.

This hazard has made worse by intelligence explosion,

in which very volatile AI could arise quickly from benign AI.

Instead of a gradual learning period,

in which we come to terms with the power of very efficient AI,

we could be thrust suddenly into a world

where AI is much more powerful than it is today.

This scenario is not inevitable,

it's mostly dependent upon

some research group, or company, or government

walking into intelligence explosion blindly.

If we can understand intelligence explosion,

and if we have sufficient will and self-control as a society,

then we should be able to avoid an AI outbreak.

There is still the problem of chain reactivity though.

It would only take one group to release AI into the world

even if nearly all groups are careful.

One group walking into intelligence explosion accidently or on purpose

without taking proper precautions,

could release an AI that will self-improve

and cause immense amounts of harm to everyone else.

I'd like to close with four questions.

These are questions that I'd like to see answered

because they'll tell us more about the theory of artificial intelligence

and that theory is what will lead us understand intelligence explosion

well enough to mitigate the risks that it poses.

Some of these questions are being actively pursued

by researchers at my home institution,

The Future of Humanity Institute at Oxford,

and by others, like The Machine Intelligence Research Institute.

My first question is,

"Can we get a precise predictive theory of intelligence explosion?"

What happens when AI starts to do AI research?

In particular, I'd like to know

how fast software can improve its intellectual capabilities.

Many of the most volatile scenarios we've examined include

a rapid self-contained take off,

such as could only happen under a software improvement circumstance.

If there is some key resource that limits software improvement

or if it's the case that such improvement isn't possible

below a certain threshold of capability,

these would be very useful facts from a safety standpoint.

Question two:

what are our options, political or technological,

for dealing with the potential harms

from super efficient artificial intelligences?

One option, of course, is to not build them in the first place.

But this would require exceedingly good cooperation

between many governments, commercial entities, and even research groups.

That cooperation and that level of understanding isn't easy to come by.

It would also depend, to some extent, on an answer to question one

so that we know how to prevent intelligence explosion.

Another option would be to make sure

that everyone knows how to devise safe tasks.

It's intuitively plausible that there are some kinds of tasks

that can be assigned by a safety conscious team

without posing too much risk.

It's another question entirely

how these kinds of safety standards could be applied

uniformly and reliably enough all over the world

to prevent serious harm.

This leads into question three: very capable AIs,

if they can be programmed correctly,

should be able to determine

what is valuable

by modeling human preferences and philosophical arguments.

Is it possible to assign a task of learning what is valuable

and then acting to pursue that aim?

This turns out to be a highly technical problem.

Some of the ground work has been laid by researchers