Placeholder Image

Subtitles section Play video

  • I'm going to talk about my research

  • on the long term future of artificial intelligence.

  • In particular, I want to tell you

  • about a very important phenomenon called "Intelligence Explosion."

  • There are two reasons that I work on intelligence explosion

  • and that I think it's worth sharing.

  • The first is that it's a phenomenon of immense theoretical interest

  • for those who want to understand intelligence on a fundamental level.

  • The second reason is practical.

  • It has to do with the effects that intelligence explosion could have.

  • Depending on the conditions

  • under which an intelligence explosion could arise

  • and on the dynamics that it exhibits

  • it could mean that AI changes very rapidly

  • from a safe technology, relatively easy to handle,

  • to a volatile technology that is difficult to handle safely.

  • In order to navigate this hazard,

  • we need to understand intelligence explosion.

  • Intelligence explosion is a theoretical phenomenon.

  • In that sense, it's a bit

  • like a hypothetical particle in particle physics.

  • There are arguments that explain why it should exist,

  • but we have not been able to experimentally confirm it yet.

  • Nevertheless, the thought experiment

  • that explains what intelligence explosion would look like

  • is relatively simple.

  • And it goes like this.

  • Suppose we had a machine

  • that was much more capable than today's computers.

  • This machine, given a task,

  • could form hypotheses from observations,

  • use those hypotheses to make plans, execute the plans,

  • and observe the outcomes relative to the task,

  • and do it all efficiently within a reasonable amount of time.

  • This kind of machine could be given science and engineering tasks

  • to do on its own, autonomously.

  • And this is the key step in the thought experiment:

  • this machine could even be tasked with performing AI research,

  • designing faster and better machines.

  • Let's say our machine goes to work, and after a while,

  • produces blueprints for a second generation of AI,

  • that's more efficient, more capable, and more general than the first.

  • The second generation can be tasked once again

  • with designing improved machines,

  • leading to a third generation, a fourth, a fifth, and so on.

  • An outside observer would see

  • a very large and very rapid increase in the abilities of these machines,

  • and it's this large and rapid increase

  • that we call Intelligence Explosion.

  • Now if it's the case

  • that in order to undergo an intelligence explosion

  • many new pieces of hardware need to be build,

  • or new manufacturing technologies,

  • then an explosion will be more slow

  • - although still quite fast by historical standards.

  • However, looking at the history of algorithmic improvement

  • it turns out that just as much improvement

  • tends to come from new software as from new hardware.

  • This is true in areas like physics simulation, game playing,

  • image recognition, and many parts of machine learning.

  • What this means is that our outside observer may not see physical changes

  • in the machines that are undergoing an intelligence explosion.

  • They may just see a series of programs

  • writing successively more capable programs.

  • It stands to reason that this process could give rise to programs

  • that are much more capable at any number of intellectual tasks than any human is.

  • Just as we now build machines that are much stronger, faster, and more precise

  • at all kinds of physical tasks,

  • it's certainly possible to build machines

  • that are more efficient at intellectual tasks.

  • The human brain is not at the upper end of computational efficiency.

  • And it goes further than this.

  • There is no particular reason

  • to define our scale by the abilities of a single human or a single brain.

  • The largest thermonuclear bombs release more energy

  • in less than a second

  • than the human population of Earth does in a day.

  • It's not out of the question to think

  • that machines designed to perform intellectual tasks

  • and then honed over many generations of improvement

  • could similarly outperfom

  • the productive thinking of the human race.

  • This is the theoretical phenomenon called Intelligence Explosion.

  • We don't have a good theory of intelligence explosion yet,

  • but there is reason to think that it could happen at software speed

  • and could reach a level of capability

  • that's far greater than any human or group of humans

  • at any number of intellectual tasks.

  • The first time I encountered this argument,

  • I more or less ignored it.

  • Looking back it seems crazy for me, someone who takes AI seriously,

  • to walk away from intelligence explosion.

  • And I'll give you two reasons for that.

  • The first reason is a theorist's reason.

  • A theorist should be interested in the large-scale features of their field

  • in the contours of their phenomena of choice as determined by

  • the fundamental forces, or interactions, or building blocks of their subject.

  • As someone who aspires to be a good theorist of intelligence,

  • I can't, in good faith, ignore intelligence explosion

  • as a major feature

  • of many simple straightforward theories of intelligence.

  • What intelligence explosion means

  • is that intelligence improvement is not uniform.

  • There is a threshold below which improvements tend to peter out,

  • but above that threshold,

  • intelligence grows like compound interest increasing more and more.

  • This threshold would have to emerge from

  • any successful theory of intelligence.

  • The way phase transitions emerge from thermodynamics,

  • intelligence would effectively have a boiling point.

  • Seeing this way,

  • exploring intelligence explosion is exactly the kind of thing

  • a theorist wants to do, especially in a field like AI,

  • where we are trying to move from our current state

  • ,partial theories, pseudotheories, arguments, and thought experiments,

  • toward a fully-fledged predictive theory of intelligence.

  • This is the intelligence explosion.

  • In its most basic form,

  • it relies on a simple premise

  • that AI research is not so different from other intellectual tasks

  • but can be performed by machines.

  • We don't have a good understanding yet,

  • but there's reason to think that it can happen at software speed

  • and reach levels of capability

  • far exceeding any human or group of humans.

  • The second reason which I alluded to at the start of the talk

  • is that intelligence explosion could change AI very suddenly

  • from being a benign technology to being a volatile technology

  • that requires significant thought into safety

  • before use or even development.

  • Today's AI, by contrast, is not volatile.

  • I don't mean that AI systems can't cause harm.

  • Weaponization of AI is ongoing, and accidental harms can arise

  • from unanticipated systemic effects or from faulty assumptions.

  • But on the whole, these sorts of harms should be manageable.

  • Today's AI is not so different from today's other technologies.

  • Intelligence explosion, however highlights an important fact:

  • AI will become more general, more capable, and more efficient

  • perhaps very quickly

  • and could become more so than any human or group of humans.

  • This kind of AI will require

  • a radically different approach to be used safely.

  • And small incidents could plausibly escalate to cause large amounts of harm.

  • To understand how AI could be hazardous,

  • let's consider an analogy to microorganisms.

  • There are two traits

  • that make microorganisms more difficult to handle safely than a simple toxin.

  • Microorganisms are goal-oriented,

  • and they are, what I'm going to call, chain reactive.

  • Goal-oriented means

  • that a microorganisms behaviors

  • tend to push towards some certain result.

  • In their case that's more copies of themselves.

  • Chain reactive means

  • that we don't expect a group of microorganisms to stay put.

  • We expect their zone of influence to grow,

  • and we expect their population to spread.

  • Hazards can arise, because a microorganisms

  • values don't often align with human goals and values.

  • I don't have particular use

  • for an infinite number of clones of this guy.

  • Chain reactivity can make this problem worse.

  • Since, small releases of a microorganism can balloon

  • into large population spending pandemics.

  • Very advanced AI, such as could arise from intelligence explosion,

  • could be quite similar in some ways to a microorganism.

  • Most AI systems are task-oriented.

  • They are designed by humans to complete a task.

  • Capable AIs will use many different kinds of actions

  • and many types of plans to accomplish their tasks.

  • And flexible AIs will be able to learn to thrive,

  • that is to make accurate predictions and effective plans

  • in a wide variaty of environments.

  • Since AIs will act to accomplish their tasks as well as possible,

  • they will also be chain reactive.

  • They'll have use for more resources, they'll want to improve themselves,

  • to spread to other computer systems, to make backup copies of themselves

  • in order to make sure that their task gets done.

  • Because of their task orientation and chain reactivity,

  • sharing an environment with this kind of AI would be hazardous.

  • They may use some of the things we care about,

  • our raw materials, and our stuff to accomplish their ends.

  • And there is no task that has yet been devised

  • that is compatible with human safety under these circumstances.

  • This hazard has made worse by intelligence explosion,

  • in which very volatile AI could arise quickly from benign AI.

  • Instead of a gradual learning period,

  • in which we come to terms with the power of very efficient AI,

  • we could be thrust suddenly into a world

  • where AI is much more powerful than it is today.

  • This scenario is not inevitable,

  • it's mostly dependent upon

  • some research group, or company, or government

  • walking into intelligence explosion blindly.

  • If we can understand intelligence explosion,

  • and if we have sufficient will and self-control as a society,

  • then we should be able to avoid an AI outbreak.

  • There is still the problem of chain reactivity though.

  • It would only take one group to release AI into the world

  • even if nearly all groups are careful.

  • One group walking into intelligence explosion accidently or on purpose

  • without taking proper precautions,

  • could release an AI that will self-improve

  • and cause immense amounts of harm to everyone else.

  • I'd like to close with four questions.

  • These are questions that I'd like to see answered

  • because they'll tell us more about the theory of artificial intelligence

  • and that theory is what will lead us understand intelligence explosion

  • well enough to mitigate the risks that it poses.

  • Some of these questions are being actively pursued

  • by researchers at my home institution,

  • The Future of Humanity Institute at Oxford,

  • and by others, like The Machine Intelligence Research Institute.

  • My first question is,

  • "Can we get a precise predictive theory of intelligence explosion?"

  • What happens when AI starts to do AI research?

  • In particular, I'd like to know

  • how fast software can improve its intellectual capabilities.

  • Many of the most volatile scenarios we've examined include

  • a rapid self-contained take off,

  • such as could only happen under a software improvement circumstance.

  • If there is some key resource that limits software improvement

  • or if it's the case that such improvement isn't possible

  • below a certain threshold of capability,

  • these would be very useful facts from a safety standpoint.

  • Question two:

  • what are our options, political or technological,

  • for dealing with the potential harms

  • from super efficient artificial intelligences?

  • One option, of course, is to not build them in the first place.

  • But this would require exceedingly good cooperation

  • between many governments, commercial entities, and even research groups.

  • That cooperation and that level of understanding isn't easy to come by.

  • It would also depend, to some extent, on an answer to question one

  • so that we know how to prevent intelligence explosion.

  • Another option would be to make sure

  • that everyone knows how to devise safe tasks.

  • It's intuitively plausible that there are some kinds of tasks

  • that can be assigned by a safety conscious team

  • without posing too much risk.

  • It's another question entirely

  • how these kinds of safety standards could be applied

  • uniformly and reliably enough all over the world

  • to prevent serious harm.

  • This leads into question three: very capable AIs,

  • if they can be programmed correctly,

  • should be able to determine

  • what is valuable

  • by modeling human preferences and philosophical arguments.

  • Is it possible to assign a task of learning what is valuable

  • and then acting to pursue that aim?

  • This turns out to be a highly technical problem.

  • Some of the ground work has been laid by researchers