It went viral, but not in the way they'd hoped.

James Clayton picks up the story.

Yo, boys, I may need some help.

I found a room with a bunch of mob spawners.

You ever wondered what it would be like to be someone else?

With ElevenLabs AI technology, you can find out.

Given a minute of anyone's voice, the software can quickly put together an imitation.

Within days, people were imitating celebrity voices, making them say things they never said.

I just think I would love to get Ratatouilled.

Ratatouilled?

Like, have a little guy up there.

Excuse me?

You know, making me cook delicious meals.

This is the point where I tell you that you're actually listening to an AI voice,

but it wasn't exactly the most convincing voice.

This is it.

Testing out my voice clone.

I don't think it sounds like me that much, but what do you think?

I'm gonna record here too, just to, you know, to help them off.

My producer, Ben, and I spoke to the CEO and co-founder of ElevenLabs, Matty Staniszewski, for the company's first broadcast interview.

So I had a go at mimicking my own voice,

and it, you know, it was sort of reasonably successful,

although it seemed to have a bit of an Australian accent.

Majority of our work is based on public domain data, where majority of the voices and the work that we have are actually US American accents.

So if Ben did it, it might be more successful.

Ben should definitely try it.

The company's terms and conditions are clear that you can only clone your own voice, but not everyone has abided by its rules.

On sites like 4chan, it's incredibly easy to find people who've taken celebrity voices and made them say racist and hateful things.

ElevenLabs accepts its platform has been used for hate speech.

You know, 99% of content is kind of positive that we've seen

and for less than 1% of what was generated, it was usually against terms of service,

whether it was hate speech or variations of that.

The company says only paid users can clone voices now and they're developing a new tool to track down people abusing its platform.

But ElevenLabs doesn't just do cloned voices.

Just give me a second.

The company has another tool where users can create a unique voice completely from scratch.

Let's use a female gender and I'm gonna do a middle age.

And I think even more fitting with a British accent.

And I'm just gonna click generate.

That's gonna create that entirely new voice that doesn't exist.

Britain's first live public broadcast was made from the factory of Marconi's Wireless telegraph company in Chelmsford.

It's never been easier to imitate a politician's voice.

Sleepy Joe, you have run this country to its absolute ground.

Well, at least I do run sometimes, if you know what I mean?

Mark Nitzberg is a professor at UC Berkeley's AI research lab.

I asked him whether we should be worried about the presidential elections in 2024.

We'll very likely see many more deep fakes than we did in the last cycle just because it's so much simpler

and I'm not sure what measures we are taking to counter what effect that might have.

There are plenty of other concerns around this tech too.

For example, scams,

this could make them far more believable and far easier to execute.

The Federal Trade Commission put out a statement in March,

warning people that scams imitating relatives can now be incredibly convincing.

ElevenLabs themselves say that as AI develops,

people are simply going to have to learn to trust voices they hear and things that they see less.

A kind of the wider education is just making sure that people across know that

as we think about audio, as we think about text, as we think about images,

a lot of the content that has been created is likely or could be AI generated.

And it's important to know that this might be the case.

You know, one needs to again take this approach where we think about we want in a society before we allow the release of these tools.

Voice cloning for the public is here to stay.

It's very difficult to see how this is going to be put back in its box now.

But this is clearly disruptive technology that can be used for both good and bad.