- There have been a lot of news about ChatGPT lately

like people using ChatGPT to write essays,

ChatGPT hitting a hundred million users,

Google launching Bard to compete against ChatGPT

and Microsoft integrating ChatGPT

into all their products, and also the viral sensation

of CatGPT where it can answer all of your queries,

but as a cat, meow, meow, meow, meow, meow, meow.

ChatGPT, if you don't know already, it's a chat bot

by OpenAI where you can ask it many things.

For example, explaining complex topics

like explain why I'm a disappointment to my parents

or ask it more technical questions like,

how do I inherit more money than my brother from my parents?

A lot of people are using it to write essays, draft emails,

and even write code.

So I tried it myself, of course, as a YouTuber obviously,

my first question to it was, who is Joma Tech?

And it answered...

Are you fucking--

You know, ChatGPT has a lot of limitations,

like here we ask it to name colors

that don't have the letter E in them,

and this is what they gave us.

Orang, yllow, red, that's clearly wrong.

In all seriousness,

this is to demonstrate how ChatGPT works.

It's a pre-trained large language model,

meaning it was trained on text data

from the internet until the end of 2021.

So it won't know anything

about things that happened recently.

It doesn't have access to the internet.

It'll only predict the answer based

on what it has consumed already,

and the way it answers your question is

by predicting each word that comes next.

For example, if you ask GPT who Bard is,

it's not going to know.

You might ask Joma, didn't your channel launch in 2017

and ChatGPT was trained on internet data until 2021,

yet it doesn't know who you are?

Yeah, so there's actually a technical reason

and fuck you.

Recently ChatGPT hit a hundred million users.

It launched November 30th, 2022,

and this article came out February 3rd, 2023.

So it took two months to hit a hundred million users.

Who are these users and what are they doing with ChatGPT?

Well, it's pretty obvious, they're cheating with it.

Everybody's cheating such that

some school districts have banned access to ChatGPT.

If they can write essays, then they can pass exams.

ChatGPT was able to pass exams from law school,

business school, and medical school.

Three prestigious industries.

Now, this is why I went into coding

because I always thought that law school,

business school, and medical school,

it was too much about memorization

and you're bound to get replaced,

it just wasn't intellectual enough, you know?

All right, well,

I guess engineering is getting replaced, too.

ChatGPT passes Google coding interview,

which is known to be hard, but I guess not.

But note that it is for a L3 engineer,

which means it's a entry level, for those not in tech,

there's no L2 and L1, it starts at L3,

but this does raise questions about ChatGPT's ability

to change engineering jobs behind it,

and we're already seeing the change

as Amazon employees are already using ChatGPT

for coding even though that immediately after,

they told them to stop, warning them not

to share confidential information with ChatGPT.

What's happening is they're feeding ChatGPT

internal documents, which are confidential,

but OpenAI stores all that data.

You know, it reminds me of when I used to intern

at Microsoft and they didn't let us use Google

for searches because they think that they might spy on us.

I was like, relax, I'm an intern.

I'm not working on anything important.

In fact, I actually wasn't working at all.

You know, I was playing Overwatch all day,

but yeah, anyways, they forced us to use Bing for searches.

One thing that's being underreported

in mainstream media is the success of GitHub Copilot.

It's probably the most useful

and most well executed AI product currently out there.

Have I used it?

No, I haven't coded in forever.

Now, here's how it works.

The moment you write your code,

it's like auto complete on steroids, like this example,

it helps you write the whole drawScatterplot function

and it knows how to use a D3 library correctly.

Another example here, you can write a comment

explaining what you want your function to do

and it'll write the code for you.

Sometimes even the name

of the function will give it enough information

to write the rest of the code for you.

It's very powerful

because it's able to take your whole code base as context

and with that, make more accurate predictions that way.

For example, if you're building a trading bot

and you write the function get_tech_stock_prices,

it'll suggest, hey, I know you're going

through a rough time,

but building a trading bot is not going

to fix your insecurities and maybe you should just accept

that you'll be a disappointment for the rest of your life.

Okay.

How did all of this happen?

Why is AI so good suddenly?

The answer is the transformer model

which caused a paradigm shift

on how we build large language models, LLM.

By the way, this diagram means nothing to me.

It makes me look smart, so that's why I put it on there.

Before transformers,

the best natural language processing system used RNN,

and then it used LSTM,

but then Google Brain published a paper

in 2017 called "Attention is All You Need"

which is also my life's motto because I'm a narcissist.

The paper proposes a simple neural network model

they call transformer, which is based

on the self attention mechanism

which I don't fully understand, so I'll pretend

like I don't have time to explain it

but I also know that it allows for more parallelization

which means you can throw more hardware,

more GPUs to make your training go faster

and that's when things got crazy.

They kept adding more data and also added more parameters

and the model just got better.

So what did we do?

We made bigger models with more parameters

and shoved it a shit ton of data.

Sorry, I'm trying my best here to make the model bigger.

All right, fuck it.

Anyway, that gave us ready

to use pre-trained transformer models like Google's Bert,

and OpenAI's GPT, generative pre-trained transformers.

They crawled the whole web to get text data

from Wikipedia and Reddit.

This graph shows you how many parameters each model has.

So as you can see, we've been increasing the number

of parameters exponentially.

So OpenAI kept improving their GPT model

like how Goku kept becoming stronger each time

he reached a new Super Saiyan form.

While editing this,

I realized how unhelpful the "Dragon Ball" analogy was.

So I want to try again.

To recap, transformer was the model architecture,

a type of neural network.

Other types of models would be like RNN and LSTM.

Compared to RNN, transformers don't need

to process words one by one,

so it's way more efficient at training with lots of data.

OpenAI used the transformer model and pre-trained it

by feeding it a bunch of data from the internet

and they called that pre-trained model GPT-1.

Back then, NLP models would be trained from scratch

for a specific task like translation or summarization.

Both transformer, we get to pre-train the model first

then fine tune it for a specific task.

Then for GPT-2, they did the same thing, but more

and with a bigger model, hence with 1.5 billion parameters,

and then with GPT-3,

they went crazy and gave it 175 billion parameters.