Subtitles section Play video Print subtitles (upbeat music) - There have been a lot of news about ChatGPT lately like people using ChatGPT to write essays, ChatGPT hitting a hundred million users, Google launching Bard to compete against ChatGPT and Microsoft integrating ChatGPT into all their products, and also the viral sensation of CatGPT where it can answer all of your queries, but as a cat, meow, meow, meow, meow, meow, meow. ChatGPT, if you don't know already, it's a chat bot by OpenAI where you can ask it many things. For example, explaining complex topics like explain why I'm a disappointment to my parents or ask it more technical questions like, how do I inherit more money than my brother from my parents? A lot of people are using it to write essays, draft emails, and even write code. So I tried it myself, of course, as a YouTuber obviously, my first question to it was, who is Joma Tech? And it answered... Are you fucking-- You know, ChatGPT has a lot of limitations, like here we ask it to name colors that don't have the letter E in them, and this is what they gave us. Orang, yllow, red, that's clearly wrong. In all seriousness, this is to demonstrate how ChatGPT works. It's a pre-trained large language model, meaning it was trained on text data from the internet until the end of 2021. So it won't know anything about things that happened recently. It doesn't have access to the internet. It'll only predict the answer based on what it has consumed already, and the way it answers your question is by predicting each word that comes next. For example, if you ask GPT who Bard is, it's not going to know. You might ask Joma, didn't your channel launch in 2017 and ChatGPT was trained on internet data until 2021, yet it doesn't know who you are? Yeah, so there's actually a technical reason and fuck you. Recently ChatGPT hit a hundred million users. It launched November 30th, 2022, and this article came out February 3rd, 2023. So it took two months to hit a hundred million users. Who are these users and what are they doing with ChatGPT? Well, it's pretty obvious, they're cheating with it. Everybody's cheating such that some school districts have banned access to ChatGPT. If they can write essays, then they can pass exams. ChatGPT was able to pass exams from law school, business school, and medical school. Three prestigious industries. Now, this is why I went into coding because I always thought that law school, business school, and medical school, it was too much about memorization and you're bound to get replaced, it just wasn't intellectual enough, you know? All right, well, I guess engineering is getting replaced, too. ChatGPT passes Google coding interview, which is known to be hard, but I guess not. But note that it is for a L3 engineer, which means it's a entry level, for those not in tech, there's no L2 and L1, it starts at L3, but this does raise questions about ChatGPT's ability to change engineering jobs behind it, and we're already seeing the change as Amazon employees are already using ChatGPT for coding even though that immediately after, they told them to stop, warning them not to share confidential information with ChatGPT. What's happening is they're feeding ChatGPT internal documents, which are confidential, but OpenAI stores all that data. You know, it reminds me of when I used to intern at Microsoft and they didn't let us use Google for searches because they think that they might spy on us. I was like, relax, I'm an intern. I'm not working on anything important. In fact, I actually wasn't working at all. You know, I was playing Overwatch all day, but yeah, anyways, they forced us to use Bing for searches. One thing that's being underreported in mainstream media is the success of GitHub Copilot. It's probably the most useful and most well executed AI product currently out there. Have I used it? No, I haven't coded in forever. Now, here's how it works. The moment you write your code, it's like auto complete on steroids, like this example, it helps you write the whole drawScatterplot function and it knows how to use a D3 library correctly. Another example here, you can write a comment explaining what you want your function to do and it'll write the code for you. Sometimes even the name of the function will give it enough information to write the rest of the code for you. It's very powerful because it's able to take your whole code base as context and with that, make more accurate predictions that way. For example, if you're building a trading bot and you write the function get_tech_stock_prices, it'll suggest, hey, I know you're going through a rough time, but building a trading bot is not going to fix your insecurities and maybe you should just accept that you'll be a disappointment for the rest of your life. Okay. How did all of this happen? Why is AI so good suddenly? The answer is the transformer model which caused a paradigm shift on how we build large language models, LLM. By the way, this diagram means nothing to me. It makes me look smart, so that's why I put it on there. Before transformers, the best natural language processing system used RNN, and then it used LSTM, but then Google Brain published a paper in 2017 called "Attention is All You Need" which is also my life's motto because I'm a narcissist. The paper proposes a simple neural network model they call transformer, which is based on the self attention mechanism which I don't fully understand, so I'll pretend like I don't have time to explain it but I also know that it allows for more parallelization which means you can throw more hardware, more GPUs to make your training go faster and that's when things got crazy. They kept adding more data and also added more parameters and the model just got better. So what did we do? We made bigger models with more parameters and shoved it a shit ton of data. Sorry, I'm trying my best here to make the model bigger. All right, fuck it. Anyway, that gave us ready to use pre-trained transformer models like Google's Bert, and OpenAI's GPT, generative pre-trained transformers. They crawled the whole web to get text data from Wikipedia and Reddit. This graph shows you how many parameters each model has. So as you can see, we've been increasing the number of parameters exponentially. So OpenAI kept improving their GPT model like how Goku kept becoming stronger each time he reached a new Super Saiyan form. While editing this, I realized how unhelpful the "Dragon Ball" analogy was. So I want to try again. To recap, transformer was the model architecture, a type of neural network. Other types of models would be like RNN and LSTM. Compared to RNN, transformers don't need to process words one by one, so it's way more efficient at training with lots of data. OpenAI used the transformer model and pre-trained it by feeding it a bunch of data from the internet and they called that pre-trained model GPT-1. Back then, NLP models would be trained from scratch for a specific task like translation or summarization. Both transformer, we get to pre-train the model first then fine tune it for a specific task. Then for GPT-2, they did the same thing, but more and with a bigger model, hence with 1.5 billion parameters, and then with GPT-3, they went crazy and gave it 175 billion parameters.