In this video, we're gonna be going over a very quick overview of both the beautiful soup module python as well as the requests module in python as well.

So if you're not familiar, a very brief introduction to both of those requests is going to be a model that's going to allow us to access Various resource is on the web, so allow us to obtain information from various websites navigate to those websites, and beautiful soup is going to allow us to parse that information.

So once we navigate to those websites, we might want to extract certain types of content from those websites, and beautiful soup is going to allow us to do that.

So this is going to be very much from the ground up video.

The only thing I assume that you have installed in your machine is python and pip P I P.

For installing python packages.

If you have both of those things installed in your machine, you should be ready to follow along one toe one and it should work out just fine.

So we're gonna go over some of the basics of both the requests and beautiful soup module in the first part of the video.

And then we're gonna bring all of those things that we learned into, Ah, very real project or a very simple project.

I should say this is really meant to be a very minimal introduction to both of these modules.

And just to see an example of how you could be effective with both of these things.

So let's just get started by installing both the requests and beautiful soup modules on your machine.

You probably will already have requests installed, but we'll just make sure that you have that installed in any case.

So go ahead and open up a terminal or, if you're on Windows Command, prompt and go ahead and type in the following two lines.

So pip install requests.

If you already have this installed in your machine like I do, you'll see some requirement already.

Satisfied messages pop up here.

That's totally fine.

If you don't, it will install in your machine.

You probably should have this installed already.

The other one is install beautiful suit before, so go ahead and say pip installed B s four and I already have this installed on their machines.

So we're good to go.

All right, so we've got all the things that we need installed.

I'm just gonna go ahead and close that window, and we're gonna move back to our file here.

So we've gone ahead and done this part.

We've installed both of these modules.

So what we're gonna do next is we're going to import them and use them, so we're gonna import requests, and then we're going to import a specific class for the B s for module that we installed specifically the beautiful soup class.

So this is going to allow us to in the terms of beautiful soup soup if I content that we obtained from a website and again that will make it possible so we can extract information from that content.

Okay, so the first thing we're gonna do is we're going to use the request module, and I guess before I should go for before we go any further, I want to mention that all of the code, along with the comments that is kind of supplementing what I'm saying here, all that is going to be provided.

Um, I get hub and linked to that will be accessible in the description so you could go ahead and download that.

If you want some additional information as to what I'm doing or saying right so back to this, we're going to create a variable called results and we're going to say request dot get And they were going to pass in the website that that we want to access.

So in this case, I'm just accessing the home page of Google.

So I'm just going to move my browser over here, so I'm just going to access this page right there, So I'm just gonna go ahead and get that And then what we're going to do to make sure that the page was actually access is we're going to print out the status code, which is just a http status code letting us know whether or not the page was accessible or not.

So four or four errors are in http code That's letting you know that the constantly looking for is not present.

A 200 code is letting you know.

The content that you're looking for is present, and the response is okay.

So I'm gonna go ahead and write this.

I'm going to run it.

So what's a python?

This file is called Beautiful Soup and request that pie.

If I read this is going to run it.

Let me just clear the terminal because it's kind of messy.

Let me read that again.

And we see that the the code that we get back from the google dot com home page is 200 which indicates that indeed the page is accessible.

So another thing that we could do is we could also print out some other information of the Web page that we just accessed, like the http headers and again for more information on both the headers and status codes you could navigate to these links that are in the comments that point to the Wikipedia articles on both of those topics.

So I'm just gonna go ahead and write this, and we'll see what the headers look like.

There's just some extra information on the google dot com home page, so friends that we can see here the domain is going with that calm.

Some other information here is well, that might be of use just going to go ahead and close that.

That's more or less just to verify that.

Not only is the page valid, but it's also indeed the page that we wanted to obtain.

Let's keep moving down here.

So the next thing we're gonna do is we're going to extract the content of the page, and that is actually the source of the page.

We're going to store that a variable.

So the the object that we created here, this result the result has these built in methods status code, headers.

That's how we were able to print out both of those types of information that we just printed out there.

Another one that it has is dot content and that is going to return the source of that page and what we're gonna do, store that into a very well called SRC for source, and then we're going to have this variable.

So let's go ahead and run that, and I'm just gonna print out the source so you could actually see what it looks like gonna run that like this.

So we see a bunch of output here, and this is really just the Web source of the google dot com home page, so I'm just gonna go ahead, delete that.

That's kind of a lot of stuff.

All just throw it on the screen.

That's great that we can actually access it.

Now we turn to beautiful suit because this is going to allow us to actually do something with that content.

So what we're gonna do now that we've obtained that contact, we've navigated that Web page we've verified that were on the actual Web page that we want.

And now what we're gonna do is we're going to pass that source variable into this beautiful soup class, creating a soup object.

And this really is just kind of object that beautiful suit creates from this source, and it allows us to extract certain types of information that we wouldn't want to extract from this thing.

So we're going to store the result of that object in this variable called soup.

So I'm passing in source here, and there's an extra parameter here this Alex ml, you could more or less just ignore this extra parameter.

It doesn't really mean too much.

If you don't have it in there, you'll get awarded essentially to tell you to put it in there.

So it's there for all intensive purposes.

But you don't necessarily need to worry so much about why this is here.

So we have the soup object.

So, for instance, one thing that we could do now that we have the soup object, is we could do things like give me all of the links on the page.

So what we could do is create a variable which I call links, and I can say soup, not find all so final is a method provided from this beautiful soup object that were accessing here.

And what do I want to find all of on this page?

I wanna find all of the eight tags.

So this a is the argument that's being passed and it tells beautiful soup to say, Hey, find all of the eight tags are all of the links on this page and then store them in this variable links, and then we're gonna go ahead and actually just print those out to the screen.

And just so we don't get like, too much content, I'm gonna go ahead and comment out the previous print statements here just so we don't have too much output of the street so the gutter links will print about little pretty do line.

So right that give it a run, Let's see what we get.

So this is the output that we just ran here and we can see that we have a list so you can see the square bracket here.

And the Ed Square bracket indicates a python list.

And the contents of this list are all of the eight tags on the page.

So if I move, let's just take a look at some of these links.

So we have images.

We have one that has the text maps.

So let me just kind of move this back over here.

Indeed.

You know, if we look, let's just look it over here.

We have this thing called images that is pointing to the leak in the top, right, maps, etcetera so we could look for other legs in this list that would correspond to the actual links of the page that we have just have searched for all the links of the page.

So that's that.

So I would go ahead, comet that out so we don't have excess output here, and let's just keep going down, so get it.

All of the list of links is one thing, but we might want to actually extract certain types of links.

So maybe the page that we're requesting the page that we've just super fied it has links that we're actually after.

So this example here says, OK, actually, I care about all of the links that have a certain string in the text field.

So what?

I mean by the text field?

If I bring this back up essentially what to look for all of the links, Let's say that have the word about in them.

So I want to see all of the links that have the word about anything.

That's an a tag where we between the A and the slash a.

There's the word about it, has that I want that link.

So that's one example one use case out of many that you could form in this in this type of work space.

So what we're gonna do is we're going to look through all of the links that we've obtained from above, and then we're going to do is we're going to say if the word about is in the text of the link.

So you get what I'm doing is abusing the dot text part of that variable.

So as I'm looping through each of these links, that list every one of those elements in that list is actually a beautiful soup element.

So that allows me to call this dot text function on each of those elements because that is defined from this element that is extracted.

It's stored in this list.

So I'm saying, Look at the text in that Lincoln that list and is the word about in that in that text?

If so, then go ahead and print out that late because I care about that link and actually go ahead and print out the, um, prints out the actual thing that it goes to a CZ.

Well, so that's what this is doing here.

So Link also has a, uh, a tribute.

Or I should say, if a function a TT rs and I'm looking for a specific attributes of that link.

So inside the a tag, there's attributes inside of that, which is the H r E f tag inside of that, and I've said give me the content of that H R E f inside of the link tag, which is the inside of this tag that has the A.

So it's kind of a lot of Russia dull kind of structure going out there.

So let's go ahead and just put that out, all right, that run it.

So we have here is we have the lake.

So we found the about Google.

Fade right here, So that's good.

And then we also have the h ref attributes of this link.

So, for instance, we see this whole This is the whole A tag.

So we have everything from the start of the A tag to the end of it, including the text, including the H Raph.

And then what we did with the next line is we said, Hey, inside of the a tag there's this attributes called a treff and we want to access the content of the attributes inside of the A tag and what we did there.

That's precisely this.

And that's what we're prints it out right here.

So I hope that makes sense.

I know that's kind of drilling down quite a bit, and you could go pretty complicated with these types of expressions, but I think once you understand this level, you could arbitrarily apply different levels of complexity to your own situations.

Your own problems, your own situations, your own scenarios said that already anyway, So we have this structure set up.

We have kind of a general idea of how to use both requests and beautiful soup.

Let's go ahead and try to apply this on.

Ah, more elaborate Web page and the Web pages I had in mind is this What page?

Right here.

So this is just a collection of briefings and statements given at the White House.

So you can see the day of this upload is September 12th.

And so these are the most recent remarks by briefings and statements that have something to do with the White House.

So there's a whole list of the and the goal of what we're going to do with beautiful super requests is we're going to navigate to this page.

We're going to try to find a way to extract all of the links of this page, and specifically all of the legs that have are sort of in this feed here.

So that's what we're going to do.

We're going to store them into a list, and then you could think about doing whatever you like to that list.

I mean, of course, like the possibilities air are quite endless, depending on your goals.

But this just kind of gives you the scaffolding, the structure, the idea to take this and then, you know, do whatever you want for your own purposes.

Let me just minimize this for now.

Let's go back to this code.

So right.

So what we're gonna do is I'm going to open up a new tab.

So I'm gonna go and say, Tab, do have another file that I've created on Have some initial commented, which I've called White House example.

Not pie.

I'm gonna go ahead and open that file here so you can see that I've just opened a python file, which is called White House example, not pie.