Or maybe you just want to get familiar with using Python for data analysis.

Well, in this video, I'll show you some of the essential skills that you'll need to know in order to perform data analysis with Python and Pandas.

Hi, my name is Rob, and I'm a data scientist and three-time Kaggle Grandmaster.

I've spent a lot of my time exploring new datasets on Kaggle.

In this video, I'm going to walk through step-by-step the process I usually take when exploring a brand new dataset.

If you're completely new to using Python and Pandas, I suggest you first watch my tutorial on an introduction to Pandas.

In this video, I'll be working entirely in a Kaggle notebook.

So this means if you don't have Python already set up on your computer, you don't need to Just click the link to the Kaggle notebook.

You can fork or copy it, and then step through the code along with me or after this video is over.

So with that, let's get started.

Okay, so here we are in a Kaggle notebook.

Kaggle notebook is very similar to a Jupyter notebook, just an environment where we can run code and write some descriptions around it in plain text.

So in general, I always import these packages.

Of course, Pandas, and we're going to import that as pd.

Then we also have NumPy, which will import NumPy as np.

We're going to use a few different visualization libraries, matplotlib.

So import matplotlib, pylab as plot.

And then we're also going to import a package called seaborn, which is really helpful for doing exploratory plots with our data.

And we're going to import seaborn as sns.

Few other things we want to do before we get too far into this is we're going to use a style sheet for matplotlib and seaborn style plots.

So the way we do that is we do plot style use ggplot.

I think it looks pretty nice.

Another setting we're going to add is we're going to expand the number of columns that are shown when we display a different data frame in our notebook.

So the way we do that is pd set option.

And then we set the max columns to, let's say, 200 just to make sure we can see them all.

Now I'm going to comment that part out now and show you why it's necessary later.

So let's go ahead and run this.

That cell has run.

And now next thing we need to do is import our data.

And with Kaggle, we can go here to this tab on the side.

I've already added our data set.

What we're going to be looking at is a data set that I created with a bunch of information about over a thousand roller coasters.

It has information like the speed of the roller coaster, what material was used to make the roller coaster, and some cool things like that.

But if you're starting in a notebook and want to look at a different data set, you can always click on add data and look through a bunch of these data sets that already exist or upload your own data.

And then, of course, if you're working on a notebook on your own machine, you can just reference whatever CSV file you have.

So I'm going to import our data and just call it df for data frame.

And we're going to use read CSV to read in the data.

So I know this is in the input folder, and it's called CoasterDB.

So that we've read in very quickly, and we have now a data frame we can work with.

So the next step is to do some basic data understanding of our data.

So very simple stuff, but the first thing I like to do is do a df.shape.

So what does this tell us?

This tells us the shape of the data frame or the data that we just loaded in.

In this data set, we have 187 rows and 56 columns of data.

We can also run a head command on this.

This shows us the first five rows.

We could change the number of rows we want to see.

Let's say if we want to see the first 20.

And now if you notice here, when we do this head command, and let's put it back at five, what we see here is it shows us all the different columns, but then at a certain point, we see just dots, and then it picks up later with the last bit of columns.

This is because pandas by default does not show you every single column in the data frame.

But for exploration, I find it easier to show them all.

So going back up here to our PD set option, we're going to make this 200, which is plenty enough to see every column in our data set.

And then rerun this head command, and we'll see all of the different columns in the data set.

Now, another thing we want to do is just list out all the columns.

Since this data frame has a lot of columns, we can just do df.columns to see them all.

And now we've seen all the columns listed.

Eventually, if we want to subset our data set, we can remove some of these columns, and I'm going to show you how to do that later.

So the next thing we might want to do is, for each of these columns, to find out actually what D type that pandas has decided it is.

So if you remember from the earlier tutorial, in a pandas data frame, every column is actually a series, and every pandas series has a type.

So if we type in df.dtypes, we can actually see, for each of these columns, what the type of column it is.

So we have a lot of objects here, which are just string-type columns, and then we have some that are float values.

Actually, down here at the bottom, I created this data set, and I added some of the numeric values of features, like the height of the roller coaster, the speed in miles per hour here to the end.

Okay, so one of the last things I want to show you here is just the describe function.

So if you type in df.describe, there we go, what it'll show us is some information and statistics about the numeric data in our data set.

So we can see here the height of the roller coasters.

We have a count of 171 values, a mean value of 101 feet, and all this information that gives us a good understanding for the data before we dive into it any further.

Okay, so now it's time to move on to the second step, which is data preparation.

We have a general understanding of the columns in our data and how many rows we have, but we want to do some cleaning before we actually get into analysis.

And it's very important that we first drop any of the columns or rows that we don't want to keep before we continue on, or else we might waste time cleaning up columns that we won't end up using.

So let's run a df.head again on our data set to remember what sort of columns we have.

One thing to note about this data set is we actually have some columns that have values that are string versions, like speed and length.

And we also have similar columns that I have created when I created this data set over to the right here that have the numeric version of those.

So it's stripped out all the text of mile per hour and converted everything into the same unit here, which is miles per hour.

Same thing with gforce, inversions.

So what we're going to want to do is we're going to want to subset this data set just to the columns we want to keep.

And there are two ways that I like to do this.

So if we run df.columns again, we can pull in all the columns and we can actually just subset by copying this list of columns here.

And then with two brackets, we put in the list of columns, and that will show us all of the columns again.

It's pretty much the same data set.

But now we can start removing column by column what we've decided we don't want to keep.

So I already know I want to keep the roller coaster name.

I want to keep the manufacturer.

So I'm just commenting out the lines in this list that we don't want to keep.

And this makes it easy to keep track of what we've actually dropped.

Height restriction, I don't think we want to keep.

Inversions, cost, trains, park section, all of these things may be interesting to look at later.

But at this time, we're not going to take a look at them.

But we do want to keep some of this information at the end.

So year introduced, latitude, longitude, I think we want to keep the type of material used here.

Opening date clean.

Speed values.

All right, so here for speed, I think we just want to keep the miles per hour.

For the height, we just want to keep the value in feet.

And yeah, this looks like a good subset of the data that we want to keep.

Actually, let's add in location and status.

Now, if I run this cell, I see that we actually have a smaller data set.

Our columns have been reduced just to the columns that we want to keep.

We have some information though we will be able to work with.

Now, there's a second way we can deal with subsetting our data sets columns, and that's by using the drop command.

So let's show an example of how to do that.

So if we only wanted to drop one column, we can just write drop and provide it either one or a list of columns that we want to drop.

Let's say we want to remove status or opening date as an example.

We'll run this cell, and we actually have to provide it axis equals one so that it knows to drop not a row but a column.

Now, if you look here, opening date, that column now has disappeared.

So that's in a way that we can, I'll just keep this example.

Example of dropping single, dropping columns.

But we're not using that way to drop our columns here.

I'm just going to put that up here as an example.

We're going to use our subsetting of a list of columns, and we're actually going to reassign our data frame to be this new subset of data frame.

So we're going to rewrite data frame by doing data frame equals this new subsetted data frame, and I am at the end here.

Now, this can be important.

We want to add the copy command to the end of our subset of data frame.

This makes sure that Python knows that it's a brand new data frame and not just a reference to the old one, and when you're manipulating the data frame later on, it's going to be nice to run this copy.

So let's go ahead and run the cell, and if I run df shape now, we see we still have 1,087 rows, but we only have 13 columns, and if we run dtypes on this, we see that we have coaster name, location, and it looks pretty good.

Now, one thing I'm noticing here is that opening date clean, this should be ...

It's saying that that's an object column, but we don't want it to be an object column because it's a date, so we can actually force this to be a datetime column by running pd to datetime, and now the dtype is actually a datetime 64 column, so this is a way of ensuring that our dtypes are correct for each column.

We can rewrite them running to datetime.

Now, another similar option would be if we had, say, a numeric column that we want to force to be numeric.

Now, year introduced we already know is an int column, but let's say it was a string.

We could run pd to numeric on this, and pandas will automatically try to make it into a numeric column.

Not necessary here, but good to know.

Now, we're going to learn how to rename our columns.

There's some columns here that I'm not too happy with the names, and there's a pretty easy way to actually rename them, so let's make sure we're all looking in the right spot, and here we go.

So, we can, again, run our df columns to see what they are, and we have some differences in lowercase and uppercase names, so I also think it's important to not have spaces in the names of our columns, which we luckily don't have here, so what I'm going to do is still rename some of these, so I'm going to run df.rename, and then we can ask it to rename columns, by writing columns, and we're going to provide it a dictionary here.

Don't worry, this is pretty easy.

All we have to do is put two brackets as our dictionary with the old name and then the new name, so this lets us rename the column name to uppercase, and we'll go ahead and do this to a few of these other ones to make them in uppercase first letter format, so here we go.

Now, I've renamed all of these columns into things that I'm more happy with, the column names.

They're all starting with uppercase.

I've removed this underscore clean because these are now the only columns we have for inversions in GeForce, and I'm going to go ahead and rewrite my data frame with these newly named columns.

Now, if I run a data frame head on this, we can see beautiful column names, and everything looks pretty good.

Okay, so the next step here is to try to identify if where missing values exist in the data frame and how often they occur, so the command we're going to run to identify missing values or null values is the na command.

If you run is na on this data frame, in every single row and column, it'll tell us if there is a null value, but this is a little bit overwhelming to look at at first, and we would like to instead see a sum of the number of null values per column, so here we can see that for this data set, the status is null for 213 of our rows.

Latitude and longitude are missing for some.

GeForce is missing for some.

Now we have a general understanding of where we might have missing values, and similarly, we will want to look and see if any of our data is duplicated, so there might be issues in our data where we have two rows that are identically the same, and we would not want that in our data set, so the way we look at that is by running duplicated on the data frame.

Duplicated by default will give us the second or all the second and additional rows that are duplicated in the data frame.

It'll ignore the first row that is a duplicate, and why that's nice is because it gives us this list of true or false if the values are duplicated, and then we can just simply do a look on the duplicated values to see which ones are duplicated, and none of them are in this data set, which is nice.

We can also, with duplicated, run this on a subset of columns, so let's just see if there are any duplicated coaster names by running subset on coaster name, and actually, it looks like there are some.

If we run a dflok on this list of duplicated rows, it'll show us just the rows the second time they occur and are duplicated, so we can see that there are actually 97 rows that have a duplicated coaster name.