Name: Buffering dataset - Creating a Chatbot with Deep Learning, Python, and TensorFlow p.3
Uploaded: 2020-03-31T18:18:32.000Z
Duration: 11 min 37 s
Description: Thousands of YouTube videos with English-Chinese subtitles! Now you can learn to understand native speakers, expand your vocabulary, and improve your pronunciation...

To-infinitive

Everybody welcome to part three of our how to make a friend.

Past simple

You're Siri's In the last tutorial, basically just kind of created our table Saurus set the scene for what we're expecting to be doing.

And now in this tutorial, what we're gonna do is start actually iterating through that file, start cleaning up the data at least a little bit.

So with that, let's go ahead and get started.

So the code that we're gonna be writing here, at least for a little bit, will be here.

So now we're gonna do is I'm gonna go ahead, start a counter for road counter and then I also just out of curiosity paired Rose.

So both of these things are gonna be these counters row counter is just gonna tell us, kind of.

How many rows have we gone through as we're iterating through these files with like, 50 to 80 plus 1,000,000 rose and then paired?

Rose tells us how many parent and child um pairs have we come up with Basically because a lot of times we're prying, not like a lot of comments will go without a reply because they don't need a reply or they're never seen or whatever.

So anyways, we've got that, uh, Now we're gonna do is with open.

And now we want to open one of those files.

Now, wherever you've stored your files is probably different from where I've stored the file.

So, um, mine are in J colon chat data slash reddit data and then one of the files, for example, like, let's say we're into 2015.

So we'll say that's about all they're already even knows.

See, people decide all for, you know, being not special or whatever.

Hasn't finally, um r c and that will be underscore.

Um, and then format is this will be timeframe dot split, and we want to split by.

No Thio split zero and then finally r c underscore er and then it will be the full time for him.

So, uh so So we wanna open this file, right?

We want to open that file and then we're gonna buffer.

We'll just say, uh and then as f and then we're ready to actually store iterating through F.

So, for example, what we could say is four row in F and then what we can say here is First of all, let's go ahead, Roe Counter plus equals one on.

And then what we're gonna do is ro equals jayson dot load string row.

And then we can say a parent i d parent I d equals row parents i d.

And then, uh, the body equals So the body's probably have some issues.

So we're gonna do it for Matt underscore data so we can create a new function for this, and then it'll be row body.

We need something to kind of sanitize and clean up that data.

So that'll be body, uh, and then created you TC b ro created you to see and then score row.

Whatever the score waas and finally subreddit will equal Uh oh.

Okay, so now we want to go ahead and do is we're gonna come up here, and we need to go ahead and create that format data function.

It's gonna take in data, and basically, we're going to his data.

Evils datadata replace, uh, place and are at this stage, we really just trying to replace a few key things.

Some because they did not replace a new line character.

Um, we're gonna replace that straight up with, um I'm just gonna put spaces around it.

Okay, uh, that way first wallet doesn't get a pendant so often times at the very end of something, you get a new line, characters.

And when we go to actually token eyes this information, um, that would be, like token eyes together, and we definitely don't want that.

We also simultaneously don't really want a token.

I like if we if we token eyes this correctly, we're gonna token eyes, most likely a backslash and as separate entities not as one single entity.

That means hey, basically, when we token eyes, we token eyes, entire words.

In theory, you can actually token eyes, chunks of words and stuff like that, Like syllables basically.

So basically, long story short, we want to make sure that new line characters stay together.

Hopefully no one on Reddit actually uses new line care.

But anyway, that's what we're doing there.

And then we're gonna do the exact same thing with return.

It's like slash or a slash and er or something like that.

So we're gonna make sure that gets replaced.

And then finally, I'm just replace one more thing, and that's going to be the double quote words gonna say all double quotes are actually single quotes just to kind of normalized the data there.

So because there's really no reason for the A I to think, um, to think that there's two there's no release mean, mean the same thing.

So anyway, we're gonna we're gonna do that.

So, uh, so that's how we're going to format data.

I want to go ahead and do is price right here.

I'd still like to do like, we'd like to do something like this.

Like a parent parent data equals, find parents parent I d.

There's gonna be times where we were gonna Maybe you want to find information from that actual parents comment.

We're gonna find the parent by the parent i d.

And then what we're gonna dio is we're just gonna say es que o equals select comments from parent reply.

So if you named it something different cities that where comments i d So, actually, Lissy parent, I d I d.

We'll private type of this more in the next the next one, but anyways, paranoid He's gonna get passed into here, and then basically, we're looking for anywhere where the comment i d is the parent.

So this is how we're gonna actually find the initial parent comment that belongs to, Uh, that's so, for example, every comment.

The string has a parent, I d but it doesn't have the parents text, right?

So when we insert this comment into our database, we actually were gonna want the parent body.