Name: Vectoring Words (Word Embeddings) - Computerphile
Uploaded: 2020-03-28T17:29:53.000Z
Duration: 16 min 56 s
Description: Thousands of YouTube videos with English-Chinese subtitles! Now you can learn to understand native speakers, expand your vocabulary, and improve your pronunciation...

Present continuous

if we're moving from capital, which is similar things.

Past simple

So we go away from Capitals, Dog, we gotta go beyond in that direction.

So the first result is dogs, which is kind of a nonsense result.

So that's like the dog use of dogs, right?

The most cat like cat, the most undock like, let's find out.

It's not really giving us anything much to work with.

I thought I would talk a little bit about wording, beddings, word avec and just wording beddings in general, the way I was introduced toward embedding So the sort of context that I'm most familiar with them in is like, how do you represent a word to a neural network?

Well, it's a set of characters, isn't it?

I mean, needed me more than the set of characters that make it up.

But you remember the thing we were talking about before in language models?

You have a problem of how far back, you can look.

I would much rather be able to look back 50 words than 50 characters.

And like if you if you're training a character based model, a lot of the capacity of your network is gonna be used up.

Just learning what characters counters valid words, right?

What combinations of characters of words.

And so if you're trying to learn something more complicated than that, you're spending a lot of your time training just like what words are.

And a lot of your network capacity is being used for that as well.

You can give the thing a dictionary, and then you're kind of that gives it gives it a jump start.

They view things as like a vector of real numbers or victor floats, which is like some of the real numbers, Um, and so if you think about something like on image, representing an image in this way is like, fairly straightforward.

You just take all of the pixels and put them in a long row on Dhe.

Then it's one, and you just have great skill in between.

For example, it's like, fairly straightforward.

It's sort of reflects some elements of the structure of what you're actually talking about.

So, um, like, if you take, if you take the same the same image and make it a little bit brighter, for example, that is just making that victor a bit longer right or a point in that configuration space that's a bit further from the origin, you could make it darker by moving it close to the origin by reducing the length of the vector.

If you take an image and you apply a small amount of noise to it, that represents just like jiggling that victor around slightly in that configuration space.

So you've got you got a sense in which to vectors that are close to each other, actually, kind of similar images on dhe um, that some of the sort of directions in the vector space air actually meaningful in terms of something that would make sense for images and the same is true with numbers and whatever else.

And this is very useful when you're training, because it allows you to say If you neural network is trying, predict the number and the value you're looking for is 10 and it gives you nine.

And if it gave you 7000 you could be like No, and it's not close.

And that gives more information that allows the system to learn.

And in the same way you can say, Yeah, that's almost the image that I want, Um, whereas if you give the thing a dictionary of words, so you've got your 10,000 words and the usual way of representing Mrs with a one heart to vector, if you have 10,000 words, you have a vector that's 10,000 long 10,000 dimensions on dhe, all of the values of zero apart from one of them, which is one so like the first word in the dictionary.

If it's like a then that's represented by a one and then the rest of the 10,002 zeroes, and then the 2nd 1 is like a zero, and then I wanted, but there you're not giving any of those clues.

If the thing is looking for one word and it gets a different word, all you can say is, Yeah, that's the correct one or no, that's not the correct one.

Something that you might try, but you shouldn't because it's a stupid idea is rather than rather than giving it as a one heart vector, you could just give it as a number.

But then you've got this indication that, like two words that are next to each other in the dictionary are similar, and that's not really true, right?

Like if you have a language model and you're trying to predict the next word and it's saying, I love playing with my pet blank like the word you're looking for his cat and the word it gives you his car lexical.

Graphically, they're pretty similar, but you don't want to be saying to your network, you know, close.

That was very nearly right, because it's not very nearly right.

But then, if it said like dog, you should be able to say no.

But that's close, right, because that is a plausible completion for that sentence on Dhe.

The reason that that makes sense is that cat and dog are like similar words.

What does it mean for a word to be similar to another word?

Um And so the assumption that word in beddings use is that two words a similar if they are often used in similar contexts.

So if you look at all of the instances of the word cat in a giant database, you know, John, corpus of text and all of the instances of the word dog they're gonna be surrounded by, you know, words like pet and words like, you know, feed and words like play.

And you know, that kind of thing cute, such right on.

So that gives some indication that these years of similar words, the challenge that word in beddings are trying to come up with It's like, How do you represent words as vectors?

Such that too similar vectors to similar words, and possibly so that directions have some meeting as well.

Um, because then that should allow our networks to be able to understand better what we're talking about in the text.

So the thing people realised was, if you have a language model that's able to get good performance if, like predicting the next word in a sentence.

Um, and the architecture of that model is such that it doesn't have that many neurons in its hidden layers.

It has to be compressing that information down efficiently.

So you've gone the inputs to your network.

Let's say, for the sake of simplicity, your language model is just taking a word and trying to guess the next word.

So we only have to deal with having one word in every port.

But so are input is this very tool thing, right?

10,000 toll, and these then feed into a hidden layer, which is much smaller.

And these are sort of the connections, and all of these is connected to all of these, and it feeds in, and then coming out the other end, you're back out to 10,000 again, right?

Because your output is it's gonna make one of these high.

You do something like soft Max to turn that into a probability distribution.

So you give it a word from the dictionary.

It then does something on DDE What comes out?

The other end is probability distribution, where you can just like, look at the highest value on the output, and that's what it thinks The next word will be on the higher that value is, the more like confident it is.

But the point is, you're going from 10,000 to 300 back out to 10,000.

If this if this is doing well at its task, this 300 has to be encoding sort of compressing information about the word because the information is passing through and it's going through this thing that suddenly 300 wide.

So in order to in order to be good at this task, it has to be doing this.

So then they were thinking, Well, how do we pull that knowledge out?

It's kind of like an egg drop competition.

Is this where you have to devise some method of safely getting the egg to the floor, right?

It's not like the teachers actually want to get an egg safely to the ground, right, But they've chosen the tasks such that if you can do well at this task, you have to have learned some things about physics and things about engineering and probably teamwork.

So it's there is the friends you make along the way.

So So the way that they the way that they build this is rather than, um trying to predict the next word.

Although that will work that will actually give you Worden beddings.

But they're not that good because they're only based on the immediately adjacent would u um, you look sort of around the word.

So you give it a word and then you sample from the neighborhood of that word randomly.