Name: How AI Taught Itself to See [DINOv3]
Uploaded: 2025-10-01T01:29:11.000Z
Duration: 15 min 57 s
Description: Thousands of YouTube videos with English-Chinese subtitles! Now you can learn to understand native speakers, expand your vocabulary, and improve your pronunciation...

Adverbs of manner

When we look at images, we can easily recognize objects, semantics, and understand the 3D geometry of the world.

In this video, we will explore how AI learned to see without label data.

For a computer, an image is just a grid of numbers.

To make sense of these numbers, AI needs to extract meaningful features from an image.

Good features help us identify which regions of the image are similar to the red cross, revealing semantic structures even in a complex cluttered scene.

Let's look at a simple example to see why features are important.

Here we have data points from two groups, blue and orange.

In this space, building a linear classifier to separate the two groups is very hard.

No matter how we try, we can't find a good way to separate them.

Now let's transform the data into radial coordinates.

By changing our perspective on the data, separating the two groups with a straight line becomes much easier.

This example highlights the importance of having good feature representations.

But how do we design an effective feature representation?

One way is to learn features by training a new network to solve tasks like classification.

On top of the extracted features, we place an output projection head, typically a linear layer or a multilayer perceptron, that predicts class scores.

We then use a softmax layer to turn the scores into probabilities for each class.

We train the model to make its predicted probabilities match this label by minimizing cross-entropy loss.

This loss measures the difference between the predicted probability distributions from the softmax layer and the actual one-hot encoded label.

After training the model on a large label dataset, we can use its learned weights as a great starting point for new tasks.

This process is called transfer learning.

It allows us to fine-tune the model for a wide range of applications, such as fine-grained classification, object detection, or semantic segmentation.

However, class labels are often too simple.

They only tell us the main object in the image, like bear or cat.

But they do not describe all the rich, fine-grained details inside the image.

Natural language, like captions or descriptions, can give us much more information.

For example, a caption might say, a friendly bear waves hello, or a cat reclines on the pavement.

This tells us about objects, their actions, and scene.

To use this extra information, we add a text encoder.

The text encoder turns the caption or description into a feature vector, just like the image encoder does for the image.

Intuitively, we want the feature vectors for the image and the feature vector for its matching caption to be as similar as possible.

This means the model learns that the image and its captions are related.

At the same time, we want the image and text features for pairs that do not match to be far apart.

Let's visualize the image and text features as 2D vectors.

We normalize them so that they have unit lengths.

We measure similarity by looking at the angle between the two vectors.

Since the vectors are normalized, the dot product directly reflects how similar they are.

The closer the dot product is to 1, the smaller the angle and the more similar the features.

This is what we want to see after training.

Along the diagonal, the features from matching image-text pairs are well aligned.

Off the diagonal, the features from non-matching pairs become dissimilar.

This is known as contrastive language image pre-training, or CLIP.

By training on 400 million image-text pairs, the image encoder learns to capture and generalize a wide range of visual concepts.

But this type of training is expensive because it requires human annotation for every image.

Can we train our image encoder without any labels, using only images?

If we can achieve this, we can leverage the enormous amount of unlabeled images already available to train our models at scale.

This is the goal of self-supervised learning.

Just like supervised learning, we extract image features, add an output head, and use a loss function.

The challenge here is to create a supervision signal for training, but without relying on any human-provided labels.

Over the years, researchers have developed creative strategies for self-supervised learning.

For example, we can train the model to colorize an image, predict the rotation angle of an image, impan the center patch from the surrounding context, or more generally, predict missing local patches from randomly masked input images.

However, training on these manually designed tasks does not reinforce learning representations that are semantically rich or transferable.

Let's revisit the contrastive learning idea.

We extract image features from two different augmented views of input images.

The objective is to bring the embedding vectors from matching pairs closer, while pushing embeddings of different images far apart.

More specifically, we compute the pairwise similarity of all pairs using vector dot product.

Here, Sij denotes the similarity score between i's embedding vector and j's embedding vector.

We highlight positive pairs with yellow boxes, and negative pairs with blue boxes.

During training, the model learns to increase the similarity for positive pairs.

Intuitively, we train the model to extract feature representations that are stable under different augmented views.

When contrastive learning has been successful, making it work is challenging.

The main issue is that it needs a large number of negative samples to avoid learning trivial representations.

As a result, training requires very large batch size and is computationally expensive.

To tackle this challenge, let's look at how it could work with a single image.

Given an input image, we create two augmented views like different crops.

We call the image encoders a student and teacher models.

We extract features, feed them into a projection head to obtain logics, and then apply a softmax function to produce probability distributions.

With this setup, we train the student network to match the teacher's predictions by minimizing the cross entropy between them.

A key detail is that we stop the gradient from flowing into the teacher network during training.

In this setup, only the student network's parameters are updated to match the teacher's predictions, while teachers provide a stable target.

This approach is known as knowledge distillation.

Typically, we use it to distill knowledge from a powerful large teacher model to a smaller student model, making inference more efficient.

In our case, both student and teacher models have the same architectures and model size.

First, if we don't have labels from people, how do we know what a class is?

Fortunately, we can define the output dimension to be a large number like 65,000.

This enables the model to discover and represent a wide range of visual concepts without requiring us to define them in advance.

Second, in knowledge distillation, the teacher model is usually already trained.

But without label data, how do we get this teacher model?

One effective method is to update the teacher model's weight as a moving average of the student's weight over time.

This allows the teacher model to change gradually, providing a stable and consistent target for the student to learn from.

Specifically, after each training step, we update the teacher's parameters using the following formula.

The value lambda controls the update rate for the teacher model.

In practice, we set the initial value of lambda as 0.996 and gradually increase it over the course of training.

However, this approach alone is not enough to prevent the model from collapsing to trivial solutions.

For example, the model could end up producing outputs where a single dimension dominates the distributions, resulting in uninformative or collapsed representations.

To address this, we use a trick called centering.

Let's illustrate how it works with an example.

Suppose we have a batch of four samples, and our model predicts probabilities over three classes for each sample.

After applying softmax, most of the probability mass is concentrated in the first dimension.

Next, we calculate the average logic, the center, across the batch, and subtract the center from each of the teacher's logics.

The centering trick encourages the model to spread its predictions more evenly, rather than focusing on a single output.

Centering helps prevent collapse by ensuring the average output of the teacher network remains balanced across all output dimensions.

Instead of computing the center vector for each batch independently, we maintain a moving average of the center over time.