Transfer Learning (4): Few-Shot Learning

Show a child one photograph of a pangolin and they will spot pangolins for life. Show a deep learning model one photograph and it will give you a uniformly random guess. Few-shot learning is the field that closes that gap: building classifiers that work with only one to ten labeled examples per class.

The trick is not to memorize individual classes harder. It is to learn how to learn from very few examples, then carry that ability over to brand-new classes at test time. This article covers the two families that dominate the field today: metric learning, which learns a good distance function, and meta-learning, which learns a good initialization.

What You Will Learn#

• The N-way K-shot evaluation protocol and why standard training fails on it
• Metric learning: Siamese, Prototypical, Matching, and Relation networks
• Meta-learning: MAML and its first-order cousins (FOMAML, Reptile)
• Episodic training: matching training-time difficulty to test-time difficulty
• A clean, end-to-end Prototypical Network implementation in PyTorch

Prerequisites: Parts 1-2 of this series; comfort with PyTorch and basic optimization.

The Few-Shot Challenge#

Problem Setup: N-way K-shot#

The community uses a single, shared evaluation protocol so that papers are comparable:

• N-way: the model must classify among $N$ classes.
• K-shot: for each class, only $K$ labeled examples are available.

A “5-way 1-shot” task is therefore: here is one labeled image from each of five classes you have never seen before; now classify a batch of new query images.

Each evaluation episode consists of:

• a support set $\mathcal{S} = \{(x_i, y_i)\}_{i=1}^{NK}$ — the $N \times K$ labeled examples,
• a query set $\mathcal{Q} = \{(x_j, y_j)\}_{j=1}^{NQ}$ — the unlabeled images to classify (with hidden labels used only to measure accuracy).

Reported numbers are averages over hundreds or thousands of episodes drawn from a held-out novel-class set, with 95% confidence intervals because the per-episode variance is large.

Why Standard Training Fails#

Three forces conspire against a vanilla classifier:

• Data scarcity. With $K = 1$ you literally cannot estimate a within-class variance. With $K = 5$ you can, but barely.
• Overfitting. A high-capacity network will memorize the support examples instead of learning a class-discriminative rule.
• Inter-class similarity. Novel classes drawn from the same domain (e.g. two breeds of dog) often differ only in subtle features that a randomly initialized classifier has no reason to attend to.

Empirical risk minimization with weight decay is not enough: regularization stops parameters from blowing up, but it does not inject the inductive bias required to generalize from a single image.

The Core Insight#

To learn from few samples you need prior knowledge. Few-shot learning gets that prior by training on a large set of base classes (with many examples each), then evaluating on disjoint novel classes (with few). The two main routes are:

1. Metric learning — train a backbone whose embedding space already separates classes, so a fresh class can be characterized by the location of its few support points. Classify queries by their distance in this space.
2. Meta-learning — train across many simulated few-shot tasks so the network learns to be adapted by a few gradient steps. Treat “fast adaptation” itself as the thing to optimize.

Both share the same data split (base vs. novel) but invest the prior knowledge differently: metric learning bakes it into the embedding; meta-learning bakes it into the optimization initialization.

Metric Learning: Classification by Distance#

The metric-learning recipe is one sentence long: learn an embedding $f_\theta$ such that same-class samples cluster together and different-class samples lie far apart, then classify a query by its proximity to the support points.

Siamese Networks#

with $y = 1$ for same-class pairs (pull together) and $y = 0$ for different-class pairs (push apart until the distance exceeds margin $m$ ). At test time, classify a query by the label of its nearest support sample.

Prototypical Networks#

Prototypical networks improve on the pairwise picture by collapsing each support class into a single point.

Computing prototypes#

Geometrically it is the centroid of the class in embedding space.

Classification#

Train end-to-end with cross-entropy on the query predictions of each episode.

Why prototypes are principled#

If we model class-conditional embeddings as Gaussians with shared isotropic covariance, $P(x \mid y = c) = \mathcal{N}(\mu_c, \sigma^2 I)$ , then the maximum-likelihood class is exactly the nearest centroid. Prototypical networks are therefore the deep-learning incarnation of the Bayes-optimal classifier under that (admittedly strong) assumption — which is why it tends to work so well in practice.

A second, cleaner, observation: under squared-Euclidean distance the decision boundary between any two classes is a hyperplane in embedding space. So Prototypical networks are equivalent to a linear classifier in the learned space, but with the linear weights tied to the prototype geometry.

Matching Networks#

Matching networks replace the hard nearest-prototype rule with a soft attention over the entire support set.

Here $y_i$ is a one-hot label vector, so the prediction is a convex combination of one-hots weighted by attention.

The other contribution of the paper is full context embeddings: a bidirectional LSTM is run over the support set so each support embedding is aware of every other support sample. The intuition is that what counts as a discriminative feature depends on the other classes you are trying to separate from — and the LSTM lets the network express that.

Relation Networks#

The training target is $r_{q, c} = \mathbb{1}\{y_q = c\}$ with mean-squared-error loss; both modules are trained jointly. Why bother? Fixed metrics implicitly assume the embedding space is isotropic — every dimension counts equally. A learned metric lets the network downweight dimensions that turn out to be uninformative for the task at hand.

Meta-Learning: Learning to Learn#

Where metric learning bakes the prior into the embedding, meta-learning bakes it into the optimization process itself. The model is trained across many tasks so that adapting it to a new task takes only a handful of gradient steps.

MAML: Model-Agnostic Meta-Learning#

MAML’s idea is simple and surprisingly effective: search for an initialization $\theta$ such that one or two gradient steps on any new task’s support set already produce a good model.

Algorithm#

For each sampled task $\mathcal{T}_i$ (with its own support and query sets):

1. Inner loop (per-task adaptation). Take one (or a few) gradient steps on the support loss:

2. Outer loop (meta-update). Evaluate the adapted parameters on the query set and update the initialization:

The outer-loop gradient differentiates through the inner-loop update, which involves second derivatives of the support loss with respect to $\theta$ — a Hessian-vector product.

First-order approximation (FOMAML)#

which is just the gradient at the adapted point, evaluated as if $\theta_i'$ did not depend on $\theta$ . Cost drops to $O(d)$ , and reported accuracies barely change.

Geometric intuition#

MAML pushes $\theta$ toward a region of the loss landscape that is flat with respect to fast adaptation: from this point a few steps in any task-specific direction reach a low loss. Think of $\theta$ as a universal launching pad rather than a universally-good model.

Reptile: Even Simpler#

That’s the whole algorithm. Despite the simplicity it works almost as well as MAML, because moving the meta-parameters toward task-specific solutions across many tasks ends up locating $\theta$ near a shared sweet spot.

*Reported in the original papers; numbers vary across implementations.

Episodic Training#

Standard supervised training shows the network the entire base-class dataset and asks it to classify. Episodic training reframes the entire training loop to look like the test loop.

How an episode is built#

Each iteration:

1. Sample $N$ classes from the base-class pool.
2. Sample $K$ examples per class for the support set.
3. Sample $Q$ additional examples per class for the query set.
4. Train the model to classify the queries given only that support set.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

for epoch in range(num_epochs):
    for episode in range(episodes_per_epoch):
        classes = sample(base_classes, N)
        support = sample_from_classes(classes, K)
        query   = sample_from_classes(classes, Q)

        prototypes = compute_prototypes(support)
        logits     = -distance(query, prototypes)
        loss       = cross_entropy(logits, query_labels)

        loss.backward()
        optimizer.step()

Why this matters#

The model never gets to see the full base-class dataset at once. Every gradient update simulates a few-shot task, so the inductive bias the network develops is precisely the one needed at test time. This is curriculum learning where the curriculum is the test-time conditions.

A good sanity check: turn off episodic training and just train a flat $|C_{\text{base}}|$ -way classifier, then drop a linear head on the frozen features. With a strong backbone (deep ResNet, large augmentation) this “Baseline++” recipe is competitive with metric- and meta-learning approaches — a result Chen et al. (ICLR 2019) used to argue that episodic training matters less than people thought, and that backbone capacity matters more.

How Well Does Any of This Work?#

The numbers above are from the original papers (with later work routinely surpassing them by using larger backbones and pre-training tricks). Two things to take away:

• The 1-shot vs. 5-shot gap is huge. Going from one example to five typically adds 10-20 percentage points — a reminder that even a tiny amount of data dominates clever architecture choices.
• Methods cluster. Once the backbone is held fixed, Prototypical, Matching, Relation, and MAML-family numbers land within a few points of each other. Pick by engineering taste (simplicity, compute budget, tooling) rather than chasing the last point of accuracy.

Complete Implementation: Prototypical Networks#

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150

import torch
import torch.nn as nn
import numpy as np
from tqdm import tqdm

class ConvBlock(nn.Module):
    """Conv -> BN -> ReLU -> MaxPool, the standard miniImageNet building block."""
    def __init__(self, in_ch, out_ch):
        super().__init__()
        self.block = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 3, padding=1),
            nn.BatchNorm2d(out_ch),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2),
        )

    def forward(self, x):
        return self.block(x)

class ProtoNetEncoder(nn.Module):
    """4-layer CNN encoder. Maps an 84x84 RGB image to a 1600-d vector."""
    def __init__(self, in_channels=3, hidden_dim=64):
        super().__init__()
        self.encoder = nn.Sequential(
            ConvBlock(in_channels, hidden_dim),   # 84 -> 42
            ConvBlock(hidden_dim, hidden_dim),    # 42 -> 21
            ConvBlock(hidden_dim, hidden_dim),    # 21 -> 10
            ConvBlock(hidden_dim, hidden_dim),    # 10 ->  5
        )

    def forward(self, x):
        return self.encoder(x).view(x.size(0), -1)

class PrototypicalNetwork(nn.Module):
    def __init__(self, encoder):
        super().__init__()
        self.encoder = encoder

    def compute_prototypes(self, support_emb, support_labels, n_way):
        """Average embeddings per class to form prototypes."""
        prototypes = [
            support_emb[support_labels == c].mean(dim=0)
            for c in range(n_way)
        ]
        return torch.stack(prototypes)  # (n_way, embed_dim)

    def forward(self, support_imgs, support_lbls, query_imgs, n_way):
        support_emb = self.encoder(support_imgs)
        query_emb   = self.encoder(query_imgs)
        prototypes  = self.compute_prototypes(support_emb, support_lbls, n_way)
        # Negative squared Euclidean distance == logits
        dists  = torch.cdist(query_emb, prototypes, p=2)
        return -dists

class EpisodeSampler:
    """Yields N-way K-shot episodes from a flat (data, labels) array."""
    def __init__(self, data, labels, n_way, n_support, n_query, n_episodes):
        self.data, self.labels = data, labels
        self.n_way, self.n_support = n_way, n_support
        self.n_query, self.n_episodes = n_query, n_episodes
        self.classes = np.unique(labels)
        self.class_indices = {c: np.where(labels == c)[0] for c in self.classes}

    def __iter__(self):
        for _ in range(self.n_episodes):
            yield self._sample()

    def _sample(self):
        chosen = np.random.choice(self.classes, self.n_way, replace=False)
        s_imgs, s_lbls, q_imgs, q_lbls = [], [], [], []

        for new_label, c in enumerate(chosen):
            idxs = np.random.choice(
                self.class_indices[c],
                self.n_support + self.n_query,
                replace=False,
            )
            for idx in idxs[:self.n_support]:
                s_imgs.append(self.data[idx]); s_lbls.append(new_label)
            for idx in idxs[self.n_support:]:
                q_imgs.append(self.data[idx]); q_lbls.append(new_label)

        return (
            torch.stack([torch.FloatTensor(x) for x in s_imgs]),
            torch.LongTensor(s_lbls),
            torch.stack([torch.FloatTensor(x) for x in q_imgs]),
            torch.LongTensor(q_lbls),
        )

def train(model, train_data, train_lbls, val_data, val_lbls,
          n_way=5, n_support=5, n_query=15, n_episodes=100,
          num_epochs=50, lr=1e-3, device='cpu'):
    """Episodic training with periodic validation."""
    model = model.to(device)
    optim = torch.optim.Adam(model.parameters(), lr=lr)
    crit = nn.CrossEntropyLoss()
    best = 0.0

    for epoch in range(num_epochs):
        # ---- train ----
        model.train()
        sampler = EpisodeSampler(train_data, train_lbls,
                                 n_way, n_support, n_query, n_episodes)
        loss_sum, acc_sum = 0.0, 0.0
        for s_img, s_lbl, q_img, q_lbl in tqdm(sampler, desc=f'Epoch {epoch+1}'):
            s_img, s_lbl = s_img.to(device), s_lbl.to(device)
            q_img, q_lbl = q_img.to(device), q_lbl.to(device)

            logits = model(s_img, s_lbl, q_img, n_way)
            loss = crit(logits, q_lbl)

            optim.zero_grad(); loss.backward(); optim.step()
            loss_sum += loss.item()
            acc_sum  += (logits.argmax(1) == q_lbl).float().mean().item()

        # ---- validate ----
        model.eval()
        val_sampler = EpisodeSampler(val_data, val_lbls,
                                     n_way, n_support, n_query, n_episodes)
        val_acc = 0.0
        with torch.no_grad():
            for s_img, s_lbl, q_img, q_lbl in val_sampler:
                s_img, s_lbl = s_img.to(device), s_lbl.to(device)
                q_img, q_lbl = q_img.to(device), q_lbl.to(device)
                logits = model(s_img, s_lbl, q_img, n_way)
                val_acc += (logits.argmax(1) == q_lbl).float().mean().item()
        val_acc /= n_episodes
        print(f"  train_loss={loss_sum/n_episodes:.4f}  "
              f"train_acc={acc_sum/n_episodes:.4f}  val_acc={val_acc:.4f}")

        if val_acc > best:
            best = val_acc
            torch.save(model.state_dict(), 'best_protonet.pt')

# ---- demo with random data ----
if __name__ == '__main__':
    num_classes, samples_per_class, img_size = 64, 600, 84
    all_data = np.random.randn(num_classes * samples_per_class,
                               3, img_size, img_size).astype(np.float32)
    all_labels = np.repeat(np.arange(num_classes), samples_per_class)

    train_classes = int(num_classes * 0.8)
    train_mask = all_labels < train_classes
    val_mask   = all_labels >= train_classes

    encoder = ProtoNetEncoder()
    model = PrototypicalNetwork(encoder)
    train(model, all_data[train_mask], all_labels[train_mask],
          all_data[val_mask], all_labels[val_mask],
          n_way=5, n_support=5, n_query=15, num_epochs=10)

Code Walkthrough#

Two implementation notes worth highlighting:

• torch.cdist(..., p=2) returns the Euclidean (not squared) distance. Negating it as logits is fine for argmax but technically does not match the Bayes-optimal Gaussian-mean derivation. In practice the difference does not matter; if you want exact correspondence, square it.
• Always relabel the support classes to $0, \ldots, N-1$ inside the sampler so that the cross-entropy targets have the expected shape.

Prototypical Networks: Bayes-Optimality and Variance-Adjusted Prototypes#

The “use the class mean” rule is not folklore. It falls out of Bayes’ theorem the moment you commit to a Gaussian model of the embedding space, and the derivation is short enough to do by hand.

The theorem#

Sketch of the proof#

The first term is independent of $c$ and drops out of the argmax. What remains is precisely $-\|x - \mu_c\|^2 / (2\sigma^2)$ — maximizing it is the same as minimizing squared distance to the mean.

So Prototypical Networks are not a heuristic. They are the deep-learning realization of an MAP classifier under one specific (and admittedly strong) generative assumption.

When the shared-variance assumption breaks#

The classifier becomes a variance-weighted prototype rule — closer-knit clusters get a higher implicit confidence, scattered ones get downweighted.

Variance-adjusted PyTorch#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

import torch
import torch.nn as nn

class WeightedProtoNet(nn.Module):
    """Prototypical Network with per-class variance estimated from the support set."""
    def __init__(self, encoder, eps=1e-3):
        super().__init__()
        self.encoder = encoder
        self.eps = eps  # floor on variance to avoid blow-up at K=1

    def class_stats(self, support_emb, support_lbls, n_way):
        prototypes, inv_var = [], []
        for c in range(n_way):
            mask = (support_lbls == c)
            x_c  = support_emb[mask]                           # (K, D)
            mu_c = x_c.mean(dim=0)                             # (D,)
            # Isotropic per-class variance: average over feature dims.
            if x_c.size(0) > 1:
                var_c = ((x_c - mu_c) ** 2).mean().clamp_min(self.eps)
            else:
                # K=1 case: no variance available, fall back to a global prior.
                var_c = torch.tensor(1.0, device=mu_c.device)
            prototypes.append(mu_c)
            inv_var.append(1.0 / var_c)
        return torch.stack(prototypes), torch.stack(inv_var)   # (N,D), (N,)

    def forward(self, s_imgs, s_lbls, q_imgs, n_way):
        s_emb = self.encoder(s_imgs)
        q_emb = self.encoder(q_imgs)
        protos, inv_var = self.class_stats(s_emb, s_lbls, n_way)

        # Weighted squared distance: -||q - mu_c||^2 / (2 sigma_c^2) - 0.5 d log sigma_c^2
        sq = ((q_emb.unsqueeze(1) - protos.unsqueeze(0)) ** 2).sum(dim=-1)  # (Q, N)
        log_norm = -0.5 * q_emb.size(1) * torch.log(1.0 / inv_var)          # (N,)
        return -0.5 * sq * inv_var + log_norm                               # logits

The clamp on var_c matters more than it looks. Without it, a class whose support points happen to land on top of each other gets infinite confidence and the softmax saturates.

Numbers on miniImageNet#

The lift is small but consistent across seeds. It is not free: with $K = 1$ the per-class variance is undefined, so the model falls back to the unit-variance prior and you should expect zero benefit. We see real gains from $K = 3$ onward, where each class has enough support points to estimate a sensible scale.

Caveat: noise in the variance estimate#

A variance computed from $K = 5$ samples has a relative error on the order of $\sqrt{2/(K-1)} \approx 70\%$ . That is enormous. The reason variance-adjustment helps at all is that even a noisy estimate is better than the implicit “all classes have the same variance” prior — but if you push to $K = 1$ or $K = 2$ the noise dominates and accuracy degrades below the baseline.

with $\lambda$ scheduled by $K$ . This is a James-Stein-style shrinkage estimator and recovers vanilla ProtoNet at $\lambda = 0$ .

A second, less-discussed pitfall: per-class variance is a single scalar in the isotropic model above. Real embedding clusters are anisotropic, with elongated directions and tight ones. Letting $\sigma_c^2$ be a vector (per-feature) instead of a scalar gives a small additional bump but also amplifies the noise problem — the variance of a per-feature variance estimate is the same $\sqrt{2/(K-1)}$ , applied to each of $D$ dimensions independently.

But what if the underlying metric isn’t Euclidean at all?

MAML Inner-Outer Loop on a 2D Toy#

MAML’s two-loop structure is easy to write down and surprisingly hard to see. A 2D toy regression problem makes the geometry concrete: you can plot the meta-initialization, the per-task adapted points, and the trajectories the inner loop walks along.

The task family#

The model is a tiny MLP $f_\theta: \mathbb{R} \to \mathbb{R}$ . We treat the parameter $\theta$ as the thing being meta-learned: a single $\theta_0$ that adapts in 5 inner-loop steps to any $(\omega_t, \phi_t)$ .

For visualization we sometimes restrict $\theta$ to two scalars and plot the per-task loss surface $\mathcal{L}_t(\omega, \phi)$ directly — the inner loop is then a literal walk along the surface, and the outer loop is the choice of starting point.

Full MAML in PyTorch#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

import torch
import torch.nn as nn
from torch.func import functional_call

class TinyMLP(nn.Module):
    def __init__(self, hidden=40):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(1, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, 1),
        )
    def forward(self, x):
        return self.net(x)

def sample_task(K=10):
    omega = torch.empty(1).uniform_(0.5, 2.0).item()
    phi   = torch.empty(1).uniform_(0.0, 6.2832).item()
    x = torch.empty(K, 1).uniform_(-5.0, 5.0)
    y = torch.sin(omega * x + phi)
    return x, y

def inner_step(model, params, x, y, alpha):
    """One SGD step on the support loss, returning *new* params (not in-place)."""
    pred = functional_call(model, params, (x,))
    loss = ((pred - y) ** 2).mean()
    grads = torch.autograd.grad(loss, params.values(), create_graph=True)
    return {name: p - alpha * g for (name, p), g in zip(params.items(), grads)}

def outer_step(model, meta_params, meta_optim, K=10, n_inner=5, alpha=0.01,
               n_tasks=4):
    """Sum query losses across n_tasks tasks, take one meta gradient step."""
    meta_optim.zero_grad()
    meta_loss = 0.0
    for _ in range(n_tasks):
        x_s, y_s = sample_task(K)
        x_q, y_q = sample_task(K)   # fresh draw -> query split
        params = meta_params
        for _ in range(n_inner):
            params = inner_step(model, params, x_s, y_s, alpha)
        pred_q = functional_call(model, params, (x_q,))
        meta_loss = meta_loss + ((pred_q - y_q) ** 2).mean()
    meta_loss.backward()         # second-order gradient flows here
    meta_optim.step()
    return meta_loss.item() / n_tasks

# --- training ---
model = TinyMLP()
meta_params = {n: p.clone().detach().requires_grad_(True)
               for n, p in model.named_parameters()}
meta_optim = torch.optim.Adam(meta_params.values(), lr=1e-3)

for it in range(10000):
    loss = outer_step(model, meta_params, meta_optim)
    if it % 1000 == 0:
        print(f"iter {it}: meta-loss={loss:.4f}")

The key line is create_graph=True in torch.autograd.grad. Without it, the inner-loop gradient is detached and the outer-loop backward has nothing to differentiate through — you’ve silently fallen back to FOMAML. With it, PyTorch keeps the inner graph alive and the second derivative (Hessian-vector product) is computed when meta_loss.backward() runs.

Cost: the Hessian-vector product#

Each outer step costs roughly $2 \times n_{\text{inner}}$ forward passes plus the backward through the unrolled inner loop. The peak memory scales as $O(n_{\text{inner}} \cdot |\theta|)$ because every intermediate parameter set must be retained for the backward. For our tiny MLP this is invisible. For a ResNet-12 it is the reason FOMAML and Reptile exist.

Numbers#

After 10k outer steps, evaluating on 100 held-out tasks with 5 inner steps:

The MAML initialization gets in 5 steps what a cold start needs 50 steps to match. That ratio — not the absolute accuracy — is the thing meta-learning buys you.

If you ever plot the loss surface for a single task and overlay the inner-loop path starting from $\theta_0$ , you’ll see that $\theta_0$ sits in a valley that is flat with respect to all task-specific minima. It is not the best place for any single task, but it is the best place from which to walk to any task in 5 steps.

Why second-order matters here (and when it doesn’t)#

In our toy, second-order MAML and FOMAML give nearly identical held-out MSE — 0.04 vs. 0.05. The Hessian-vector product is real but small in magnitude because the inner loss landscape is well-behaved: the per-task surfaces are smooth bowls and the linearization that FOMAML implicitly performs is tight. On harder problems (image classification with deep encoders, RL with sharp reward landscapes) the second-order term is what tells the meta-update how the inner-loop trajectory itself bends in response to a parameter change — and dropping it costs a few percentage points.

The practical rule: start with FOMAML or Reptile because they fit in a single GPU and are easy to debug; only enable second-order MAML if you have evidence that the inner loss surface is curved enough to matter.

Bridge: prototypes plus a learned init still both assume Euclidean geometry. The next step is to learn the geometry too.

Beyond Euclidean: Learned Task-Conditional Metrics#

Squared-Euclidean distance treats every embedding dimension as equally important. That is fine when the network has been trained long enough to whiten its features — and a problem the moment two classes differ along a low-variance direction the embedding has chosen to compress away.

Mahalanobis distance#

Setting $M = I$ recovers Euclidean. Setting $M = \Sigma^{-1}$ for a class-conditional covariance recovers Mahalanobis distance in its statistical sense. Learning $M$ end-to-end interpolates between the two and lets the network rescale dimensions to whatever the task demands.

To keep $M$ PSD, parametrize as $M = L L^T$ with $L$ unconstrained — gradients flow through $L$ , and $M$ is PSD by construction.

Task-conditional metrics#

A fixed $M$ is still a global commitment. A more flexible move is to let $M$ depend on the episode: a small hypernetwork reads the support set’s statistics and emits the per-episode metric.

The summary can be as light as the concatenated mean-and-trace-of-covariance per class. The hypernetwork is then a small MLP that outputs the entries of $L$ .

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

import torch
import torch.nn as nn

class TaskConditionalMetric(nn.Module):
    """Generate a per-episode Mahalanobis matrix from support-set summary stats."""
    def __init__(self, embed_dim, hidden=128, rank=None):
        super().__init__()
        self.D = embed_dim
        self.rank = rank or embed_dim
        # Summary: per-class (mean, mean-of-squared-deviations) -> 2*D scalars.
        # Hypernet collapses across classes via averaging, then maps to L.
        self.hyper = nn.Sequential(
            nn.Linear(2 * embed_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, embed_dim * self.rank),
        )

    def summary(self, support_emb, support_lbls, n_way):
        feats = []
        for c in range(n_way):
            x_c  = support_emb[support_lbls == c]
            mu   = x_c.mean(dim=0)
            var  = ((x_c - mu) ** 2).mean(dim=0) if x_c.size(0) > 1 else torch.zeros_like(mu)
            feats.append(torch.cat([mu, var], dim=-1))
        return torch.stack(feats).mean(dim=0)            # (2D,)

    def forward(self, support_emb, support_lbls, n_way):
        summary = self.summary(support_emb, support_lbls, n_way)
        L = self.hyper(summary).view(self.D, self.rank)
        M = L @ L.T + 1e-3 * torch.eye(self.D, device=L.device)   # PSD, well-conditioned
        return M

def mahalanobis_logits(query_emb, prototypes, M):
    diff = query_emb.unsqueeze(1) - prototypes.unsqueeze(0)       # (Q, N, D)
    # (Q, N, D) @ (D, D) -> (Q, N, D); then dot with diff -> (Q, N)
    return -torch.einsum('qnd,de,qne->qn', diff, M, diff)

Drop this module into a Prototypical Network and replace torch.cdist with mahalanobis_logits(q_emb, protos, M).

Numbers on miniImageNet (5-way 1-shot)#

The trend is consistent: more flexibility in the metric helps, up to the point where parameter count starts to overfit the meta-train set.

Practical caveat: overfitting and weight decay#

The hypernetwork adds parameters that see only the support summary — a bottleneck of just $2D$ scalars per episode. With a small meta-train set (the standard 64 base classes of miniImageNet) it is easy to memorize a metric that works for the training tasks and generalizes poorly. Three things help in practice: (1) a low-rank parametrization (rank << D), (2) explicit weight decay on the hypernet, (3) dropout on the summary itself before it enters the hyper-MLP.

Cross-domain wrinkle#

Task-conditional metrics learned on one domain rarely transfer to another. The Cross-Domain Few-Shot Learning benchmark (Guo et al. 2020) showed that methods which beat ProtoNet on miniImageNet routinely underperform it when evaluated on CropDisease, EuroSAT, ISIC, or ChestX. The metric module learns to exploit domain-specific feature statistics — which is exactly what makes it brittle when the target domain shifts.

A safe rule: use a learned metric only when meta-train and meta-test distributions are well-matched. Across domains, fall back to plain Euclidean and rely on a stronger backbone instead.

Diagnostic: when does the metric actually help?#

A quick check before you commit to a learned metric: compute the between-class to within-class variance ratio in your frozen embedding space, on a held-out set of base classes. If the ratio is high (say, > 5), the embedding has already done the geometric work and Euclidean will be hard to beat. If the ratio is low (< 2), there is room for a learned $M$ to rotate and rescale dimensions and you should see clear gains.

This same ratio also predicts when variance-adjusted prototypes pay off: low ratio means classes overlap, which means per-class scale information is informative; high ratio means classes are already well-separated, and the variance estimate is just adding noise.

Bridge: with the metric, the prototypes, and the initialization all learnable, the next question is how to choose between these levers — which is what the benchmark numbers in the next section answer.

FAQ#

How is few-shot learning different from ordinary transfer learning?#

It is the limit case. Transfer learning assumes you have at least hundreds of target labels, so a fine-tuned head can do most of the work. Few-shot learning has 1-10. That gap is large enough that you need training-time machinery — episodic sampling, metric or meta objectives — not just a downstream training trick.

Why do Prototypical networks use the mean as the prototype?#

Under Gaussian class-conditionals with shared isotropic covariance, the class mean is the Bayes-optimal classifier. The mean is also robust enough to be useful even when that assumption fails — especially for $K \ge 5$ .

MAML or Prototypical Networks — which should I use?#

Default to Prototypical Networks: simpler, faster, the prototypes are interpretable, and they tend to match or beat MAML on standard image benchmarks. Reach for MAML when (a) the tasks are diverse and look qualitatively different from one another, (b) the data is non-image and you do not have a great pretrained embedding, or (c) you specifically need adaptation that updates the entire network rather than just a final classifier.

How many base classes do I need?#

More is always better for generalization. Standard benchmarks use 64 base classes (miniImageNet) up to 1200+ (Omniglot). With fewer than ~30 base classes you tend to see severe overfitting to the base set itself, and novel-class accuracy collapses.

Does any of this work for non-image data?#

Yes. Prototypical Networks work for anything with a meaningful embedding — text (use a transformer encoder), graphs (use a GNN), audio (use a spectrogram CNN). MAML and Reptile are model-agnostic by design. The episodic protocol does not care about modality.

Why are confidence intervals always reported?#

The per-episode accuracy variance is large — a single hard episode can swing 10-20 points. Reporting the mean over a few hundred episodes plus a 95% CI is the only way to make numbers comparable across papers.

Summary#

Few-shot learning attacks deep learning’s biggest practical bottleneck: data scarcity in the long tail.

• Metric learning (Siamese, Prototypical, Matching, Relation Networks) learns an embedding space where distance equals dissimilarity. Simple, fast, interpretable. Prototypical Networks are the workhorse default.
• Meta-learning (MAML, FOMAML, Reptile) learns an initialization from which a few gradient steps reach the optimum of any new task. More flexible, costlier, less interpretable.
• Episodic training is the unifying training paradigm: each iteration is a fresh few-shot task, so training-time difficulty matches test-time difficulty.

Across families, accuracies converge once the backbone is held fixed — a reminder that backbone capacity and pretraining quality matter at least as much as the few-shot algorithm on top.

Next: Part 5 — Knowledge Distillation , where we compress a large teacher model into a small student that mimics it.

References#

1. Snell et al. (2017). Prototypical Networks for Few-shot Learning. NeurIPS. arXiv:1703.05175 ↗
2. Finn et al. (2017). Model-Agnostic Meta-Learning (MAML). ICML. arXiv:1703.03400 ↗
3. Vinyals et al. (2016). Matching Networks for One Shot Learning. NeurIPS. arXiv:1606.04080 ↗
4. Sung et al. (2018). Learning to Compare: Relation Network for Few-Shot Learning. CVPR. arXiv:1711.06025 ↗
5. Nichol et al. (2018). On First-Order Meta-Learning Algorithms (Reptile). arXiv:1803.02999 ↗
6. Koch et al. (2015). Siamese Neural Networks for One-shot Image Recognition. ICML Deep Learning Workshop.
7. Chen et al. (2019). A Closer Look at Few-shot Classification. ICLR. arXiv:1904.04232 ↗
8. Wang et al. (2020). Generalizing from a Few Examples: A Survey on Few-Shot Learning. ACM Computing Surveys. arXiv:1904.05046 ↗