Ian Goodfellow's PhD Defense and the Quiet Death of Principled Probabilistic Models

---

Original Video

Title: IanGoodfellow PhD Defense Presentation Uploader: @nouiz Duration: 45:21 Published: 2014-09-03 Views: 166,465 | Likes: 2,393

---

Someone on my team shared this in our Tuesday paper discussion slot — not a fresh arxiv preprint, but a 12-year-old PhD defense video. Ian Goodfellow, University of Montreal, 2014. I almost skipped it. Then I watched the first ten minutes and cancelled my afternoon meetings.

What follows is not a summary. It's a meditation on what this defense reveals about the tectonic shift that was happening in deep learning research — a shift I lived through, and one that still shapes how I think about AI research at FindTube today.

The Two Halves of a Thesis

Goodfellow's defense is structured as four papers, but it really splits into two intellectual halves.

The first half — spike-and-slab sparse coding and multi-prediction deep Boltzmann machines — belongs to what I'd call the probabilistic era of deep learning. These are models where you define a probability distribution, derive a variational bound, construct an approximate posterior, and carefully optimize within the mathematical framework you've built. Every approximation is acknowledged. Every assumption is stated. The beauty lies in the rigor.

The second half — maxout networks and Street View digit transcription — belongs to a different world entirely. No variational inference. No partition functions. Just: define a network, write a loss function, run backpropagation, see what happens. The beauty lies in the results.

Watching Goodfellow navigate between these two worlds in 45 minutes felt like watching the geological record of a paradigm shift, compressed into a single talk.

What Spike-and-Slab Actually Taught Us

The spike-and-slab sparse coding paper is easy to dismiss in hindsight — Goodfellow himself admits it "didn't really push forward the state of the art a whole lot." But I think this undersells what it contributed.

The core idea is elegant: decompose an image into components where each component has a binary "spike" variable (is this edge present?) and a continuous "slab" variable (how strong is it?). This gives you true sparsity — the model can genuinely say "this feature is absent," unlike L1-regularized sparse coding where generated samples almost never have exact zeros.

This matters more than you might think. At FindTube, when we're doing temporal grounding — finding the exact moment in a video where a concept appears — we face an analogous problem. Our model needs to say "this concept is NOT present at timestamps 0 through 14:31, and IS present at 14:32." That's a spike. The confidence level is the slab. The insight that true sparsity improves generalization with limited labels? We use that intuition every day in our few-shot domain adaptation work.

The other lesson from spike-and-slab is about scaling. Goodfellow showed a log-space plot where every previous approach was stuck on a diagonal — you couldn't scale both data and model simultaneously. His breakthrough was redesigning the inference for GPU parallelism. This is a pattern I've seen repeatedly in my career: the theoretical bottleneck and the practical bottleneck are usually different things. The theory says "inference is intractable." The practice says "inference is slow because you're updating variables sequentially on a CPU." Fix the practical bottleneck, and suddenly the theoretical one becomes manageable.

The DBM Section: A Love Letter to Models That Think Backwards

The multi-prediction deep Boltzmann machine section is the one I want to talk about the most, because I think it represents a road not taken in deep learning — one that still haunts me.

Deep Boltzmann machines have feedback connections. Information flows both up and down. The model doesn't just extract features from pixels in a feed-forward sweep; it lets high-level knowledge (like "this is a cat") influence low-level representations (like which edges are salient). Goodfellow describes this as "corners can influence edges."

This is, in my view, one of the most important ideas in the entire talk, and it's embedded in a model that almost nobody uses anymore.

Why? Because training DBMs was — and Goodfellow is admirably honest about this — a nightmare. A four-stage pipeline involving layer-wise pretraining, model stitching, joint generative training with sampling-based approximations, and then bolting on a discriminative classifier with a mysterious variable deletion hack. The sampling required Markov Chain Monte Carlo, which has mixing issues that change unpredictably during training.

Goodfellow's multi-prediction training simplifies this to a single stage. Instead of sampling from the model (which is hard), you do variational inference to predict missing variables (which is easier). The result is cleaner, more stable, and more robust to hyperparameter choices.

But here's what strikes me: the box-and-whisker plots he shows comparing centering-based DBMs to multi-prediction DBMs reveal something important. Multi-prediction DBMs are much better for classification, while centering is better for log-likelihood of pixels. Goodfellow says "we don't really understand exactly why centering fails for classification."

That's the kind of honest admission that I find most valuable in research. At our Tuesday deep dives, I always push my team to identify the moment in a paper where the authors admit they don't understand something. That's usually where the next paper lives.

Maxout: The Activation Function as a Learnable Object

The maxout paper represents a phase transition in Goodfellow's thinking — from "let me carefully derive the right probabilistic framework" to "let me build something simple and see if it works better."

The idea is almost trivially simple: instead of applying a fixed nonlinearity like sigmoid or ReLU to a single linear response, take the maximum over two (or more) linear responses. That's it. No activation function to choose — the network learns its own.

What makes this intellectually interesting to me is the framing. Goodfellow points out that with two linear pieces, you can learn absolute value rectification, standard ReLU, or any other piecewise linear function. The activation function becomes a learnable parameter rather than a design choice.

This connects to something I think about constantly in the context of FindTube's multimodal fusion architecture. How many of our design choices — attention patterns, pooling strategies, feature concatenation schemes — are actually arbitrary? How many could be replaced by a more general mechanism that learns the right operation from data?

The maxout results also illustrate a principle I call the efficiency frontier: maxout achieves better performance than rectifier networks with more parameters. It's not just about being better — it's about being better per parameter. In production AI where inference latency and memory footprint matter, this distinction is everything. Ben on our ML team would say "GPU hours are precious; waste not." I'd add: GPU milliseconds at inference time are even more precious.

Street View: When the Research Becomes the Product

The Street View house number paper is where Goodfellow the researcher becomes Goodfellow the engineer, and the transition is seamless.

Three design decisions stand out to me:

First, the bounded output assumption. At most 5 digits. Anything longer gets routed to a human. This is not a limitation — it's a system design choice. You define the scope of what the model handles, and you design a fallback for everything else. At FindTube, we do the same: if our temporal grounding model's confidence drops below threshold, we fall back to keyword-based timestamp matching. The system as a whole is more reliable than any single component.

Second, the joint inference at test time. During training, you know the true sequence length from labels. During inference, you don't. Goodfellow describes a procedure where the length predictor and digit predictors collaborate — a digit position with diffuse probability (0.1 across all classes) is evidence against including that position. This is essentially model self-awareness: the system uses its own uncertainty as a signal.

We apply this exact principle in our search ranking. When the multimodal model is uncertain about a timestamp match, that uncertainty itself becomes a feature for the ranker. Confidence about confidence — it sounds recursive, but it works.

Third, the depth ablation. Networks with more parameters but constant depth just overfit. More depth — more sequential processing stages — genuinely helps. Goodfellow frames this through LeCun's lens of "programs that execute more than one instruction," but I think there's an even deeper point: some tasks have an intrinsic sequential complexity that cannot be parallelized away. You need to segment before you can recognize. You need to detect edges before you can find contours. You need to understand what a number looks like before you can read a sequence of them.

In kendo, we call this isshin — single-minded focus on each strike in sequence. You cannot execute the second strike while thinking about the first. Each must complete before the next begins. Depth in neural networks serves the same function: it enforces sequential discipline on the computation.

The Ghost of GANs

The elephant in the room — or rather, the elephant conspicuously absent from the room — is GANs. The Generative Adversarial Network paper was published in 2014, the same year as this defense. But GANs appear nowhere in this talk.

And yet, in retrospect, every ingredient is here. The variational inference experience from spike-and-slab. The deep generative modeling intuition from DBMs — especially the understanding that feedback between model components enables richer representations. The gradient flow insights from maxout — ensuring learning can proceed without getting stuck. The engineering pragmatism from Street View — knowing when mathematical elegance should yield to "does it work."

I find this deeply instructive. The components of a breakthrough are often visible years before the breakthrough itself. They just don't look like components at the time. They look like four loosely related papers stapled together for a PhD defense.

This is why I insist on our "20% blue sky" policy at FindTube's AI lab. Last month, one of my researchers spent a day reading about how rodents process visual sequences. It seemed completely disconnected from video search. But the minimal temporal reasoning circuits they found in rodent visual cortex gave us an idea for a lightweight attention module that processes video timestamps with 40% fewer FLOPs than our current approach. We're still testing it, but the initial results are promising.

You never know which branch of the lichen will be the one that finds sunlight.

What I Took Away

Watching this defense in 2026, knowing everything that came after — GANs, diffusion models, transformers, foundation models — gives it an almost archaeological quality. You can see the fossils of ideas that would later dominate the field, embedded in the sedimentary layers of ideas that didn't survive.

But I want to resist the temptation to judge the "failed" ideas harshly. Spike-and-slab sparse coding didn't become the dominant paradigm, but it taught its author about sparsity and scaling. Multi-prediction DBMs didn't replace backpropagation, but they taught their author about the limitations of sampling and the power of unified training. These were not dead ends — they were the training data for the researcher himself.

Goodfellow closes by saying unsupervised learning "hasn't yet fully reached its potential." Twelve years later, with self-supervised learning powering everything from CLIP to GPT, I think we can say it finally did — just not in the form anyone expected in 2014.

The best research is patient. Like lichen growing on a rock: invisible day to day, transformative over decades.

— Yuki Tanaka, VP of AI Research @ FindTube.ai