Brendan Lake (New York University) “Towards More Human-Like Learning in Machines: Bridging the Data and Generalization Gaps”

Abstract
There is an enormous data gap between how AI systems and children learn language: The best LLMs now learn language from text with a word count in the trillions, whereas it would take a child roughly 100K years to reach those numbers through speech (Frank, 2023, “Bridging the data gap”). There is also a clear generalization gap: whereas machines struggle with systematic generalization, people excel. For instance, once a child learns how to “skip,” they immediately know how to “skip twice” or “skip around the room with their hands up” due to their compositional skills. In this talk, I’ll describe two case studies in addressing these gaps:
1) The data gap: We train deep neural networks from scratch (using DINO, CLIP, etc.), not on large-scale data from the web, but through the eyes and ears of a single child. Using head-mounted video recordings from a child (61 hours of video slices over 19 months), we show how deep neural networks can acquire many word-referent mappings, generalize to novel visual referents, and achieve multi-modal alignment. Our results demonstrate how today’s AI models are capable of learning key aspects of children’s early knowledge from realistic input.
2) The generalization gap: Can neural networks capture human-like systematic generalization? We address a 35-year-old debate catalyzed by Fodor and Pylyshyn’s classic article, which argued that standard neural networks are not viable models of the mind because they lack systematic compositionality — the algebraic ability to understand and produce novel combinations from known components. We’ll show how neural network can achieve human-like systematic generalization when trained through meta-learning for compositionality (MLC), a new method for optimizing the compositional skills of neural networks through practice. With MLC, a neural network can match human performance and solve several machine learning benchmarks.
Given this work, we’ll discuss the paths forward for building machines that learn, generalize, and interact in more human-like ways based on more natural input.
Related articles:
Vong, W. K., Wang, W., Orhan, A. E., and Lake, B. M (2024). Grounded language acquisition through the eyes and ears of a single child. Science, 383.
Orhan, A. E., and Lake, B. M. (in press). Learning high-level visual representations from a child’s perspective without strong inductive biases. Nature Machine Intelligence.
Lake, B. M. and Baroni, M. (2023). Human-like systematic generalization through a meta-learning neural network. Nature, 623, 115-121.
Biography
Brenden M. Lake is an Assistant Professor of Psychology and Data Science at New York University. He received his M.S. and B.S. in Symbolic Systems from Stanford University in 2009, and his Ph.D. in Cognitive Science from MIT in 2014. He was a postdoctoral Data Science Fellow at NYU from 2014-2017. Brenden is a recipient of the Robert J. Glushko Prize for Outstanding Doctoral Dissertation in Cognitive Science, he is a MIT Technology Review Innovator Under 35, and his research was selected by Scientific American as one of the 10 most important advances of 2016. Brenden’s research focuses on computational problems that are easier for people than they are for machines, such as learning new concepts, creating new concepts, learning-to-learn, and asking questions.