Akshay Krishnamurthy (MSR): The coverage principle in language models: From pre-training to test-time scaling
Abstract:
Test-time compute has emerged as a new axis for scaling language model capabilities, yet we lack a principled understanding of this paradigm. What are the right algorithms and trade-offs for test-time scaling? What properties of the pre-trained model enable it? And can we better align pre-training recipes for test-time success? This talk addresses these questions through a unified lens of coverage. We first show that test-time scaling strategies like best-of-N sampling succeed if and only if the pre-trained model has coverage over high-quality responses. We then demonstrate that coverage, and hence best-of-N performance, can be improved through deliberate exploration, either purely at test time or via RL-style post-training. Finally, we ask why pre-training via next-token prediction yields models with good coverage in the first place. We uncover a rich theoretical landscape driven by an implicit bias of the next-token prediction objective, while also identifying a fundamental misalignment between next-token prediction and coverage, raising the possibility of future algorithmic innovations.
Biography:
I am a senior principal research manager at Microsoft Research, New York City. Previously, I spent two years as an assistant professor in the College of Information and Computer Sciences at the University of Massachusetts, Amherst and a year as a postdoctoral researcher at Microsoft Research, NYC. Before that, I completed my PhD in the Computer Science Department at Carnegie Mellon University, advised by Aarti Singh. I received my undergraduate degree in EECS at UC Berkeley.
My research interests are in machine learning and statistics. I am most excited about interactive learning, or learning settings that involve feedback-driven data collection. My recent interests revolve around decision making with limited feedback, including contextual bandits and reinforcement learning, and how these frameworks manifest in language modeling and generative AI.
Zoom: https://upenn.zoom.us/j/91064109577
