Kyunghyun Cho (NYU): “Reality Checks”

Abstract:

Despite its amazing success, leaderboard chasing has become something researchers dread and mock. When implemented properly and executed faithfully, leaderboard chasing can lead to both faster and easily reproducible progress in science, as evident from the amazing progress we have seen with machine learning, or more broadly artificial intelligence, in recent decades. It does not however mean that it is easy to implement and execute leaderboard chasing properly. In this talk, I will go over four case studies demonstrating the issues that ultimately prevent leaderboard chasing from a valid scientific approach. The first case study is on the lack of proper hyperparameter tuning in continual learning, the second on the lack of consensus on evaluation metrics in machine unlearning, the third on the challenges of properly evaluating the evaluation metrics in free-form text generation, and the final one on wishful thinking. By going over these cases, I hope we can collectively acknowledge some of our own fallacies, think of underlying causes behind these fallacies and come up with better ways to approach artificial intelligence research.

Biography:

Kyunghyun Cho is a professor of computer science and data science at New York University and an executive director of frontier research at the Prescient Design team within Genentech Research & Early Development (gRED). He became the Glen de Vries Professor of Health Statistics in 2025. He is also a CIFAR Fellow of Learning in Machines & Brains and an Associate Member of the National Academy of Engineering of Korea. He tries his best to find a balance among machine learning, natural language processing, and life, but almost always fails to do so.

Zoom: https://upenn.zoom.us/j/95189835192, Passcode: 797599