- This event has passed.
Aditi Raghunathan (Carnegie Mellon University): “Robust Machine Learning with Foundation Models”
December 6, 2023 @ 12:00 PM - 1:15 PM
Presentation Abstract:
In recent years, foundation models—large pretrained models that can be adapted for a wide range of tasks—have achieved state-of-the-art performance on a variety of tasks. While the pretrained models are trained on broad data, the adaptation (or fine-tuning) process is often performed on limited data. As a result, the challenges of distribution shift, where a model is deployed on a different distribution as the fine-tuning data remain, albeit in a different form.
First, via experiments on pretrained vision and language models, we show different kinds of “catastrophic forgetting’’ where pretrained information is forgotten and correspondences between and in-distribution and out-of-distribution features are weakened. As a result, fine-tuned models are not maximally robust to distribution shifts. We then provide new fine-tuning and prompting methods, backed by theoretical insights, that minimize such distortion and vastly improve accuracy and robustness. On the flip side, our work shows that pretrained knowledge can be hard to get rid of, thereby underlining the potential perils of overreliance on fine-tuning for safety.
Speaker Bio:
Aditi is an Assistant Professor in the Computer Science Department at Carnegie Mellon University. She is also affiliated with the Machine Learning Department.
Aditi works broadly in machine learning and her goal is to make machine learning more reliable and robust. Her work spans both theory and practice, and leverages tools and concepts from statistics, convex optimization, and algorithms to improve the robustness of modern systems based on deep learning.
Aditi’s group research is generously supported by an AI2050 Early Career Fellowship from Schmidt Futures, Apple, Google, and Open Philanthropy.
Until recently, she was a postdoc at Berkeley AI Research. Aditi received her PhD from Stanford University in 2021 where she was fortunate to be advised by Percy Liang. Her thesis won the Arthur Samuel Best Thesis award at Stanford. Previously, she obtained her BTech in Computer Science from IIT Madras in 2016.
Recording of Seminar: