On this week's episode of the podcast, freeCodeCamp founder Quincy Larson interviews Yifan Mai, a Senior Software Engineer on Google's TensorFlow team who left the private sector to go do AI research at Stanford. He's the lead maintainer of the open source HELM project, where he benchmarks the performance of Large Language Models.
We talk about:
- Open Source VS Open Weights in LLMs
- The Ragged Frontier of LLM use cases
- AI impact on jobs and our predictions
- What to learn so you can stay above the waterline
Can you guess what song I'm playing in the intro?
Also, I want to thank the 10,993 kind people who support our charity each month, and who make this podcast possible. You can join them and support our mission at: https://www.freecodecamp.org/donate
Links we talk about during our conversation:
- Yifan's personal webpage: yifanmai.com
- HELM Leaderboards: https://crfm.stanford.edu/helm/
- HELM GitHub Repository: https://github.com/stanford-crfm/helm
- Stanford HAI Blog: https://crfm.stanford.edu/helm/
Chapters
0:00:00 Introduction to Yifan Mai and His Journey
0:03:53 Transitioning from Industry to Academia
0:07:03 The Role of a Research Engineer
0:10:18 Bridging the Gap: Engineering Practices in Research
0:13:15 The Importance of Software Quality in Research
0:16:02 Understanding Research Incentives
0:19:06 Overview of the HELM Project
0:22:11 Benchmarking Language Models
0:25:00 Open Weights vs. Closed Weights in AI Models
0:27:57 The Impact of Open Models on Research
0:31:04 Ethics and Legalities in AI Training Data
0:34:02 Evaluating Model Performance and Win Rates
0:40:31 The Scalability of AI and Human Alignment
0:42:55 Synthetic Data and Model Biases
0:44:40 The Role of Human Feedback in AI Training
0:47:46 Quality vs. Quantity in Training Data
0:50:25 The Limits of Current Data Sources
0:53:58 Challenges in Evaluating AI Performance
0:57:57 The Ragged Frontier of AI Capabilities
1:02:51 Programming as a Tool for Logical Thinking
1:08:07 Impressive AI Capabilities and Future Predictions
1:12:53 Skepticism About Future AI Developments
1:16:40 The Invisible Role of AI in Society
1:21:47 AI in Various Industries: Opportunities and Challenges
1:24:40 Optimism vs. Fear: The Dual Nature of AI
1:32:16 Automation: Augmentation vs. Displacement
1:40:51 The Power Dynamics of AI Implementation
1:47:09 The Importance of Fundamentals in AI Education
1:54:17 Full Cover Song