
Building AI Evaluations You Can Actually Trust - Tejas Kumar
Om avsnittet
Tejas Kumar joins Alexey Grigorev to discuss building evaluation datasets, calibrating LLM judges against human preferences, and testing agents beyond their final answers.
CHAPTERS
00:00 Podcast intro
00:08 Tejas's path to AI engineering
12:24 Why evals go beyond a manual check
17:37 Sourcing evaluation cases and calibrating a judge
28:33 Refreshing datasets and pinning severe cases
33:10 Judge agreement versus agent performance
50:01 Retraining judges and keeping data fresh
55:59 Human preferences, sample sizes and audience questions
59:46 Evaluating final answers versus the agent's trajectory
1:02:46 Faster judging and further evaluation resources
Resources
AI evals: https://tej.as/blog/ai-evals-llm-judge
Code, demos and slides: https://github.com/TejasQ/basically-agent-evals
Tejas: https://tej.as
Watch the original recording: https://www.youtube.com/watch?v=oWVmQDqZ7tw
Join DataTalks.Club: https://datatalks.club/slack.html
Fler avsnitt
Visa alla avsnitt av DataTalks.ClubDataTalks.Club med DataTalks.Club finns tillgänglig på flera plattformar. Informationen på denna sida kommer från offentliga podd-flöden.