Sveriges mest populära poddar
DataTalks.Club
DataTalks.Club

Building AI Evaluations You Can Actually Trust - Tejas Kumar

1 tim 6 min•23 oktober 2026

Om avsnittet

Tejas Kumar joins Alexey Grigorev to discuss building evaluation datasets, calibrating LLM judges against human preferences, and testing agents beyond their final answers.


CHAPTERS

00:00 Podcast intro

00:08 Tejas's path to AI engineering

12:24 Why evals go beyond a manual check

17:37 Sourcing evaluation cases and calibrating a judge

28:33 Refreshing datasets and pinning severe cases

33:10 Judge agreement versus agent performance

50:01 Retraining judges and keeping data fresh

55:59 Human preferences, sample sizes and audience questions

59:46 Evaluating final answers versus the agent's trajectory

1:02:46 Faster judging and further evaluation resources


Resources

AI evals: https://tej.as/blog/ai-evals-llm-judge

Code, demos and slides: https://github.com/TejasQ/basically-agent-evals

Tejas: https://tej.as


Watch the original recording: https://www.youtube.com/watch?v=oWVmQDqZ7tw

Join DataTalks.Club: https://datatalks.club/slack.html

DataTalks.Club med DataTalks.Club finns tillgänglig på flera plattformar. Informationen på denna sida kommer från offentliga podd-flöden.