Course Overview
This course teaches how to test AI agents effectively using Python, focusing on catching failures that traditional unit tests miss. It covers building a complete AI agent testing and LLM evaluation harness with pytest and DeepEval, working on a realistic customer-support refund agent project. The course emphasizes practical skills such as testing tool calls, multi-agent systems, security testing, observability, and CI/CD quality gates without requiring an API key or cloud account.
Key Takeaways
- Learn to catch AI agent failures by testing actions, tool calls, arguments, and customer interactions.
- Write LLM evaluations with pytest and DeepEval including test cases, golden datasets, metrics, thresholds, and reports.
- Build and calibrate an LLM-as-a-judge and custom metrics like G-Eval against human-labeled data.
- Evaluate Retrieval-Augmented Generation (RAG) pipelines by testing retrieval and generation separately.
- Test tool calling and Model Context Protocol (MCP) contracts on a real MCP server, including multi-agent handoffs and loops.
- Perform red-team security testing for prompt injection, PII leaks, and unauthorized actions using promptfoo.
- Trace agent execution step-by-step with OpenTelemetry and export traces to Langfuse.
- Implement regression testing, CI/CD quality gates with GitHub Actions, and a one-command release capstone.
Prerequisites
- Python 3.11 or newer installed and comfort reading Python code.
- A terminal and code editor on Windows, macOS, or Linux.
- No machine-learning or data-science background required.
- Optional: OpenAI key, Node.js, Langfuse account, and GitHub account for some exercises.
Target Learners
- QA engineers, SDETs, and test automation engineers testing LLM applications and AI agents.
- Python developers and AI engineers shipping AI agents who need evidence before release.
- Tech leads and engineering managers responsible for quality gates and release decisions for AI features.
0.1 Reproduce the Wrong Refund on Your Machine (Setup, No API Key) 6:12
1.1 Tokens, Context, and Temperature: What Testers Must Know 3:02
1.2 Chatbot versus Agent: Test the Action, Not Just the Reply 2:43
1.3 The Agent Loop, Tools, Memory, and Planning 3:12
1.4 Six Ways AI Agents Fail (Hallucination, Wrong Tool, Loops) 3:16
2.1 Deterministic versus Non-Deterministic Testing 3:02
2.2 The Five Dimensions of Agent Quality 2:26
2.3 Designing an AI Agent Test Strategy 3:09
2.4 Measuring Non-Determinism: One Ticket, Five Model Runs 4:06
3.1 Meet DeepEval: pytest for AI 3:02
3.2 Test Cases and Golden Datasets 1:40
3.3 Running the First Evaluation 2:07
3.4 Project 1: Customer Support Agent Evaluation 2:07
4.1 Quality Metrics: Relevancy, Faithfulness, Task Match 3:02
4.2 Agent-Specific Metrics: Tool Correctness and Task Completion 3:10
4.3 LLM as Judge: Calibrate a Judge Against Human Labels 2:36
4.4 Custom Metrics and G-Eval 2:49
4.5 A Second Agent, No Map: Evaluate an Unknown Agent 2:44
5.1 The RAG Quality Problem 2:05
5.2 RAG Metrics: Context Precision, Recall, and Faithfulness 2:17
5.3 Isolating Retrieval and Generation Failures 2:39
5.4 Project 2: Enterprise RAG Evaluation 2:25
6.1 Why Tool Calling Is High Risk 1:39
6.2 Tool Selection, Arguments, and Returns 2:07
6.3 MCP Contract Testing with a Real MCP Server 2:28
6.4 Project 3: Multi-Tool Test Suite 2:53
7.1 Multi-Agent Systems: What Can Go Wrong 2:10
7.2 Agent Communication and Delegation: Testing Handoffs 2:14
7.3 Loops, Shared State, and Failure Propagation 2:33
8.1 The Threat Model for AI Agents 1:45
8.2 Prompt Injection Resistance and promptfoo Red Teaming 2:24
8.3 PII, Data Exfiltration, and Authorization 2:02
8.4 Security Review Project: Red-Team Findings 2:51
9.1 Why Traces Matter: Logs versus Traces 1:56
9.2 Finding the Failing Step with a Span Trace 2:53
9.3 Standardized Telemetry: OpenTelemetry Names and Redaction 2:10
9.4 The Same Trace in Langfuse (OpenTelemetry Export) 3:41
9.5 Tracing the Second Agent: Nine Failures, Four Causes 2:20
Access Files No pi required.
Free Course files 1 file package
- Full Pack
Discussions are closed
Comments are currently disabled for this course.
