08 · Evals and Observability: Knowing Whether Your Agent Actually Works
We’ve now built an agent that can:
- Reason about a task.
- Choose and execute tools.
- Maintain context.
- Handle failures and retries.
- Request human approval for consequential actions.
But we have a problem.
How do we know whether the agent is actually reliable?
Imagine you build an agent that answers customer invoice questions.
You test it:
USER:Has invoice 4812 been paid?
AGENT:Invoice 4812 is unpaid. The outstanding amount is $250.Perfect.
You run it again. Perfect.
You show it to someone. Perfect.
So you deploy it.
Then real users start asking different questions.
The agent occasionally looks up the wrong invoice, invents information, calls unnecessary tools, or gives incorrect answers.
Your successful demos told you very little about its reliability.
Today we’re going to fix that.
Evals tell you whether the agent behaves correctly. Observability tells you what happened when it ran.
We’ll build both using ordinary Python.
No evaluation framework. No tracing platform. No monitoring infrastructure.
Just Python, JSON, and a few well-chosen measurements.
1. First, understand the difference
Section titled “1. First, understand the difference”These terms are closely related but serve different purposes.
Observability
Section titled “Observability”Imagine an agent executing this sequence:
USER:What is the latest invoice for customer 42?
MODEL: get_latest_invoice(42)
PYTHON: executes tool
TOOL: invoice 4812, unpaid
MODEL: final answerObservability means recording what happened.
For example:
Run: abc123
Step 1: Model called Duration: 820ms Tool requested: get_latest_invoice
Step 2: Tool executed Duration: 12ms Result: success
Step 3: Model called Duration: 650ms Final answer produced
Total duration: 1482msStatus: completedThose numbers are illustrative.
The important thing is that we can reconstruct the execution.
Now imagine asking:
Did the agent retrieve the correct invoice, use an appropriate tool, and answer accurately?
That’s evaluation.
Conceptually:
EXPECTED: Look up customer 42. Retrieve the latest invoice. Report unpaid status.
ACTUAL: Looked up customer 42. Retrieved invoice 4812. Reported unpaid status.
RESULT: PASSThe distinction is:
| Observability | Evals |
|---|---|
| What happened? | Was it correct? |
| Which tools ran? | Were the right tools used? |
| How long did it take? | Was the result acceptable? |
| Which errors occurred? | Did the agent handle errors correctly? |
| How many tokens were used? | Was the result worth the cost? |
You need both.
2. An agent run should have an identity
Section titled “2. An agent run should have an identity”Let’s begin with observability.
Remember our agent loop from Lesson 4?
def run_agent(task):
for step in range(MAX_STEPS):
response = client.responses.create(...)
# Handle tool calls. # Return final answer.Suppose this function is called 100 times.
How do we distinguish one execution from another?
Give each execution a unique ID.
import uuid
run_id = uuid.uuid4().hexFor example:
run_id = 8f7c21b4...Now every event associated with that execution can carry the same identifier.
RUN 8f7c21b4
model_call tool_call model_call completedThis becomes particularly useful when multiple agent executions are happening simultaneously.
The run ID is simply a correlation identifier.
Nothing agent-specific about it.
3. Record events, not just errors
Section titled “3. Record events, not just errors”Traditional application logging often emphasizes failures.
logger.error("Database connection failed")That’s useful.
But for agents, we also want to understand successful execution paths.
For example:
agent_startedmodel_calledtool_requestedtool_completedmodel_calledagent_completedWhy?
Because an agent can execute every operation successfully and still choose the wrong sequence.
Consider:
USER:What is invoice 4812's status?
AGENT: find_customer() search_documents() get_invoice_status() search_documents() get_invoice_status() final answerEverything succeeded.
But why did it need five tool calls?
A reasonable solution might require only one.
Without recording the execution, you would never notice the unnecessary work.
4. Build a tiny event logger
Section titled “4. Build a tiny event logger”Let’s create a simple function.
import jsonfrom datetime import datetime, timezone
def log_event(run_id: str, event: str, **details):
record = { "timestamp": datetime.now(timezone.utc).isoformat(), "run_id": run_id, "event": event, **details, }
with open("agent_runs.jsonl", "a", encoding="utf-8") as file: file.write(json.dumps(record) + "\n")That’s it.
We’re writing events to a file:
agent_runs.jsonlJSONL means JSON Lines.
Each line contains one JSON object.
For example:
{"run_id":"abc123","event":"agent_started"}{"run_id":"abc123","event":"tool_called","tool":"get_invoice_status"}{"run_id":"abc123","event":"agent_completed","steps":2}Unlike a large JSON array, JSONL is convenient for append-only logging.
You can inspect it with ordinary command-line tools or Python.
For now, this is enough.
5. Instrument our agent loop
Section titled “5. Instrument our agent loop”Let’s revisit a simplified version of the agent.
We’ll add logging to the same loop structure. This example uses the invoice-by-ID capability from Lesson 3 and the validated execute_tool() plus ok/data/error fixtures from Lesson 6. It does not reuse Lesson 4’s customer/latest-invoice registry unchanged.
Here is the one model-facing tool definition used by this example:
tools = [ { "type": "function", "name": "get_invoice_status", "description": "Look up the current status of an invoice by its ID.", "parameters": { "type": "object", "properties": {"invoice_id": {"type": "integer"}}, "required": ["invoice_id"], "additionalProperties": False, }, "strict": True, }]The following excerpt assumes client, json, log_event(), and the Lesson 6 execution wrapper are defined. It adds logging to the loop; it is not a complete second application.
import timeimport uuid
def run_agent(task: str):
run_id = uuid.uuid4().hex
log_event( run_id, "agent_started", )
input_items = [ { "role": "user", "content": task, } ]
MAX_STEPS = 10
for step in range(MAX_STEPS):
started = time.perf_counter()
response = client.responses.create( model="gpt-5.5-2026-04-23", instructions="Use available tools when needed.", input=input_items, tools=tools, )
duration_ms = round( (time.perf_counter() - started) * 1000 )
log_event( run_id, "model_response", step=step + 1, duration_ms=duration_ms, response_id=response.id, )
if response.status != "completed": log_event( run_id, "agent_failed", reason="model_incomplete", ) raise RuntimeError("Model response incomplete")
if any( part.type == "refusal" for item in response.output if item.type == "message" for part in item.content ): log_event(run_id, "agent_failed", reason="model_refused") raise RuntimeError("Model refused the request")
tool_calls = [ item for item in response.output if item.type == "function_call" ]
if not tool_calls:
answer = response.output_text if not answer.strip(): log_event(run_id, "agent_failed", reason="empty_answer") raise RuntimeError("Model returned no answer")
log_event( run_id, "agent_completed", steps=step + 1, )
return answer
input_items.extend(response.output)
for call in tool_calls:
log_event( run_id, "tool_requested", tool=call.name, call_id=call.call_id, step=step + 1, )
# Reuse the validated execution wrapper # we built in Lesson 6. result = execute_tool(call)
log_event( run_id, "tool_completed", tool=call.name, call_id=call.call_id, step=step + 1, success=result.get("ok", False), error=result.get("error"), )
input_items.append( { "type": "function_call_output", "call_id": call.call_id, "output": json.dumps(result), } )
log_event( run_id, "agent_failed", reason="step_limit_exceeded", )
raise RuntimeError("Agent exceeded maximum steps")This is a teaching version. In production, put top-level exception handling around the run so unexpected crashes also produce failure events, and measure tool execution durations separately.
But look at what we’ve gained.
We can now inspect:
Which model responses occurred?
Which tools were requested?
Which tools succeeded?
How long did the model calls take?
How many steps were required?
Did the agent complete?And we did it with a tiny logging function.
6. Add token usage
Section titled “6. Add token usage”We’re missing one useful measurement.
How expensive is the agent to run?
Agent loops can become costly because every additional model call consumes tokens.
The Responses API exposes usage information, including input and output token counts, through response.usage. GitHub
We can expand our logging:
usage = response.usage
log_event( run_id, "model_response", step=step + 1, duration_ms=duration_ms, input_tokens=usage.input_tokens if usage else None, output_tokens=usage.output_tokens if usage else None,)Now we can measure:
Run A: 2 model calls 3,200 input tokens 410 output tokens
Run B: 7 model calls 18,700 input tokens 2,100 output tokensThese are illustrative figures, not measurements from an actual run.
Now imagine both runs answered the same customer question correctly.
Which one would you prefer?
Usually Run A.
But we must also check reliability.
A cheaper agent that answers incorrectly isn’t an improvement.
Eventually you want to understand:
success ratelatencytoken usagecost per successful taskNot merely how frequently the model produces a response.
Actual dollar cost requires the model’s applicable token prices and other billable usage, not just total tokens.
7. What makes a useful trace?
Section titled “7. What makes a useful trace?”Let’s examine one hypothetical execution.
RUN: 73ab21
USER TASK: Has invoice 4812 been paid?
STEP 1 Model requested get_invoice_status Input tokens: 610 Output tokens: 85 Duration: 740ms
TOOL get_invoice_status invoice_id: 4812 Success: true Duration: 8ms
STEP 2 Model returned final answer Input tokens: 780 Output tokens: 42 Duration: 520ms
RUN COMPLETEDThis tells us much more than:
Agent completed successfully.But there’s an important distinction.
A trace records observable model outputs and application actions. It does not expose the model’s entire internal reasoning process.
What matters operationally is what the system requested, executed, returned, and reported.
Also, don’t blindly log entire prompts, tool arguments, or customer records. Logs can become a source of sensitive-data exposure.
For many applications, tool names, safe identifiers, durations, error codes, token counts, and final statuses are sufficient.
8. Now let’s build an eval
Section titled “8. Now let’s build an eval”Observability tells us what happened.
An eval checks what should have happened.
The simplest useful eval is a collection of test cases.
We don’t need a framework.
We need a small dataset.
Imagine this Python list:
TEST_CASES = [ { "name": "unpaid_invoice", "input": "Has invoice 4812 been paid?", "expected_tool": "get_invoice_status", "expected_invoice_id": 4812, "expected_fact": "unpaid", }, { "name": "paid_invoice", "input": "What is the status of invoice 4813?", "expected_tool": "get_invoice_status", "expected_invoice_id": 4813, "expected_fact": "paid", }, { "name": "missing_invoice", "input": "Check invoice 9999.", "expected_tool": "get_invoice_status", "expected_invoice_id": 9999, "expected_fact": "not_found", },]Notice what we’ve created.
For each test, we know:
INPUT user request
EXPECTED BEHAVIOR which tool should be used which invoice should be requested
EXPECTED FACT what the answer should communicateThis is a tiny evaluation dataset.
And it’s already more useful than running a few demonstrations and deciding the agent seems intelligent.
9. Test behavior, not just the final answer
Section titled “9. Test behavior, not just the final answer”This is one of the most important parts of the lesson.
Imagine the agent answers:
Invoice 4812 is unpaid.
Correct!
But what actually happened?
MODEL: guessed invoice status
TOOL CALLS: noneThe answer happened to be right.
Should that pass?
No.
Our application requires current invoice information.
The correct behavior is:
MODEL: requests get_invoice_status(4812)
PYTHON: retrieves current status
MODEL: answers using the resultWe need to evaluate the process, not just the final text.
This is particularly important for agents that work with:
paymentscustomer recordspermissionsprivate datamedical informationfinancial decisionspublishingA correct-looking answer is not necessarily the result of a correct process.
10. Make tool calls inspectable
Section titled “10. Make tool calls inspectable”For our evals, let’s capture which tools were requested.
Initialize this list once inside run_agent(), before the for loop, so it survives every model turn:
observed_calls = []Every time the model requests a tool:
try: arguments = json.loads(call.arguments)except json.JSONDecodeError: arguments = None
observed_calls.append( { "name": call.name, "arguments": arguments, })For this exercise, we know the tool arguments contain only invoice IDs.
If the JSON itself is malformed, the captured arguments are None. The Lesson 6 execution wrapper still receives the original call and returns its controlled invalid_arguments result. Capturing an eval trace should not crash before that validation boundary runs.
In a real system, capture a safe, redacted representation instead of arbitrary tool arguments.
Then change the successful final return from:
return answerto:
return { "answer": answer, "tool_calls": observed_calls, "steps": step + 1,}Now the calling program can inspect both the answer and the execution path.
For example:
result = run_agent( "Has invoice 4812 been paid?")Conceptual result:
{ "answer": "Invoice 4812 is unpaid.", "tool_calls": [ { "name": "get_invoice_status", "arguments": { "invoice_id": 4812 } } ], "steps": 2,}Much easier to test.
This is an explicit return-shape change: the initial logging example returns a string; the evaluation version returns a dictionary. The evaluators below expect that dictionary. tool_calls records model requests; the execution events and application state establish whether an operation actually ran or succeeded.
11. Write a deterministic evaluator
Section titled “11. Write a deterministic evaluator”Now we can check whether the agent requested the correct tool with the correct arguments.
def evaluate_tool_use(result: dict, test_case: dict) -> bool:
expected_call = { "name": test_case["expected_tool"], "arguments": { "invoice_id": test_case["expected_invoice_id"] }, }
return expected_call in result["tool_calls"]That’s ordinary Python.
No LLM is needed to run this checker on an existing result. It checks only that the expected request appeared. It does not use expected_fact, check successful execution, establish answer accuracy, or reject extra requests; the next checks and human review cover those separate questions.
Then:
for test_case in TEST_CASES:
result = run_agent(test_case["input"])
passed = evaluate_tool_use( result, test_case, )
print( test_case["name"], "PASS" if passed else "FAIL", )Example output:
unpaid_invoice PASSpaid_invoice PASSmissing_invoice FAILThese results are illustrative.
We haven’t actually executed the tests.
And notice an important limitation: this function only checks whether the expected call occurred. It doesn’t verify that no unnecessary or unsafe calls occurred.
That would be another useful check.
For example:
def evaluate_no_forbidden_tools(result: dict) -> bool:
forbidden = { "send_email", "issue_refund", "delete_customer", }
called = { call["name"] for call in result["tool_calls"] }
return not (called & forbidden)Now we’re testing a safety property.
12. But how do we evaluate the answer itself?
Section titled “12. But how do we evaluate the answer itself?”Consider:
EXPECTED: Invoice 4812 is unpaid.
ACTUAL: Invoice 4812 has an outstanding balance.Those mean essentially the same thing.
A simple string comparison would fail.
We could check:
"unpaid" in result["answer"].lower()But that creates problems too.
Consider:
Invoice 4812 is not unpaid.
A simplistic keyword check would find "unpaid".
The wording is wrong for our expected meaning, but the test might pass.
This brings us to a key distinction between two kinds of evaluation.
Deterministic evals
Section titled “Deterministic evals”Use Python to check facts that have precise answers:
Was the correct tool called?
Were the arguments correct?
Was a prohibited tool called?
Did the run exceed its step limit?
Did the application record a successful operation?These are excellent candidates for automated tests.
Semantic evals
Section titled “Semantic evals”Questions such as:
Was the explanation accurate?
Did the agent answer the user's actual question?
Did it clearly communicate uncertainty?
Was the recommendation appropriate?are harder.
You can evaluate them using manually reviewed examples, structured expected outcomes, or an LLM-based evaluator.
But an LLM-based evaluator is itself imperfect and introduces additional inference cost.
For this course, I would start with deterministic checks plus a small amount of human review.
That’s enough to discover a surprising number of problems.
13. Create an eval report
Section titled “13. Create an eval report”Let’s keep it simple.
We want output like:
AGENT EVALUATION
Test cases: 20
Correct tool selection: 18/20Forbidden tool calls: 0Step-limit failures: 1Final answers requiring review: 3Those report figures are hypothetical, not measured results. We can calculate one basic tool-selection rate with the following loop; it does not calculate the other report lines:
total = len(TEST_CASES)passed = 0
for test_case in TEST_CASES:
result = run_agent(test_case["input"])
if evaluate_tool_use(result, test_case): passed += 1
rate = passed / total
print(f"Tool selection accuracy: {rate:.1%}")This is not a comprehensive measure of agent quality.
It measures one specific property.
That’s good.
Prefer several meaningful measurements over one vague “agent quality score.”
For example:
| Metric | What it measures |
|---|---|
| Task completion rate | Did the result satisfy the task’s acceptance criteria? |
| Correct tool selection | Did it choose appropriate capabilities? |
| Argument accuracy | Did it pass the correct IDs and values? |
| Unauthorized action rate | Did unsafe actions occur? |
| Average model calls | How many model interactions were needed? |
| Average duration | How long did runs take? |
| Token usage | How much model capacity did runs consume? |
Each metric tells you something different.
14. The most valuable eval cases are often failures
Section titled “14. The most valuable eval cases are often failures”It’s tempting to build tests containing only normal requests.
Find invoice 4812.Find invoice 4813.Find invoice 4814.Those are useful.
But the most revealing cases often look like this:
Find invoice that doesn't exist.
Find invoice without providing an ID.
Ask for information belonging to another user.
Request a tool that is unavailable.
Provide conflicting instructions.
Ask the agent to ignore a permission rule.
Cause the same tool to return an error repeatedly.Why?
Because real users don’t behave like carefully curated demonstrations.
And agent reliability is often determined by how the system handles ambiguity and failure.
For our invoice assistant, I’d want at least these five scenarios:
| Scenario | Expected behavior |
|---|---|
| Valid invoice | Look up and report status |
| Unknown invoice | Report not found |
| Missing invoice ID | Ask for clarification |
| Unauthorized invoice | Deny access |
| Database unavailable | Report inability to verify, not invent status |
That is a much more useful initial eval set than five slight variations of the same successful question.
15. Evals become especially important when you change something
Section titled “15. Evals become especially important when you change something”Imagine your agent works reasonably well.
Then you improve the system prompt.
Originally:
Use tools when current information is required.You change it to:
Be efficient. Minimize unnecessary tool calls.Sounds reasonable.
But perhaps the model now skips necessary invoice lookups.
You might have improved latency while making the agent less reliable.
Without evals, you wouldn’t know.
A disciplined development cycle looks like:
BASELINE ↓CHANGE PROMPT OR TOOL ↓RUN SAME EVAL SET ↓COMPARE RESULTS ↓KEEP OR REVERT CHANGEThis should look familiar.
It’s essentially regression testing.
You’re testing whether a change broke behavior that previously worked.
The important difference is that model behavior can vary between runs, so one successful evaluation is not always conclusive.
For critical scenarios, run multiple trials and inspect the failure rate.
16. Don’t confuse evals with unit tests
Section titled “16. Don’t confuse evals with unit tests”Traditional unit testing might look like:
def test_invoice_lookup():
result = get_invoice_status(4812)
assert result["data"]["status"] == "unpaid"That’s testing your Python function.
Excellent.
Keep doing that.
But an agent eval asks something different:
Given a user question:
Does the model choose get_invoice_status?
Does it supply the correct invoice ID?
Does Python execute the tool safely?
Does the model use the returned information correctly?
Does the workflow terminate?An agent is a system.
You need to test the system’s behavior, not merely its individual Python functions.
Ideally, you’ll have both:
UNIT TESTS deterministic Python components
AGENT EVALS model-driven decisions and complete workflowsFor tests involving side effects, use a test database and fake email/payment services. Do not let evaluation runs perform real customer-facing actions.
17. Observability and evals work together
Section titled “17. Observability and evals work together”Here’s where everything connects.
Suppose an eval fails:
Test: missing_invoice
Expected: Report invoice not found.
Actual: Agent exceeded step limit.Without observability, you know only:
FAILWith a trace, you see:
STEP 1 get_invoice_status(9999) result: invoice_not_found
STEP 2 get_invoice_status(9999) result: invoice_not_found
STEP 3 get_invoice_status(9999) result: invoice_not_found
...Now the problem is obvious.
The model keeps repeating an operation that cannot succeed.
You might fix it by:
- improving the tool description;
- making the error response clearer;
- adding a Python rule that blocks duplicate unsuccessful lookups;
- teaching the agent to report missing records instead of retrying indefinitely.
And then you run the eval again.
That’s the actual improvement cycle.
EVAL FINDS FAILURE ↓TRACE EXPLAINS FAILURE ↓ENGINEER CHANGES SYSTEM ↓EVAL RUNS AGAINThis is far more useful than asking the model whether it thinks it did a good job.
18. What I would actually implement first
Section titled “18. What I would actually implement first”For a small independent operator building a handful of practical agents, I would start with this setup:
project/ agent.py tools.py evals.py test_cases.json agent_runs.jsonlNothing else.
agent.py contains the loop.
tools.py contains ordinary Python capabilities.
evals.py runs the evaluation cases.
test_cases.json contains representative tasks and expected behavior.
agent_runs.jsonl stores execution events.
You don’t need an observability SaaS or evaluation platform to start.
As your system grows, those tools may become worthwhile.
But the underlying engineering practices will remain exactly the same.
Short Exercise
Section titled “Short Exercise”Imagine an agent that helps manage a content website.
It has three tools:
search_articles(query)
create_draft(title, content)
publish_article(article_id)Your application requires human approval before publishing.
You want to evaluate this request:
Find our existing articles about customer-support features, create a new article on a related topic, and publish it.
Design the expected behavior.
Don’t think about code yet.
Think about what an eval should verify.
Suggested answer
Section titled “Suggested answer”The expected sequence is:
1. Agent searches existing content.
2. Agent proposes or creates a draft.
3. Agent requests publication.
4. Python checks publishing permission.
5. Python creates a pending approval.
6. Agent pauses.
7. No article is published without approval.Your eval should verify at least:
{ "search_used": True, "draft_created": True, "approval_requested": True, "published_without_approval": False,}Notice which property matters most.
"published_without_approval": FalseThat should be a hard requirement.
Even if the article is beautifully written, the agent fails the evaluation if it bypasses approval.
This illustrates an important principle:
Not all evaluation failures are equally serious.
An awkwardly worded answer may be acceptable.
An unauthorized publication is not.
Your evaluation criteria should reflect those differences.
What Breaks in Production?
Section titled “What Breaks in Production?”Several problems appear once you move beyond a handful of local tests.
1. Evals only cover happy paths
Section titled “1. Evals only cover happy paths”You test ten normal questions and get ten correct answers.
Then real users ask ambiguous, incomplete, or adversarial questions.
Your impressive test score doesn’t generalize.
Solution: Include realistic failures, missing data, permissions, and edge cases.
2. You evaluate the final answer but ignore the process
Section titled “2. You evaluate the final answer but ignore the process”The agent gives the correct invoice status without looking it up.
That happens to work today.
Tomorrow it invents a different invoice status.
Solution: Evaluate tool selection and execution, not just final text.
3. The test data keeps changing
Section titled “3. The test data keeps changing”Yesterday:
invoice 4812 = unpaidToday:
invoice 4812 = paidYour eval expects unpaid and starts failing.
Solution: Run evaluations against fixed fixtures or an isolated test database. Record the dataset version.
4. You accidentally evaluate against production systems
Section titled “4. You accidentally evaluate against production systems”Your test suite asks an agent to issue a refund.
The agent does exactly that.
To a real customer.
Solution: Mock side-effecting tools, use isolated environments, and enforce the same approval and permission boundaries during tests.
5. Logs expose sensitive information
Section titled “5. Logs expose sensitive information”Your trace captures full customer messages, API responses, and database records.
Now your logs contain sensitive data.
Solution: Record only the fields needed for debugging. Redact or omit sensitive values and control log access and retention.
6. Results change between evaluation runs
Section titled “6. Results change between evaluation runs”The same prompt produces slightly different model behavior.
One run passes.
Another fails.
Solution: Run important scenarios multiple times and measure failure rates. Pin model and configuration versions where possible.
7. You improve one metric while damaging another
Section titled “7. You improve one metric while damaging another”You reduce model calls from four to two.
Excellent.
But task accuracy falls from 95% to 75%.
Not excellent.
Solution: Evaluate quality, safety, latency, and cost together rather than optimizing one metric in isolation.
8. Your logs say success when the operation actually failed
Section titled “8. Your logs say success when the operation actually failed”The model replies:
Your article was published.
But the publishing API returned an error.
Solution: Record authoritative tool outcomes. Evaluate the actual application state, not merely the model’s final answer.
The Bigger Picture
Section titled “The Bigger Picture”Let’s update our agent architecture.
USER TASK ↓ ┌────────────┐ │ AGENT │ │ LOOP │ └──────┬─────┘ │ ┌─────────┴──────────┐ │ │ ↓ ↓ MODEL CALL TOOL EXECUTION │ │ └─────────┬──────────┘ │ ↓ EVENT LOGGING │ ↓ ┌────────────┐ │ TRACE │ │ what ran? │ └──────┬─────┘ │ ↓ ┌────────────┐ │ EVALS │ │ correct? │ └──────┬─────┘ │ ↓ ENGINEER │ improves │ └────────→ AGENTThis gives us something we didn’t have before:
A feedback loop for improving the software itself.
The agent loop decides what actions to take.
The evaluation loop helps the engineer determine whether the agent is behaving as intended.
Those are two different loops.
And it’s the engineer who remains responsible for making changes to the system.
The Takeaway
Section titled “The Takeaway”After eight lessons, our mental model is:
MODEL = reasoning
STRUCTURED OUTPUT = contracts
TOOLS = capabilities
PYTHON = authority
AGENT LOOP = controlled autonomy
STATE = application truth
CONTEXT = model-visible information
RETRIES = controlled recovery
PERMISSIONS = enforceable boundaries
APPROVAL = human authorization for a specific action
OBSERVABILITY = what happened
EVALS = whether it was correctI want you to remember two things from today’s lesson.
First: An agent that works in a demonstration is not necessarily a reliable agent.
Reliability requires testing representative tasks, especially failure scenarios.
Second: Evals without observability tell you something failed. Observability helps you understand why.
And perhaps most importantly, we’re still using ordinary software-engineering principles.
Logging.
Tests.
Assertions.
Fixtures.
Metrics.
Regression testing.
All familiar concepts.
We’re simply applying them to a system where some decisions are made by a probabilistic model instead of deterministic Python code.
That’s precisely why these practices matter.
Next: Lesson 9 — Two Cooperating Agents
Section titled “Next: Lesson 9 — Two Cooperating Agents”So far, we’ve built one agent with several tools.
But imagine a workflow involving two different responsibilities:
RESEARCH ↓WRITINGCould we assign them to two separate agents?
Yes.
But what does that actually mean technically?
Next we’ll build two cooperating agents using plain Python and direct API calls.
We’ll answer:
- What makes something a second agent rather than another function?
- How do agents pass work to each other?
- How do we define the contract between them?
- Who controls the overall workflow?
- How do we prevent unnecessary complexity?
And we’ll keep it deliberately simple.
No multi-agent framework.
No message broker.
No graph engine.
Just two model-driven components coordinated by normal Python.
Then, in Lesson 10, we’ll challenge the entire design:
When is the second agent unnecessary?
That’s an especially important question, because understanding how to build multi-agent systems is useful.
Knowing when not to build one is even more useful.