06 · Retries and Failure Handling: Making the Agent Dependable
In Lesson 4, we built the agent loop. In Lesson 5, we separated application state from model context.
Our architecture now looks like this:
USER REQUEST ↓PYTHON ↓MODEL ↓TOOL REQUEST ↓PYTHON EXECUTES TOOL ↓TOOL RESULT ↓MODEL ↓FINAL ANSWERSo far, we’ve assumed everything works.
The model responds. The arguments are valid. The database is available. The tool succeeds.
In production, any of those operations can fail.
Today we’re going to handle those failures without introducing a retry framework, workflow engine, or complicated recovery architecture.
The principle is simple:
When an agent encounters a failure, Python must decide whether to retry, report the failure, or stop.
Do not delegate that responsibility entirely to the model.
1. There are three different places where things break
Section titled “1. There are three different places where things break”Start by separating failures into three categories.
A. Model/API failures
Section titled “A. Model/API failures”Your Python application calls the model:
response = client.responses.create(...)But something goes wrong:
network timeoutrate limitserver unavailableinvalid API keyinvalid requestThe problem occurred while communicating with the model provider.
B. Tool failures
Section titled “B. Tool failures”The model successfully requests:
get_invoice_status(4812)But the Python function encounters:
database unavailableinvoice not foundexternal API timeoutinvalid argumentspermission deniedThe model API worked.
The tool did not.
C. Agent-level failures
Section titled “C. Agent-level failures”Every individual operation might work, but the overall agent still fails.
For example:
MODEL: search for customer
TOOL: customer not found
MODEL: search for customer again
TOOL: customer not found
MODEL: search for customer again...Or the agent performs ten successful tool calls without answering the user’s question.
These are three different engineering problems.
Don’t try to solve all of them with one generic retry function.
2. First: handling model API failures
Section titled “2. First: handling model API failures”Let’s revisit our ordinary model call.
from openai import OpenAI
client = OpenAI()
response = client.responses.create( model="gpt-5.5-2026-04-23", input="Explain database transactions.",)
print(response.output_text)What happens if the API doesn’t respond?
Without exception handling, your Python program raises an exception.
For a learning script, that’s acceptable.
For an application serving real users, we need something better.
Fortunately, the official Python SDK already does much of the work.
The SDK already retries certain failures
Section titled “The SDK already retries certain failures”The current OpenAI Python SDK automatically retries certain transient failures twice by default.
These include connection errors and HTTP statuses 408, 409, 429, and 500 or higher. It also provides configurable timeouts and retry limits. GitHub
So you do not need to immediately write your own exponential-backoff system.
Start here:
from openai import OpenAI
client = OpenAI( timeout=30.0, max_retries=2,)Let’s understand those settings.
timeout=30.0Configures the SDK’s network timeouts with a scalar value of 30 seconds. The underlying transport has connect, read, write, and connection-pool timeouts. This is not a guaranteed 30-second wall-clock deadline for a whole request attempt.
max_retries=2Allows up to two automatic retries after the initial attempt.
Conceptually:
REQUEST 1 ↓temporary failure ↓REQUEST 2 ↓temporary failure ↓REQUEST 3 ↓still failing ↓raise exceptionThe SDK handles the backoff between attempts.
One important detail: this timeout setting is not a deadline for the entire multi-attempt operation or agent run. An attempt may involve multiple network waits; retries and backoff add more time. A real overall time limit needs its own application policy. Lesson 4’s step limit bounds model turns, not elapsed time.
3. Catch the failure at the application boundary
Section titled “3. Catch the failure at the application boundary”Let’s wrap the model call in a small function.
from openai import ( OpenAI, APIConnectionError, APIStatusError, RateLimitError,)
client = OpenAI( timeout=30.0, max_retries=2,)
def ask_model(question: str) -> str:
try: response = client.responses.create( model="gpt-5.5-2026-04-23", input=question, )
if response.status != "completed": return "The model could not complete the request."
if any( part.type == "refusal" for item in response.output if item.type == "message" for part in item.content ): return "The model could not answer this request."
if not response.output_text.strip(): return "The model returned no answer."
return response.output_text
except RateLimitError: return "The AI service is busy. Please try again later."
except APIConnectionError: return "Unable to connect to the AI service."
except APIStatusError as error: print(f"API error: {error.status_code}") return "The AI service encountered an error."Notice the exception order.
RateLimitError is a more specific kind of API status error.
APIConnectionError covers connection-related failures, including API timeouts.
APIStatusError handles other HTTP errors returned by the service.
This example returns simple messages to keep the flow visible. In a real application, you would usually log the technical details and return a structured failure to the calling code.
The important thing is what we’re not doing:
while True: try: call_model() except Exception: continueThat would be terrible.
It could keep calling the API indefinitely.
We’re using bounded retries and explicit failure handling.
4. Not every failure should be retried
Section titled “4. Not every failure should be retried”This is one of today’s most important ideas.
Imagine these two failures.
Failure A:
HTTP 503Service temporarily unavailableRetrying might work.
Failure B:
HTTP 401Invalid API keyRetrying the same request with the same credentials won’t fix anything.
The problem is configuration, not availability.
Here’s a useful mental model:
| Failure | Usually retry? | Why |
|---|---|---|
| Network timeout | Yes, bounded | Could be temporary |
| HTTP 503 | Yes, bounded | Service may recover |
| HTTP 429 rate limit | Sometimes, with backoff | Capacity may become available |
| HTTP 400 | No | Request needs correction |
| HTTP 401 | No | Credentials need correction |
| Permission denied | No | Authorization must change |
| Record not found | No | Repeating the lookup probably changes nothing |
The distinction is:
Transient failure: something temporarily prevented an otherwise valid operation.
Permanent failure: the operation cannot succeed without changing something.
There is one additional complication: a timeout doesn’t necessarily mean the server did nothing. The request may have completed while its response was lost.
We’ll return to that shortly.
5. Tool failures are different
Section titled “5. Tool failures are different”Now let’s return to our invoice agent.
We have a tool:
def get_invoice_status(invoice_id: int) -> dict:
invoices = { 4812: { "status": "unpaid", "amount": 250, }, 4813: { "status": "paid", "amount": 175, }, }
return invoices.get( invoice_id, {"error": "invoice_not_found"}, )The model requests:
get_invoice_status(9999)Python executes the function.
The result is:
{ "error": "invoice_not_found"}Is that a system failure?
No.
The application worked perfectly.
The invoice simply doesn’t exist in our data.
This is what we call an expected domain failure.
Compare it with:
def get_invoice_status(invoice_id: int) -> dict:
database = connect_to_database()
return database.get_invoice(invoice_id)What if:
connect_to_database()raises a connection error?
Now we have an infrastructure failure.
Different problem.
Different response.
6. Give tools a consistent result format
Section titled “6. Give tools a consistent result format”Here’s a small improvement I recommend.
Instead of returning arbitrary structures, give your tools a simple convention.
Successful result:
{ "ok": True, "data": { "status": "unpaid", "amount": 250, }}Unsuccessful result:
{ "ok": False, "error": "invoice_not_found",}This is not a framework.
It’s simply an interface convention.
Our function becomes the following. The invoice fixtures have not changed: 4812 is unpaid, 4813 is paid, and 9999 is absent. Only the result envelope changes to ok plus data or error.
def get_invoice_status(invoice_id: int) -> dict:
invoices = { 4812: { "status": "unpaid", "amount": 250, }, 4813: { "status": "paid", "amount": 175, }, }
invoice = invoices.get(invoice_id)
if invoice is None: return { "ok": False, "error": "invoice_not_found", }
return { "ok": True, "data": invoice, }Now every caller can check:
result = get_invoice_status(4812)
if result["ok"]: print(result["data"])else: print(result["error"])And our model can interpret the same structure when Python sends it as a tool result.
The function-calling API allows applications to return tool results as strings, including JSON representations of successful operations or errors. OpenAI Developers
7. Catch tool failures in Python
Section titled “7. Catch tool failures in Python”Now imagine the tool occasionally depends on an external API.
We don’t want an unexpected exception to crash the entire agent.
We can build a small execution wrapper.
import json
def execute_tool(tool_call) -> dict:
if tool_call.name != "get_invoice_status": return { "ok": False, "error": "unknown_tool", }
try: arguments = json.loads(tool_call.arguments)
if not isinstance(arguments, dict) or set(arguments) != {"invoice_id"}: return { "ok": False, "error": "invalid_arguments", }
invoice_id = arguments["invoice_id"]
if type(invoice_id) is not int: return { "ok": False, "error": "invalid_arguments", }
return get_invoice_status(invoice_id)
except (json.JSONDecodeError, KeyError, TypeError): return { "ok": False, "error": "invalid_arguments", }
except Exception as error: # Log technical details internally. print(f"Tool failed: {type(error).__name__}")
return { "ok": False, "error": "tool_unavailable", }Walk through the code.
First:
if tool_call.name != "get_invoice_status":We’re enforcing the allowed capability.
Then:
arguments = json.loads(tool_call.arguments)We’re parsing the model’s requested arguments.
Then:
if type(invoice_id) is not int:We’re validating what the function expects.
Finally:
return get_invoice_status(invoice_id)We execute the real tool.
And if something unexpected happens:
except Exception:we return a controlled error.
That broad exception handler is appropriate here only because this is an outer execution boundary for a read-only tool. We should record unexpected failures rather than silently treating programming bugs as ordinary business errors.
Notice another important distinction:
Returning tool_unavailable does not mean the operation should automatically be retried.
It tells the agent something failed.
Your application still owns retry policy.
8. Return the failure to the model
Section titled “8. Return the failure to the model”Remember Lesson 3?
The model requested a function.
Python executed it.
Then we returned the result using:
{ "type": "function_call_output", "call_id": tool_call.call_id, "output": json.dumps(result),}That still works when result represents failure.
For example:
result = { "ok": False, "error": "invoice_not_found",}We send it back.
The model now sees:
USER: Has invoice 9999 been paid?
MODEL: get_invoice_status(9999)
TOOL: ok = false error = invoice_not_foundThe model can produce a useful answer:
I couldn’t find invoice 9999.
The agent didn’t crash.
It handled the business failure.
That’s a perfectly valid outcome.
9. A tool failure doesn’t always mean retry
Section titled “9. A tool failure doesn’t always mean retry”This is where agent engineering starts requiring judgment.
Suppose the model asks:
search_documents("refund policy")The tool returns:
{ "ok": false, "error": "temporary_search_failure"}What should happen?
There are several reasonable possibilities.
OPTION A: Retry search once.
OPTION B: Try a different search capability.
OPTION C: Tell user the search is unavailable.
OPTION D: Stop and escalate.But consider:
{ "ok": false, "error": "permission_denied"}The answer should not be:
Try another tool until we find a way around permission.Authorization failures are boundaries, not puzzles for an agent to solve.
Your Python application should enforce those boundaries regardless of the model’s next request.
We’ll make that explicit in Lesson 7.
10. Add failure awareness to the agent loop
Section titled “10. Add failure awareness to the agent loop”Recall our loop from Lesson 4.
We can now modify its core. This is an excerpt inside an agent function. It assumes input_items, the tool definitions, and a call_model(input_items) helper that returns a Responses API response or raises an API exception. That helper is not ask_model() above: ask_model() returns user-facing strings, which this loop cannot inspect as response objects.
The application must handle API exceptions at its boundary, while the loop checks the returned response before dispatch:
MAX_STEPS = 10
for step in range(MAX_STEPS):
response = call_model(input_items)
if response.status != "completed": raise RuntimeError("Model response did not complete.")
if any( part.type == "refusal" for item in response.output if item.type == "message" for part in item.content ): raise RuntimeError("Model refused the request.")
tool_calls = [ item for item in response.output if item.type == "function_call" ]
if not tool_calls: answer = response.output_text if not answer.strip(): raise RuntimeError("Model returned no answer.") return answer
input_items += response.output
for tool_call in tool_calls:
result = execute_tool(tool_call)
input_items.append( { "type": "function_call_output", "call_id": tool_call.call_id, "output": json.dumps(result), } )
raise RuntimeError("Agent exceeded maximum steps.")What’s changed?
Previously, we assumed:
result = function(**arguments)would always succeed.
Now:
result = execute_tool(tool_call)always gives our loop a controlled result for the failures handled by that wrapper.
It might be:
{"ok": True, "data": ...}or:
{"ok": False, "error": ...}Either way, we can feed that result back to the model.
The response checks preserve Lesson 4’s completion boundary. The separate model-call helper still needs the API-exception policy discussed earlier, without converting exceptions into strings where this loop expects a response.
But the overall design remains simple.
11. The dangerous case: duplicate side effects
Section titled “11. The dangerous case: duplicate side effects”Now we get to what I consider the single most important production issue in this lesson.
Imagine our agent has this tool:
def send_invoice_reminder(invoice_id: int): send_email(...)The model requests:
send_invoice_reminder(4812)Python executes it.
The email gets sent.
But immediately afterward, the connection to the email provider drops before the provider’s success response reaches us.
Our application sees:
TIMEOUTWhat actually happened?
We don’t know.
Possibility A:
Email was never sent.Possibility B:
Email was sent successfully.Response was lost.Now imagine we automatically retry:
send_invoice_reminder(4812)We might send the same email twice.
For emails, that’s annoying.
For this:
issue_refund(order_id=123)it could be expensive.
For this:
charge_credit_card(amount=500)it could be disastrous.
A timeout means you don’t know whether the operation completed.
It does not necessarily mean the operation failed.
12. Idempotency solves a different problem than retries
Section titled “12. Idempotency solves a different problem than retries”Here’s an important engineering term worth knowing.
Idempotency means that repeating an operation does not produce additional unintended effects.
For example:
SET invoice status = paidis naturally easier to make idempotent than:
ADD $250 to account balanceRun the first operation twice:
paidpaidNo additional state change is necessary.
Run the second twice:
balance += 250balance += 250You’ve added $500.
That’s a very different outcome.
We can design operations with an idempotency key.
Conceptually:
send_invoice_reminder( invoice_id=4812, operation_id="reminder-4812-2026-10-08",)The application or external provider records that operation ID.
If the same operation is requested again, it can recognize the duplicate.
Conceptually:
REQUEST: operation_id = X
DATABASE: Have we processed X?
IF completed: RETURN previous result
IF currently processing: RETURN pending / reconcile
IF new: CLAIM operation X PERFORM action RECORD resultIn real systems, this typically requires durable storage, uniqueness constraints, careful handling of concurrent requests, and ideally idempotency support from the downstream service.
A Python set() is not enough for production because it disappears on restart and cannot coordinate multiple workers.
Also, do not let the model invent a fresh operation ID every time it retries.
The application should generate and preserve the ID for the original logical action.
The key principle:
Retries handle temporary failures. Idempotency handles duplicate execution.
These solve two different problems.
13. When should the model retry a tool?
Section titled “13. When should the model retry a tool?”My recommendation for a simple agent:
Keep retry policy primarily in Python.
For example:
READ-ONLY TOOL
temporary failure: Python may retry once or twice
permanent failure: return errorFor a side-effecting tool:
SEND EMAIL / UPDATE RECORD / ISSUE REFUND
timeout: do not blindly repeat
check operation statususe idempotency keyor require reconciliationYou can certainly tell the model:
If a tool is temporarily unavailable, try another approach.
But that is not a replacement for application-level safeguards.
There is another important detail here.
Suppose your agent loop gets:
{ "ok": false, "error": "temporary_failure"}The model might immediately call the same tool again.
Then again.
Then again.
Your MAX_STEPS would eventually stop it, but you may have wasted several calls.
For a production application, Python should also maintain a small retry budget for individual operations.
Conceptually:
state = { "steps_used": 0, "max_steps": 10, "tool_attempts": {},}You don’t need to implement a complicated mechanism today.
Just recognize that the agent’s step limit and a tool’s retry limit are different controls.
14. A useful distinction: failure versus uncertainty
Section titled “14. A useful distinction: failure versus uncertainty”These are not always the same.
Consider three results.
Known success:
{ "ok": true, "status": "email_sent"}Known failure:
{ "ok": false, "error": "recipient_not_found"}Unknown outcome:
{ "ok": false, "error": "delivery_status_unknown"}The third case is special.
It means:
We cannot establish whether the action completed.
That’s not something you should hide behind a generic error.
The correct next step may be to check the external service, look up the operation ID, or ask a human to investigate.
We’ll explore the human-in-the-loop side in Lesson 7.
15. A small reliability checklist
Section titled “15. A small reliability checklist”Before putting even a simple agent into production, I would answer these questions:
| Question | Simple solution |
|---|---|
| What if the model API is unavailable? | Timeout + bounded SDK retries |
| What if API credentials are invalid? | Stop and report configuration error |
| What if the model response is incomplete? | Treat as failure, not final answer |
| What if tool arguments are invalid? | Validate and return a controlled error |
| What if a record doesn’t exist? | Return explicit domain error |
| What if the database is unavailable? | Log and report tool failure |
| What if the agent loops? | Maximum steps |
| What if the agent retries a payment? | Idempotency + operation tracking |
| What if an action’s outcome is unknown? | Reconcile before repeating |
That’s most of the reliability architecture you need to think about initially.
No workflow framework required.
16. What I would actually implement first
Section titled “16. What I would actually implement first”For a small personal agent with read-only tools, I’d start with just this:
client = OpenAI( timeout=30.0, max_retries=2,)Then:
MAX_STEPS = 10Then a wrapper:
def execute_tool(tool_call): # Validate requested tool. # Parse arguments. # Execute allowed function. # Catch expected failures. # Return structured result. ...And finally:
try: answer = run_agent(task)
except Exception as error: # Log enough to diagnose the problem. # Do not claim success. ...That outer try is a last-resort boundary, not a substitute for handling known errors inside the agent.
For read-only operations, this is a reasonable starting point.
Once we introduce actions that change the outside world, we need stronger guarantees.
17. Short exercise
Section titled “17. Short exercise”Imagine you’re building an agent with these three tools:
def search_documents(query: str): ...
def get_customer(customer_id: int): ...
def send_email(customer_id: int, message: str): ...The model is helping a small business answer customer questions.
Three failures occur.
Scenario A
Section titled “Scenario A”search_documents() ↓HTTP 503What should happen?
Reasonable answer:
Python may retry with bounded backoff because this is a read-only operation experiencing a likely transient failure.
If retries fail, return an explicit error.
Scenario B
Section titled “Scenario B”get_customer(9999) ↓customer_not_foundWhat should happen?
Reasonable answer:
Don’t retry blindly.
The lookup worked. The record doesn’t exist.
Return the result to the model so it can ask the user to verify the customer ID.
Scenario C
Section titled “Scenario C”send_email(42, "Your invoice is overdue.") ↓TIMEOUTWhat should happen?
Reasonable answer:
Do not immediately send another email.
First determine whether the original operation completed.
Use an operation ID or provider-side message lookup where available.
If the outcome cannot be established safely, stop and surface the uncertainty.
If you understand why Scenario C requires different handling from Scenario A, you’ve understood the most important concept in today’s lesson.
What Breaks in Production?
Section titled “What Breaks in Production?”Let’s connect everything to actual operating conditions.
1. Retry storms
Section titled “1. Retry storms”Imagine 500 agent sessions encounter the same external API failure.
Every agent immediately retries three times.
Suddenly:
500 failures ↓1,500 additional requests ↓service gets even more overloadedRetries should have limits, backoff, and eventually system-level rate control.
Sometimes the correct response is to stop sending requests temporarily.
2. Layered retries multiply work
Section titled “2. Layered retries multiply work”The OpenAI SDK already retries certain failures.
If you wrap it in your own three-attempt retry function, you may unintentionally generate up to nine underlying requests.
Don’t stack retry mechanisms without understanding what each layer does.
3. Invalid arguments reach real functions
Section titled “3. Invalid arguments reach real functions”A function expects:
invoice_id: intbut receives:
{ "invoice_id": "banana"}The model may usually respect your schema, but your application should still validate at the execution boundary.
4. Sensitive exception details leak to the model
Section titled “4. Sensitive exception details leak to the model”A database exception might contain internal hostnames, SQL statements, connection parameters, or personal information.
Don’t automatically send:
str(exception)back to the model.
Log appropriate diagnostic details internally. Return a safe error code externally.
5. Partial success
Section titled “5. Partial success”Imagine:
create customer ✓
create invoice ✓
send welcome email ✗The task partially succeeded.
You cannot simply rerun the entire workflow without checking what already exists.
This is ordinary transaction and workflow engineering, made more important by agent autonomy.
6. The agent claims success after failure
Section titled “6. The agent claims success after failure”The email tool returns:
{ "ok": false, "error": "delivery_status_unknown"}But the model replies:
Your email was sent successfully.
That’s unacceptable.
For consequential operations, Python should verify recorded outcomes before reporting success.
Model-generated language is not evidence of successful execution.
7. Failures are impossible to diagnose
Section titled “7. Failures are impossible to diagnose”Your logs show:
Agent failed.That’s not enough.
A useful minimal log should record facts such as:
task_idstep_numbertool_namedurationsuccess/failuresafe error codeFor model API failures, a request ID can also help with debugging. The OpenAI SDK exposes request IDs on responses and relevant exceptions. GitHub
We’ll make this more systematic in Lesson 8.
The Bigger Picture
Section titled “The Bigger Picture”Let’s update the agent architecture we’ve been developing.
Before today:
MODEL ↓TOOL REQUEST ↓PYTHON ↓TOOL ↓RESULT ↓MODELNow:
USER TASK ↓ ┌────────┐ │ PYTHON │ └───┬────┘ ↓ MODEL CALL │ ┌──────┴──────┐ │ │ SUCCESS FAILURE │ │ ↓ ↓ TOOL REQUEST RETRY OR STOP │ ↓ VALIDATE REQUEST │ ↓ EXECUTE TOOL │ ┌──────┴─────────┐ │ │ SUCCESS FAILURE │ │ ↓ ↓ RETURN RESULT CLASSIFY FAILURE │ ┌─────────┴────────┐ │ │ RETRY SAFE? NOT SAFE? │ │ ↓ ↓ BOUNDED RETRY REPORT / STOPNotice who’s controlling everything.
Python.
The model still reasons about which capabilities would help complete the task.
But Python controls:
- execution;
- retries;
- error classification;
- step limits;
- permissions;
- operation identity;
- authoritative success or failure.
That separation is what turns a clever AI demonstration into software you can trust.
The Takeaway
Section titled “The Takeaway”We’ve now learned six major ingredients:
MODEL = reasoning
STRUCTURED OUTPUT = contracts
TOOLS = capabilities
PYTHON = authority
LOOP = agency
STATE = application truth
CONTEXT = what the model sees
RETRY POLICY = controlled recoveryToday, the two principles I especially want you to remember are:
A retry is only safe when you understand what repeating the operation might do.
And:
A failed request is not always a failed operation.
That second principle is particularly important for systems that send messages, modify records, charge money, or trigger external workflows.
The good news is that none of this requires sophisticated agent tooling.
You’re applying established software-engineering principles to a program that happens to use an LLM.
That’s exactly the right foundation.
Next: Lesson 7 — Human Approval and Permissions
Section titled “Next: Lesson 7 — Human Approval and Permissions”Now that our agent can execute tools and recover from failures, we need to answer a more consequential question:
What should the agent be allowed to do without asking?
We’ll build a simple approval boundary using ordinary Python.
The model might request:
delete_document(document_id=73)But Python will decide whether to:
EXECUTEor:
REQUIRE HUMAN APPROVALWe’ll also separate authentication, authorization, and approval, because those are three different things.
And we’ll do it without introducing an approval framework, state-machine library, or orchestration system.
Just explicit Python rules around the capabilities we’ve already built.