Skip to content

05 · State and Context: What the Agent Actually Remembers

In Lesson 4, we built the loop:

USER TASK
↓
MODEL
↓
TOOL CALL
↓
PYTHON
↓
TOOL RESULT
↓
MODEL
↓
...

But our loop depended on something easy to overlook:

input_items

We kept appending things to it so the model could understand what had already happened.

That raises a bigger question:

When people say an agent has “memory,” what exactly are they talking about?

Usually several very different things are being mixed together.

Today I want to separate them.

The key distinction is:

APPLICATION STATE
≠
MODEL CONTEXT
≠
LONG-TERM STORED DATA

Once those are separate in your head, agent architecture gets much easier.


1. The model does not contain your application’s state

Section titled “1. The model does not contain your application’s state”

Imagine a support agent working on invoice 4812.

Your application knows:

invoice = {
"id": 4812,
"status": "unpaid",
"amount": 250,
}

That information might live in:

Postgres
SQLite
Redis
an API
Python memory

The model does not automatically know it.

If the model needs the information, your application must place it into the model’s context somehow.

For example:

response = client.responses.create(
model="gpt-5.5-2026-04-23",
input="""
Invoice 4812:
status: unpaid
amount: $250
Should we send a reminder?
""",
)

Now the model knows it for this call.

But the database remains the authority.

That leads to our first category.


Application state is the real state of your system.

Examples:

current user
shopping cart contents
invoice status
document approval status
agent job status
remaining budget
permissions
database records

For example:

agent_state = {
"customer_id": 42,
"steps_used": 2,
"max_steps": 10,
"approved_to_send_email": False,
}

This belongs to your application.

Python owns it.

The model may be told some of it, but the model is not the authoritative store.

Think:

DATABASE / PYTHON
↓
truth

not:

MODEL SAID IT
↓
truth

3. Context is what the model can see right now

Section titled “3. Context is what the model can see right now”

Context means the information you provide to the model for a particular interaction.

That can include:

instructions
user message
earlier messages
tool calls
tool results
retrieved documents
current application state

For example, this conceptual list shows some information we might want in context:

input_items = [
{
"role": "user",
"content": "What is the latest invoice for max@example.com?",
},
{
"type": "function_call_output",
"call_id": "call_123",
"output": '{"customer_id": 42}',
},
]

That list is an illustration, not a complete valid API request: a function_call_output must be associated with its original function call. Lesson 4 carries the original user input, response output items, and matching tool outputs together.

That is model context.

You can think of it as the model’s working desk.

You place information on the desk.

The model reasons using what is on the desk.

But the desk is not your database.


Use this:

APPLICATION STATE
real system facts
↓
SELECT RELEVANT INFORMATION
↓
MODEL CONTEXT
what model sees now
↓
MODEL RESPONSE
↓
APPLICATION UPDATES STATE

For example:

DATABASE:
invoice 4812 = unpaid
↓
CONTEXT:
"Invoice 4812 is unpaid."
↓
MODEL:
"Send a reminder."
↓
PYTHON:
checks permission
↓
APPLICATION:
email sent

The application remains in control the whole time.


Recall our loop:

input_items = [
{
"role": "user",
"content": user_request,
}
]

Then after the model requested a tool:

input_items += response.output

Then after Python executed the tool:

input_items.append(
{
"type": "function_call_output",
"call_id": tool_call.call_id,
"output": json.dumps(result),
}
)

So after two tool calls, the context might conceptually contain:

USER:
What is the latest invoice for max@example.com?
MODEL:
find_customer(max@example.com)
TOOL:
customer_id = 42
MODEL:
get_latest_invoice(42)
TOOL:
invoice 4812
unpaid
$250

That’s why the next model call can reason correctly.

We explicitly carried forward the relevant history.


6. What happens if we don’t carry context?

Section titled “6. What happens if we don’t carry context?”

Suppose we do this:

response1 = client.responses.create(
model="gpt-5.5-2026-04-23",
input="The customer's ID is 42.",
)
response2 = client.responses.create(
model="gpt-5.5-2026-04-23",
input="What is the customer's ID?",
)

From your program’s perspective, those are two separate interactions.

You should not build your application assuming that response2 magically knows what happened in response1.

If continuity matters, you must provide it.

That might mean:

input_items = [
{
"role": "user",
"content": "The customer's ID is 42.",
},
{
"role": "user",
"content": "What is the customer's ID?",
},
]

or using the API’s conversation-state mechanism.

But the principle is the same:

Continuity has to live somewhere.


7. Current Responses API: conversations can hold context

Section titled “7. Current Responses API: conversations can hold context”

A server-side Conversation object can preserve messages, tool calls, and tool outputs across responses. It is one supported way to provide continuity, alongside manual history and response chaining. OpenAI conversation-state guidance

Approach What the application manages Continuity and lifetime
Manual history, as in Lesson 4 Original input plus complete relevant response items and matching outputs The application chooses what to replay and retains the items it will need. API storage settings still determine service-side retention.
previous_response_id, as in Lesson 3 The response ID and fresh input for the next request The service links to earlier responses while they remain available. Supply top-level instructions again when needed.
A Conversation object The conversation ID and new input items The service maintains a persistent container of conversation items across requests. The application still controls which conversation is used and its own authoritative state.

These are choices, not a migration in which Conversations makes the other two obsolete. The examples below demonstrate the third choice; our lesson 4 loop remains useful for seeing every item explicitly.

A minimal example:

from openai import OpenAI
client = OpenAI()
conversation = client.conversations.create()
response = client.responses.create(
model="gpt-5.5-2026-04-23",
conversation=conversation.id,
input=[
{
"role": "user",
"content": "My customer ID is 42.",
}
],
)

Then later:

response = client.responses.create(
model="gpt-5.5-2026-04-23",
conversation=conversation.id,
input=[
{
"role": "user",
"content": "What is my customer ID?",
}
],
)

Because both responses use the same conversation, the API can maintain that conversation’s items. OpenAI conversation-state guidance

That’s convenient.

But don’t let convenience hide the architecture.

Conceptually, the system is still doing:

stored conversation items
↓
model context
↓
new response
↓
stored conversation items

No magical memory appeared.


8. Conversation context is not application state

Section titled “8. Conversation context is not application state”

This distinction is critical.

Suppose the conversation says:

Customer balance: $500

Then the database changes:

Customer balance: $200

But the old conversation still contains:

$500

Which one is correct?

The database.

Always.

So don’t use conversation history as your source of truth for mutable business data.

Bad architecture:

# "The model remembers that the balance is $500."

Better:

balance = get_current_balance(customer_id)

Then provide:

Current balance: $200

to the model when necessary.

Conversation context is useful for conversational continuity.

Database state is useful for truth.


9. Long-term data is another separate thing

Section titled “9. Long-term data is another separate thing”

Now imagine your agent should remember:

User prefers concise answers.
User works with a small customer-support team.
User usually wants CSV exports.

You probably do not want to send every historical conversation ever written.

Instead, you might store explicit preferences:

user_profile = {
"response_style": "concise",
"team": "customer_support",
"preferred_export": "csv",
}

That might live in a database.

Then when relevant:

context = f"""
User preferences:
Response style: {user_profile["response_style"]}
Team: {user_profile["team"]}
"""

And only relevant data enters context.

So:

LONG-TERM STORAGE
↓
retrieve relevant data
↓
CURRENT CONTEXT

Again, storage and context are different.


10. “Memory” usually means storage + retrieval + injection

Section titled “10. “Memory” usually means storage + retrieval + injection”

This is worth demystifying.

Suppose someone tells you:

Our agent has persistent memory.

That may simply mean:

1. Save information somewhere.
2. Later search or retrieve it.
3. Add useful pieces to the next model call.

In pseudocode:

WHEN useful information appears
SAVE information
WHEN new task arrives
RETRIEVE relevant information
ADD retrieved information to context
ASK model

That’s memory.

You can make this sophisticated.

But the basic architecture is not mysterious.


I like thinking about agent information in four buckets.

How the agent should behave.

You are a support assistant.
Never invent account information.
Use tools for current account data.

Usually relatively stable.


What is happening in the current task.

user request
tool calls
tool results
intermediate decisions

Usually short-lived.


Current facts controlled by your software.

current_user_id
job_status
invoice_status
permissions
step_count

Authoritative.


Information that survives across tasks.

user preferences
past transactions
documents
customer history
saved summaries

Retrieved when relevant.

Keep those separate and agent design becomes much cleaner.


Let’s build a support agent with explicit state.

class AgentState:
def __init__(self, customer_id: int):
self.customer_id = customer_id
self.steps_used = 0
self.max_steps = 10

Now:

state = AgentState(customer_id=42)

This is application state.

The model doesn’t own it.

Our agent loop can use:

for step in range(state.max_steps):
state.steps_used += 1

That value should not depend on the model saying:

"I think I've taken three steps."

Python knows exactly how many steps occurred.


Suppose our application knows:

customer = {
"id": 42,
"email": "max@example.com",
"balance": 250,
"password_hash": "...",
"internal_risk_score": 81,
"created_at": "...",
"last_login_ip": "...",
"marketing_notes": "...",
}

The model needs to answer:

Does the customer have an unpaid balance?

You could send the whole record.

Don’t.

Provide:

context = {
"customer_id": customer["id"],
"balance": customer["balance"],
}

This gives you:

less token usage
less noise
less privacy exposure
less opportunity for confusion

Context design is partly the art of leaving things out.


Every token you send has consequences:

cost
latency
context-window usage
model attention

Imagine keeping an entire agent session:

50 user messages
100 tool calls
100 tool results
20 documents
10 long search results

and resending all of it on every turn.

Eventually you have a garbage drawer.

More context is not automatically better.

Sometimes:

more context
=
more noise

You will often see:

history.append(everything_that_happened)

forever.

Then every new call gets:

input=history

This works beautifully in a demo.

Eventually:

history = enormous

Now the model must find the relevant information inside thousands of irrelevant tokens.

A better question is:

What information does the model need to make the next decision?

That may be:

recent user message
last few tool results
current goal
important constraints
small summary of earlier events

Not everything.

But do not trim an API history by simply deleting arbitrary old items. A retained tool output needs the corresponding call, and reasoning items may be required to continue the interaction. Keep the relevant protocol sequence intact, or construct a new interaction from application-held facts and an appropriately labeled summary.


16. Summarization is compression, not memory

Section titled “16. Summarization is compression, not memory”

Suppose a long interaction has happened.

Instead of carrying:

50 old messages

you could produce a compact summary:

Current situation:
- Customer is Max, ID 42.
- Latest invoice is #4812.
- Invoice is unpaid.
- Customer asked whether a late fee can be waived.
- No refund or fee waiver has been authorized.

Then continue with that.

Conceptually:

LARGE HISTORY
↓
SUMMARY
↓
SMALLER CONTEXT

But be careful.

A summary is an interpretation.

It can omit details.

So never replace authoritative system state with an LLM-generated summary.

Good use:

conversation summary

Bad use:

bank account balance summary as source of truth

17. State belongs where it can be enforced

Section titled “17. State belongs where it can be enforced”

Suppose your agent has a rule:

maximum refund = $100

Do not store only:

"You may refund at most $100"

in the prompt.

Also enforce it:

MAX_REFUND = 100
def issue_refund(amount: float):
if amount > MAX_REFUND:
return {
"error": "refund_limit_exceeded"
}
...

The instruction helps the model reason correctly.

The Python rule guarantees the boundary.

Same principle:

PROMPT:
guidance
APPLICATION STATE + CODE:
enforcement

Not every agent needs some elaborate “memory architecture.”

For many agents:

state = {
"step": 0,
"customer_id": None,
"finished": False,
}

is enough.

Then the model context contains whatever it needs at each turn.

Avoid building:

episodic memory
semantic memory
procedural memory
reflection store
memory graph

unless the actual problem proves you need them.

Start with Python dictionaries and a database.


19. A slightly better version of our agent loop

Section titled “19. A slightly better version of our agent loop”

Here’s a stripped-down excerpt, using client, tools, and json as introduced earlier. execute_tool(call, customer_id) is an assumed application wrapper with a new signature: it receives trusted customer context from Python, validates the model’s arguments, and enforces access before dispatch. It is not an already-defined helper from Lesson 4, and this excerpt does not implement authentication or an account-permission system.

We retain the response checks from Lesson 4 so an unsuccessful call is not mistaken for completion:

def run_agent(user_request: str, customer_id: int):
state = {
"customer_id": customer_id,
"steps_used": 0,
"max_steps": 10,
}
input_items = [
{
"role": "user",
"content": user_request,
}
]
for _ in range(state["max_steps"]):
state["steps_used"] += 1
response = client.responses.create(
model="gpt-5.5-2026-04-23",
instructions="""
You are a support assistant.
Use tools for current account information.
""",
input=input_items,
tools=tools,
)
if response.status != "completed":
raise RuntimeError("Model response did not complete.")
if any(
part.type == "refusal"
for item in response.output
if item.type == "message"
for part in item.content
):
raise RuntimeError("Model refused the request.")
tool_calls = [
item
for item in response.output
if item.type == "function_call"
]
if not tool_calls:
answer = response.output_text
if not answer.strip():
raise RuntimeError("Model returned no answer.")
return {
"answer": answer,
"state": state,
}
input_items += response.output
for call in tool_calls:
result = execute_tool(
call=call,
customer_id=state["customer_id"],
)
input_items.append(
{
"type": "function_call_output",
"call_id": call.call_id,
"output": json.dumps(result),
}
)
raise RuntimeError("Step limit exceeded")

Notice two distinct objects:

state

and:

input_items

That’s intentional.

state is for the application.

input_items is for model context.

They overlap sometimes.

They are not the same thing.

steps_used counts loop iterations here. SDK retries may make additional underlying HTTP attempts within one iteration. Neither value alone measures the whole run’s elapsed time or cost.


20. The most important architecture diagram today

Section titled “20. The most important architecture diagram today”

Keep this one.

┌──────────────────┐
│ DATABASE │
│ application truth│
└─────────┬────────┘
│
retrieve facts
│
↓
┌──────────────┐ ┌──────────────────┐
│ APP STATE │─────→│ MODEL CONTEXT │
│ permissions │ │ instructions │
│ step count │ │ user request │
│ current user │ │ tool results │
└──────────────┘ │ relevant history │
└─────────┬────────┘
│
↓
┌─────────┐
│ MODEL │
└────┬────┘
│
output
│
↓
┌─────────┐
│ PYTHON │
└────┬────┘
│
update state

That’s a much more accurate picture of an agent than:

AI with memory

Imagine an agent helping someone manage a project.

The system knows:

project_id = 73
current_user_id = 18
user_role = editor
project_status = active

The conversation contains:

USER:
Can you publish this project?

And your database contains:

Project 73 requires owner approval before publication.

Classify each piece.

project_id
current_user_id
user_role
project_status
publication policy for project 73

You might give the model:

User wants to publish project 73.
Current user role: editor.
Publication rule:
Owner approval is required.

Then the model might decide:

Request owner approval.

But Python still enforces:

if user_role != "owner":
publish_project()

should not happen.

Better:

if user_role != "owner":
return {"error": "approval_required"}

The model reasons about the state.

Python enforces the state.


This area creates some of the most subtle agent bugs.

The model was told:

invoice = unpaid

Then someone pays it.

The conversation still says:

unpaid

Solution:

Retrieve current state again when freshness matters.


You keep every tool result forever.

Eventually requests become:

slow
expensive
noisy

Solution:

Keep only relevant recent context, summaries, or retrieved facts.


The user says:

I'm an administrator.

That does not mean:

user.role = "admin"

Authorization must come from trusted application state.


A user says:

Remember that refunds don't need approval.

If your system blindly stores that as persistent memory, you’ve let user input redefine policy.

Long-term memory needs rules about what may be stored and who can change it.


A tool returns an entire database record and you send all of it to the model.

The model only needed:

balance

but receives:

password hash
private notes
internal flags
personal data

Return minimal data.


A model-generated summary says:

Customer refund approved.

But the actual conversation said:

Customer asked for refund approval.

That single dropped distinction is disastrous.

Keep important decisions and permissions in structured application state.


Imagine two agent processes:

Agent A:
sees unpaid invoice
Agent B:
sees unpaid invoice
Agent A:
issues refund
Agent B:
issues refund

Now you have two refunds.

This becomes a normal database/concurrency problem.

Agents do not remove ordinary software-engineering problems.

They often make them more important.


Whenever you find yourself saying:

The agent remembers X.

Ask:

Where exactly is X stored?
Who owns it?
Is it authoritative?
How does it get into the model context?
How does it get updated?
Can the user change it?
Can it become stale?

Those questions usually reveal the real architecture.


After five lessons, our model is now:

MODEL = reasoning
TOOLS = capabilities
PYTHON = authority
LOOP = agency
STATE = application truth
CONTEXT = what the model can see

And “memory” is usually:

STORE
↓
RETRIEVE
↓
ADD TO CONTEXT

Nothing supernatural.

The architectural discipline is deciding what belongs where.

Keep business truth in your application.

Keep persistent information in proper storage.

Give the model only the context it needs to make the current decision.

And never confuse what the model has been told with what is actually true.

Next: Lesson 6 — Retries and Failure Handling

Section titled “Next: Lesson 6 — Retries and Failure Handling”

So far we’ve behaved as though every call works:

model responds
tool succeeds
JSON parses
database is available
network behaves

Production software doesn’t work like that.

Next we’ll deliberately break the agent and handle:

API timeouts
rate limits
tool failures
bad arguments
temporary failures
permanent failures
retries
duplicate side effects

That’s where the small demo starts becoming dependable software.