Erica Wang / AI systems
Was escalation the right decision?
A correct answer can still come from the wrong path.
YOUR AI
ONLINE
YOU
AI ASSISTANT
Was escalation the right decision?
- Intent
- Retrieval
- Routingwrong route
- Tool
- Rules
- Response
A correct answer can still come from the wrong path.
The problem
A response is not the system.
A lot of AI evaluation still focuses heavily on the final output.
Was the answer correct? Was it relevant? Was the tone appropriate?
Those questions matter — but they only describe what surfaced at the end of the system.
A plausible answer may still come from stale knowledge, the wrong routing decision, an incorrect tool call, or a missing policy constraint.
The output can look correct while the system behind it is wrong.
This is the problem I want to understand.
I want to make the steps behind an AI decision easier to understand, so people can see what happened and judge whether the action made sense.
Before the answer
What happened before the response?
The final response is step 08.
The final answer often doesn’t show where the failure actually began.
- 01
User intent
What did the system think the user wanted?
- 02
Prompt / context
What instructions shaped the decision?
- 03
Retrieved knowledge
What evidence entered the system?
- 04
Routing
Why was this path selected?
- 05
Tool call
What action was actually invoked?
- 06
Policy / rules
What constraints allowed or blocked it?
- 07
Model output
What did the model generate?
- 08
Final response
What did the user eventually see?
Field notes
Things I currently believe — and want to test.
A correct answer can still come from the wrong path.
If an agent retrieves the wrong document, chooses an unintended route, and still produces a plausible answer, output-level evaluation may classify the interaction as successful. The source and routing errors could then go undetected.
For the systems I’m interested in, reliability often means looking beyond what the system said to how it got there.
Open question → How should evaluation catch a plausible answer produced by an invalid path?
RAG failure, routing failure, and source-data failure are different problems.
I want to separate missing evidence from a bad decision about good evidence. My current assumption is that a prompt change alone can leave an outdated policy document or an unintended tool route unaddressed. I’m testing whether distinguishing these failures helps me choose a more useful intervention.
Example → A current source sent to the wrong tool is a routing problem, not a retrieval problem.
Retrieved information is not enough to justify an action.
In the refund example, a FAQ describes general conditions. It may not tell me whether this particular order qualifies or needs human review. I want to understand how the system checks the order, identifies the applicable policy, and connects that information to the action it takes.
Open question → What checks connect retrieved information to an action?
I want evidence before intervention.
Changing a prompt can make a visible symptom disappear without explaining its cause. Before changing behavior, I want to understand what evidence is available about the source, route, tool call, and rules that shaped the result. Otherwise, it becomes difficult to tell whether the next version actually fixed the problem or only changed the symptom.
Open question → What is the smallest useful record of a decision?
What I want to work on
The problems I want to solve.
These are directions I want to explore. I’m starting with how to explain an AI decision and what needs to be recorded to review it.
Make AI decisions easier to follow
Show what the system received, what information it used, how it selected the next step, and what it did. I want someone without an engineering background to be able to follow the process and ask whether each step made sense.
ECP0 — a record of how the decision happened
Explore what should be recorded about sources, decision points, and actions, so an unexpected result can be traced back to the step that needs attention.
Areas of work
Three ways into the system.
Decision paths
How agents move from intent to action.
Evidence
What evidence would help me reconstruct a decision?
Governance
Where might explicit rules be more useful than model judgment?
What I currently believe
Working beliefs.
- 01
Evidence before intervention.
I want to understand the failure before changing behavior.
- 02
Traceability by default.
When possible, preserve enough of the decision path to review what happened — not only the final response.
- 03
Determinism where it helps.
Explore where explicit constraints are more useful than model judgment, especially for consequential actions.
What I still don't know
Unresolved questions.
- 01
How much of an agent decision can realistically be reconstructed?
- 02
What evidence is actually worth preserving?
- 03
Where should deterministic rules stop and model judgment begin?
About
I’m Erica Wang. This is where I share what I’m working through.
I work with production AI systems from the application and customer-facing side, including LLM evaluation, RAG behavior, routing, and API integrations. I’m interested in how a system turns information into decisions and actions — and in making that process easier to inspect and question. ECP0 is where I organize those questions, document what I observe, and build small experiments around them.