How to Mock LLM Responses Without Hiding Real AI Failures

Mock LLM responses when the test is about your application. Run live evals when the test is about the model.

That boundary sounds simple, but it is where many LLM test suites go wrong. A parser test hits a real provider. A retry test fails because a rate limit fired first. A prompt regression hides behind a handwritten fixture that always returns the answer the code wants. The team gets slow tests and still does not know whether the AI behavior is healthy.

The better plan is layered. Use fakes, mocks, and recorded fixtures to test deterministic code paths. Use live evals, golden datasets, and observability to test whether the model, prompt, and agent workflow still produce acceptable outputs.

Short Answer

To mock LLM responses without hiding real AI failures, separate application behavior from model behavior before writing the test.

• Use a fake LLM client when you are testing prompt assembly, parsing, retry behavior, UI state, or tool orchestration.
• Use record and replay fixtures when the real provider request and response shape matters.
• Use a mock server or simulator when you need controlled streaming, latency, malformed JSON, rate-limit responses, or retriable provider errors.
• Use live evals against a golden dataset when the question is output quality, prompt regression, model upgrade risk, safety behavior, or tool-use trajectory.
• Attach the evidence to the work item before merging: fixture diff, cassette refresh reason, eval command, threshold, source link, and artifact path.

Mocks protect fast feedback. Evals protect quality. If one layer tries to do both jobs, it usually does neither well.

Why Direct LLM Calls Make Bad Default Tests

Live model calls are useful evidence in the right place. They are a poor default for ordinary unit and integration tests.

OpenAI documents rate limits across metrics such as requests per minute, requests per day, tokens per minute, tokens per day, images per minute, and audio minutes per minute, with work throttled by whichever limit is hit first [12]. OpenAI pricing is also model-specific and priced per 1 million tokens, with separate prices for input, cached input, and output on supported models [11]. OpenAI's production guidance recommends separate staging and production projects so development and testing work can be isolated with their own rate and spend limits [14].

Prompt caching can reduce latency and input token cost when repeated prompt prefixes qualify, but it is an optimization for live calls, not a reason to make every parser test depend on a live provider [13].

Those facts point to a practical testing rule: a test that is checking your parser should not need the current provider, current model, current token budget, and current rate-limit state to be healthy. The live dependency adds noise to a deterministic assertion.

There is also a behavior cost. If a pull request suite regularly waits on external completions, engineers learn to avoid running it locally. If tests fail because a provider returned a different phrasing, the team learns to rerun instead of debug. That is how expensive tests become weak tests.

The LLM Testing Decision Table

Start by naming the question the test is allowed to answer.

Start With Fake LLM Clients For Unit Tests

The first layer is the fake client. It belongs in unit tests and narrow integration tests where the assertion is about your code, not the provider.

LangChain's testing docs describe a split between deterministic in-memory fakes for unit tests, real network calls for integration tests, and evals for agent execution trajectories [1]. LangChain's JavaScript unit testing docs also describe a fakeModel that can script exact responses, tool calls, and errors, and help assert what the model received [2].

That is the right boundary. A fake should sit where your application meets the LLM dependency. It should not reach deep into private framework internals, because that turns a test into a bet on library implementation details.

A useful fake covers more than "return this text":

const fakeModel = createFakeModel([
  { content: '{"status":"approved","reason":"matches policy"}' },
  { toolCall: { name: 'lookupTicket', args: { id: 'T-421' } } },
  { error: new Error('provider timeout') },
]);

const result = await routeSupportRequest({
  model: fakeModel,
  message: 'Can this refund be approved?',
});

expect(result.status).toBe('approved');
expect(fakeModel.lastPrompt()).toContain('refund policy');

The exact helper will vary by stack. The shape should not: script the model boundary, run your application code, and assert the behavior you own.

Fake clients are also the fastest way to remove live calls from pull request tests. If the code path formats a prompt, parses JSON, maps a tool call, retries a timeout, or updates streaming UI state, a fake response is usually stronger evidence than a live completion.

Use Record And Replay When API Shape Matters

Handwritten mocks are too tidy for some integration tests. They prove your code handles the response you imagined. They do not prove it handles the response shape your provider actually returned.

That is where record and replay helps. VCR.py records an HTTP request once and replays the response later, which makes tests deterministic and usable offline [3]. pytest-vcr supports filtering headers, including replacing authorization headers in cassettes [4].

Use this layer when your assertion is about compatibility:

• Does the request include the fields the provider expects?
• Does your parser handle the nested response structure?
• Does your SDK wrapper behave the same after a dependency upgrade?
• Does your tool-call parser handle the provider's captured payload?

Treat cassettes like code. Review the diff when they change. Redact credentials before committing. Store a refresh note that says why the cassette changed: provider SDK upgrade, endpoint version change, schema drift, or intentionally expanded fixture coverage.

A cassette is not proof that the answer is good. It is proof that your application can handle a captured exchange.

Simulate The Provider Edge Cases Your Code Must Survive

The hardest LLM bugs often live at the wire edge: a slow first token, a stream that ends halfway through a JSON object, a 429 that should back off, a retriable 5xx, or a tool call with the wrong shape.

WireMock's 2023 MockGPT post introduced MockGPT as a mock module for OpenAI-powered applications and said it could return canned responses, delays, and controlled unpredictability [15]. Treat that as a vendor claim, not current benchmark evidence. The useful idea is broader than one endpoint: a mock server lets the whole application talk to a controlled OpenAI-compatible surface.

Practitioner discussions around LLM mock APIs often focus on SSE or EventStream format, time to first token, jitter, malformed responses, and mid-stream failures. Those are community signals, not proof of market size, but they name the cases engineers actually need to test.

Good simulation tests make failure precise:

it('keeps partial assistant text when the stream fails mid-response', async () => {
  server.stream('/v1/chat/completions', [
    chunk('The deployment failed because'),
    chunk(' the database migration'),
    networkClose(),
  ]);

  const state = await renderAssistantResponse('Why did deploy fail?');

  expect(state.partialText).toContain('database migration');
  expect(state.status).toBe('recoverable_error');
  expect(state.retryLabel).toBe('Retry');
});

Faker-style generated values still have a place here. Use them to broaden parser coverage: empty content, long paragraphs, unexpected language, truncated JSON, missing fields, and extra fields. Do not use fake variation as a substitute for provider-shape tests. It finds resilience bugs. It does not validate the real API contract.

Keep Live Evals Out Of Ordinary Pull Request Tests

Live evals belong in CI/CD, but not every pull request needs to run every live eval.

Promptfoo documents CI/CD workflows for evals and red teaming, including quality gates, reports, cost tracking, and commands such as promptfoo eval [5]. Its assertion system can validate equality, JSON structure, similarity, custom functions, and score thresholds [6]. DeepEval documents datasets, test cases, and running deepeval test run in CI/CD [7]. OpenAI describes evals as tests for model outputs against style and content criteria, and says they are important when upgrading or trying new models [10].

That gives you the CI shape:

This is also where cost and speed of agentic workflows and token efficiency in AI-assisted development become testing concerns. A slow, expensive eval can still be the right gate. It just should not hide inside the same path as a parser unit test.

Golden Datasets, Agent Trajectories, And Observability

Mocks verify how your code behaves given a controlled input. Golden datasets verify whether the real system still behaves acceptably on important examples.

A useful golden dataset includes:

• The input or scenario.
• Expected answer characteristics, not always one exact output.
• Negative examples where the model should refuse, ask for clarification, or avoid a tool.
• The scoring method and pass condition.
• The owner who can decide whether a failure is a real regression.

Agent workflows need one more layer. DeepEval's agent docs describe agent evals as different because workflows can include tools, chained LLM calls, and RAG modules, with tracing spans for components [8]. The final text can look fine while the agent chose the wrong tool, skipped retrieval, or used stale context.

Observability helps catch that difference. OpenTelemetry's GenAI semantic conventions define attributes for request model, stream flag, temperature, response model, time to first chunk, token type, and input or output token usage [9]. That does not mean every stack captures every field. It does show that model identity, streaming behavior, and token usage are observable parts of an LLM system, not vibes to argue about after a failure.

For agent-heavy applications, test evidence should include both final-output quality and trajectory quality:

• Did the agent call the right tool?
• Did it pass the expected arguments?
• Did retrieval happen before synthesis?
• Did token usage or time to first chunk change materially?
• Did the model change between passing and failing runs?

Those questions are not good fits for a handwritten mock. They need evals, traces, and reviewable artifacts.

What An AI Coding Agent Should Leave Behind

When an AI coding agent changes LLM-facing code, the hard part is not only whether the tests pass. The hard part is whether the next human or agent can see what changed and why.

Agiflow should sit in that workflow as task evidence infrastructure, not as an eval framework or mock server. The product scope is a commercial, MCP-connected project board with durable task state, artifacts, vault entries, workflow locks, and acceptance criteria. That is enough to make LLM testing work more legible.

A strong task record for a prompt or fixture change includes:

• The test layer being changed: fake, cassette, simulation, live eval, trace, or acceptance criteria.
• The fixture or cassette diff.
• The reason for refreshing recorded output.
• The eval command that ran, such as a Promptfoo or DeepEval command from the team's project.
• The dataset or case group used.
• The project-specific threshold or rubric.
• The artifact path for reports, traces, screenshots, or logs.
• The source link that justifies any new factual behavior.

This matters more when external assistants work through MCP. An assistant that can re-read acceptance criteria as durable task state before changing a test is less likely to invent a new contract mid-stream. A workflow lock can also prevent two humans or agents from refreshing the same prompt, cassette, or eval suite at the same time.

The same principle shows up in architecture checks for AI-generated code and MCP-native project management: AI-assisted development needs visible feedback loops, not only chat memory.

When Should You Not Mock LLM Responses?

Do not mock the LLM when the model behavior is the thing under test.

Use a live eval or production-like observation when you need to know:

• Whether the answer is useful, relevant, complete, or safe.
• Whether a prompt change regressed important examples.
• Whether a model upgrade changed output style or content quality.
• Whether an agent chose the right tool sequence.
• Whether retrieval, chained calls, or RAG behavior worked as intended.
• Whether current provider latency, token usage, or rate-limit behavior is acceptable.

This is the counterweight to the whole mocking strategy. Mocks are valuable because they remove irrelevant uncertainty. They become dangerous when they remove the exact uncertainty you needed to measure.

Practical Starting Point

If your current suite hits a live provider from ordinary tests, start with the smallest useful cleanup.

1. Label each LLM-facing test by layer: fake, cassette, simulation, live eval, trace, or task evidence.
2. Replace live calls in unit tests with fake clients.
3. Add record and replay where real API shape matters.
4. Add provider simulations for streaming and failure paths.
5. Build a small golden dataset for the most important user flows.
6. Run live evals on prompt changes, model upgrades, release gates, or schedules.
7. Attach the evidence to the task before merging.

The point is not to mock more. The point is to stop asking one test to prove everything. Use the LLM test-layer checklist before merging the next AI integration change, so you know when to mock LLM responses and when to run live evals.

FAQ

When should I use a fake LLM client instead of the real provider API?

Use a fake when the assertion is about code you own: prompt assembly, parser behavior, retry logic, UI streaming state, error handling, or tool orchestration. A live provider adds noise to those assertions.

When are recorded HTTP fixtures better than handwritten mocks?

Use recorded fixtures when real request and response shape matters. They are especially useful around SDK upgrades, wrapper changes, and provider schema compatibility. Redact credentials and review cassette diffs before committing.

How should LLM evals fit into CI/CD?

Keep fast mock-based tests in every pull request. Run live eval subsets for prompt, model, retrieval, or agent behavior changes. Run fuller eval suites on schedules, release gates, and model upgrades.

How do I test streaming, latency, malformed JSON, and provider errors?

Use a network-level mock or simulator. Script slow first token, jitter, mid-stream failure, malformed payloads, 429 responses, and retriable 5xx errors so your application behavior is repeatable under failure.

What evidence should an AI coding agent leave after changing prompt or test behavior?

It should leave the fixture diff, cassette refresh reason, eval command, dataset or case group, threshold or rubric, source links, and artifact path on the task. Passing tests without that trail are hard to trust later.

References

1. LangChain testing docs
2. LangChain JS unit testing docs
3. VCR.py usage docs
4. pytest-vcr docs
5. Promptfoo CI/CD docs
6. Promptfoo assertions docs
7. DeepEval CI/CD docs
8. DeepEval agent evaluation docs
9. OpenTelemetry GenAI semantic conventions
10. OpenAI evals docs
11. OpenAI pricing docs
12. OpenAI rate limits docs
13. OpenAI prompt caching docs
14. OpenAI production best practices
15. WireMock MockGPT post