How to Set Up a Mock LLM Server for Local and CI Testing

A mock LLM server is a local process that speaks the same HTTP API as a model provider and returns responses you control. Your application points its base URL at the mock instead of the real endpoint, and nothing else in the code changes. Tests then run without network calls, without provider keys, and without paying per token.

That is the whole idea. The interesting part is deciding what belongs behind the mock and what does not.

Short Answer

Use a mock LLM server when the thing under test is your application, not the model.

• Point your client at a local OpenAI-compatible endpoint by overriding the base URL.
• Return deterministic responses so the same test input always produces the same output.
• Simulate the failure modes that matter: rate limits, timeouts, malformed JSON, truncated streams.
• Keep the mock out of any test whose actual question is output quality.

The trap is letting the mock answer questions it was never able to answer. A mock proves your retry logic works. It cannot prove your prompt still produces a good summary.

When to Mock and When to Call the Real API

The boundary is easier if you ask what would make the test fail for a legitimate reason.

Mock when the failure would be your code. Prompt assembly, response parsing, retry and backoff behavior, token accounting, error handling, streaming UI state, database writes after a completion, and tool dispatch logic are all deterministic paths. They do not need a real model, and involving one makes them flaky.

Call the real API when the failure would be the model. Output quality, prompt regressions after an edit, behavior changes after a model upgrade, safety responses, and multi-step tool-use trajectories all depend on the actual model. A mock will happily return whatever you told it to return, which means these tests pass even when the real behavior has broken.

Most teams need both, running on different schedules. Mocked tests belong on every commit. Live evaluation belongs on a slower cadence, on a golden dataset, with thresholds you agreed on in advance. The deeper treatment of that split is in effective practices for mocking LLM responses, which covers the layered approach in more detail than this post does.

What a Local Mock Server Needs to Simulate

Most mock servers start as a single hardcoded response and grow from there. These are the capabilities that tend to matter, roughly in the order teams need them.

Recorded fixtures

Capture real request and response pairs once, store them as files, and replay them on later runs. This is the record-and-replay pattern familiar from HTTP testing libraries, and it keeps the response shape honest because a real provider produced it.

The cost is maintenance. Fixtures drift when the provider changes its response format or when your prompt changes and no longer matches the recorded request. Decide up front how a fixture gets refreshed and who reviews the diff, because a silently regenerated fixture can hide a real regression.

Deterministic template responses

Not every test needs a recorded payload. Often you want a response built from the request: echo the last user message, return a fixed JSON object matching your schema, or produce a response whose length depends on an input parameter. Template responses are cheaper to maintain than fixtures and are usually the right default for testing parsing and orchestration.

Keep them deterministic. A mock that returns random content reintroduces exactly the flakiness you were trying to remove.

Error and latency injection

This is the capability that justifies a mock server over a simple in-process fake. Real providers fail in specific ways, and your retry logic is only as good as the failures you have tested against.

Worth simulating: HTTP 429 with and without a Retry-After header, HTTP 500 and 503, connection timeouts, responses that arrive slower than your client timeout, malformed JSON in the body, and a stream that ends mid-message. A mock lets you trigger each on demand rather than waiting for production to produce one.

Streaming

If your application streams tokens, the mock has to stream too. Server-sent events with the same chunk framing as the real API will surface bugs that a single-shot response never will: partial UTF-8 sequences split across chunks, buffering that breaks the UI, and cleanup that never runs when the client disconnects early.

Being able to control chunk timing is what makes this useful. A stream that delivers everything instantly does not test the states your interface actually passes through.

Tool and function calls

Applications that use tool calling need the mock to return tool-call responses, accept the tool result on the next turn, and continue. Without this, the entire orchestration layer stays untested. Being able to script a sequence, such as tool call, then result, then final answer, is what lets you test the loop rather than one request.

Running It in CI

A few practical points make the difference between a mock server that helps and one that becomes its own maintenance problem.

Configure the base URL through an environment variable that defaults to the real endpoint, so production code has no test-specific branching. The test environment sets it to the local mock, and nothing else changes.

Start the mock as a service in the CI job rather than inside the test process. It survives across test files, and you avoid port conflicts between parallel workers by allocating a port per job instead of hardcoding one.

Fail the build if a test unexpectedly reaches a real provider. The simplest version is to set an obviously invalid API key in the test environment, so any call that escapes the mock fails loudly instead of quietly billing you.

Keep fixtures in version control and treat a fixture change as a reviewable diff. That is the only way a reviewer can tell the difference between a legitimate refresh and a regression that someone papered over.

What Mocking Cannot Tell You

A mock server gives you speed and determinism. It does not give you confidence about model behavior, and it is worth being blunt about the gap.

A mock cannot tell you whether a prompt edit degraded output quality, whether a newer model version changed the format your parser depends on, whether the model still refuses what it should refuse, or whether an agent picks the right tool when the choice is genuinely ambiguous. Every one of those tests passes against a mock regardless of the answer, because the mock is returning your recorded expectation rather than the model's actual behavior.

The related risk is cost blindness. Mocked runs consume no tokens, so a change that doubles real token usage looks free in CI. Token accounting is worth measuring separately, and token efficiency in AI-assisted development covers where that consumption tends to accumulate. For the broader latency and spend picture in multi-step systems, optimising cost and speed of agentic workflows is the companion piece.

Where This Fits in Agent Workflows

Once more than one person or assistant works on the same test suite, the mocking setup itself becomes shared state. Which fixtures are current, why a cassette was refreshed, which evaluation run justified a threshold change, and who is currently regenerating a fixture are all questions that outlive any single session.

This is the part that tends to live in chat history and then disappear. Keeping it attached to the task instead means the next person, or the next assistant session, can read the reasoning rather than reconstruct it. Agiflow is an MCP-connected project board built for that: tasks, artifacts, acceptance criteria, and workflow locks that external assistants such as ChatGPT, Claude, Cursor, or Codex can read and update through approved tools. Agiflow does not run the agents and is not a mock server or an evaluation framework. It holds the evidence around the work.

Wrap Up

A mock LLM server is a narrow tool that does its job well. Point your client at a local endpoint, return controlled responses, and simulate the errors and streaming behavior your code has to survive. Your test suite gets fast and deterministic, which is exactly what you want on every commit.

Just keep the boundary honest. The moment a mocked test starts standing in for a question about model quality, it has stopped testing anything real. Mock the application, evaluate the model, and keep the two on separate schedules.