Your Prompts Need Integration Tests Too

A prompt looks like configuration. Edit some text, try a few examples, ship.

That's convenient. It works while the prompt is the whole feature. It stops working as soon as there is retrieval, parsing, and application code around it.

Prompts evolve. Product finds new use cases. Edge cases pile up. Models change. Retrieval changes. Input formatting changes. Output schemas change.

At that point, walking through examples by hand is no longer enough.

The prompt is one hop in the pipeline

A production prompt rarely lives alone.

There's usually code that prepares the input, retrieves context, formats it, renders the prompt, calls the model, parses the response, validates structured output, and maybe transforms it again.

A playground can tell you whether the prompt behaves for a given input. It cannot tell you whether the application handed it the right input in the first place.

This shows up hardest with RAG. Retrieval can return the right facts in the wrong field, the wrong order, or a slightly different shape. The prompt is fine. The thing feeding it isn't.

Integration tests catch that boundary.

The suite is the team's memory

Ten examples by hand is easy at the start.

Months later those ten are fifty. A product manager found an edge case. Production failed. The model did something unexpected.

Nobody remembers all of them. People leave. The failures stay.

Every meaningful failure should become a test. Over time the suite is the record of what the system has to handle.

That's especially true when prompts live outside the repo, in tools like Langfuse. Changing a prompt gets easy. Accidentally breaking something that used to work gets easy too.

Model changes are regressions waiting to happen

Even models from the same provider can disagree with the same prompt.

Treat a model bump the way you'd treat an important dependency bump.

Run the same cases against the new model before you switch. The question isn't whether the new model is generally better. It's whether it still does what this application depends on.

Don't assert the wording

LLM integration tests should not check exact prose.

For structured output, assert what the application actually uses: required fields, allowed values, ranges, classifications, relationships between fields, business rules.

For looser output, semantic or model-based evaluation can help, as long as the tests tolerate some drift.

You're testing behavior, not copy.

Test the pipeline, not the prompt

Prompt-level tests are useful while you're writing the prompt.

Once the feature is in a product, the question isn't:

"Does this prompt work?"

It's:

"Does the system still work with this prompt, this model, this retrieval result, and this application code together?"

That's what integration tests are for.

If nobody wants to touch the prompt for fear of breaking something else, the testing strategy is already behind.

Need help?

If an LLM feature is too fragile to change, I can help. The usual fix isn't a bigger playground. It's a pipeline-level suite that catches the next prompt edit, model bump, or retrieval change before production does.

I wrote the ideas and the substance of this article. AI helped with proofreading and formatting. The technical details were accurate at publication.