# Acceptance tests for non-deterministic AI output

[Skip to content](#lm-inhoud)Network/[NL](/en/acceptatietests-inrichten-voor-niet-deterministische-output)EN[Hubhub.llmnet.nlCompare models on task, language, cost and licence.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organisation, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fen%2Facceptatietests-inrichten-voor-niet-deterministische-output&text=Acceptance%20tests%20for%20non-deterministic%20AI%20output)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fen%2Facceptatietests-inrichten-voor-niet-deterministische-output)[](https://www.reddit.com/submit?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fen%2Facceptatietests-inrichten-voor-niet-deterministische-output&title=Acceptance%20tests%20for%20non-deterministic%20AI%20output)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fen%2Facceptatietests-inrichten-voor-niet-deterministische-output&text=Acceptance%20tests%20for%20non-deterministic%20AI%20output)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fen%2Facceptatietests-inrichten-voor-niet-deterministische-output)[](https://www.reddit.com/submit?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fen%2Facceptatietests-inrichten-voor-niet-deterministische-output&title=Acceptance%20tests%20for%20non-deterministic%20AI%20output)[](#)

 
# Setting up acceptance tests for non-deterministic output

 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 Software development has traditionally leaned on determinism: the same input under identical conditions always produces exactly the same binary output. Traditional test suites check whether a variable yields the value true , whether a database query returns exactly seven rows and whether an API endpoint returns status code 200 . As soon as generative AI systems or large language models become part of the application logic, that certainty disappears. A model phrases its answers slightly differently every time, picks varying synonyms and shows subtle variation in sentence structure, even when the underlying parameters are set strictly.

 Anyone who tries to set up acceptance tests using classic assertions runs into brittle tests that fail at random or offer false confidence. Operationalizing language models requires a fundamental shift from binary checks to statistical, heuristic and semantic quality measurements. In this article we cover how to build a robust acceptance framework that makes non-deterministic output verifiable, testable and reliable enough to move to production.

 
## Why classic unit tests fail with language models

 In classic test automation a test usually works with an exact string comparison: assert response.body == expected_output. With language models this principle fails immediately. Even when a system instruction asks for a concise summary of a policy document, one call may start with “The insured is entitled to...” and the next with “The terms show that the coverage applies to...”. Both answers are correct in substance, but an exact string comparison fails right away.

 Reducing randomness through parameters only helps up to a point. For a deeper understanding of how model variables such as temperature and top-p work, the explanation of [sampling parameters and their influence on token choice](https://leren.llmnet.nl/en/sampling-parameters). Even at a temperature of zero, a model stays non-deterministic across different hardware clusters, model versions or optimized floating-point implementations. Acceptance tests should therefore not check the exact letters, but the structural validity, factual consistency and semantic meaning of the generated text.

 
## The three levels of acceptance: deterministic, heuristic and semantic

 An effective test strategy splits quality checks into three consecutive layers. By ordering tests from cheap and hard to more expensive and probabilistic, you avoid unnecessary evaluation costs and keep the test suite fast.

 The first layer is deterministic validation. This checks whether the output meets strict structural requirements, such as a valid JSON structure, the presence of mandatory fields or staying within a maximum length. Read more about enforcing machine-readable answers in the article on [structured output and JSON schema enforcement](https://api.llmnet.nl/en/structured-output). To test automated validation rules up front in a browser environment, the [JSON Schema output validator and benchmark tool](https://benchmark.llmnet.nl/en/tool-json-schema-validator) offers direct verification of schema conformity. If a model does not produce valid JSON while the system requires it, no further evaluation of the content is needed: the test fails immediately.

 The second layer consists of heuristic checks. These are fast, rule-based tests that look for known patterns. Think of regular expressions (regex) to verify that citizen service numbers, credit card details or inappropriate terms are absent. Measuring the share of keyword overlap between the source file and the generated answer also falls into this category.

 The third layer is semantic evaluation. Here the test system checks the factual correctness, tone and relevance of the answer against a gold standard or reference context. Because this cannot be done with fixed rules, it happens through embedding distances or a secondary evaluation model.

 
 
 
 
 Test level | 
 Method | 
 Speed & cost | 
 Typical use | 
 

 
 
 
 1. Deterministic | 
 JSON Schema, regex, type checking | 
 < 1 ms (no API costs) | 
 Payload validation, format check | 
 

 
 2. Heuristic | 
 Length limits, keyword matching, blacklist checks | 
 < 5 ms (no API costs) | 
 PII detection, missing disclaimers | 
 

 
 3. Semantic | 
 Embeddings (cosine similarity), LLM-as-a-Judge | 
 200 ms – 2000 ms (API costs) | 
 Factual accuracy, hallucination detection | 
 

 
 
 

 
## Building fixed test samples and golden datasets

 No probabilistic test suite can function without a representative reference collection, often called a golden dataset. Such a dataset consists of at least fifty to a few hundred carefully selected input variants with matching reference answers, acceptance criteria and known edge cases.

 Building this dataset starts with domain experts. They decide which questions are crucial for the business and which errors are unacceptable. A good dataset contains four types of test cases:

 
 
- Standard scenarios (happy path): Common prompts with unambiguous context and clear answers.
 
- Edge cases: Complex, ambiguous or contradictory documents where the model has to admit that information is missing.
 
- Adversarial prompts: Attempts at prompt injection, questions outside the domain or attempts to extract internal system instructions.
 
- Format extremes: Very short input, exceptionally long input and input with typos or unusual formatting.
 

 The quality of the test suite depends directly on the quality of the underlying data. For an overview of how data sources are cleaned and prepared, see the article on [data quality for AI and preventing failed pilots](https://consultancy.llmnet.nl/en/datakwaliteit-voor-ai). Without clean source data you are testing noise against noise.

 
## LLM-as-a-Judge: patterns, pitfalls and calibration

 For semantic acceptance criteria — such as “is the answer factually supported by the source provided?” or “is the tone professional and neutral?” — an advanced language model is increasingly used as the assessor (the so-called LLM-as-a-Judgepattern). The evaluation model is given a strict rubric and grading instruction.

{
 "evaluatie_instructie": "Beoordeel of de gegenereerde samenvatting uitsluitend feiten bevat uit de brontekst.",
 "criteria": {
 "volledigheid": "Score 1-5: Bevat de tekst alle hoofdpunten?",
 "feitelijkheid": "Score 1-5: Bevat de tekst beweringen die NIET in de bron staan?",
 "beknoptheid": "Score 1-5: Is de tekst vrij van overtollige herhaling?"
 },
 "uitvoerformaat": {
 "type": "object",
 "properties": {
 "score": { "type": "integer" },
 "motivering": { "type": "string" },
 "bevat_hallucinatie": { "type": "boolean" }
 },
 "required": ["score", "motivering", "bevat_hallucinatie"]
 }
}

 Although this pattern makes scalable quality checks possible, it has considerable pitfalls. Evaluation models have a built-in preference for longer texts (verbosity bias), side more often with answers generated by the same model (self-enhancement bias) and struggle to score consistently on subtle semantic nuances.

 To make this method reliable for an acceptance test, calibration against human assessors is mandatory. Take a sample of at least a hundred evaluations and compare the model's judgment with that of two human experts. Keep adjusting the prompts and scoring rubrics until the correlation (measured for example with Cohen's kappa) is at least 0.80. Only once that threshold is met may the evaluation model decide on an acceptance gate on its own.

 
## Statistical thresholds and metrics: pass@k and semantic overlap

 Because a single run of a test case can pass or fail through randomness, acceptance has to be approached statistically. Instead of a binary “pass / fail” outcome per test case, the test suite runs multiple iterations per prompt and calculates a pass rate across the entire golden dataset.

 Commonly used metrics within this framework are:

 
 
- Pass@k: The percentage of test cases where at least one of the k generated answers fully meets the acceptance criteria. For business-critical processes without a human check in between, teams instead use Pass^k (all k attempts must pass, to prove that the system is consistently stable).
 
- Embedding cosine similarity: The vector distance between the generated answer and the ideal reference answer. A threshold of 0.88 to 0.92 usually indicates strong semantic agreement, provided the embedding model is sensitive to negations.
 
- RAG triad metrics: Context Relevance (is the retrieved documentation relevant to the question?), Groundedness (can the answer be traced back to the context?) and Answer Relevance (does the output answer the actual user question?).
 

 Before a rollout to production, the product owner and the audit team set the minimum thresholds. For the formal transition from a trial setup to a live environment, the step-by-step guide on [evaluating a PoC with clear go/no-go criteria](https://consultancy.llmnet.nl/en/een-poc-evalueren-criteria-voor-go-no-go) helps to record these hard thresholds contractually.

 
## Regression tests and automated CI/CD integration

 A crucial goal of acceptance tests is preventing quality regression. With LLM systems, regression does not only occur on code changes in the application, but also when the model vendor rolls out an update, when the system prompt is tightened or when the chunking strategy of a vector database changes.

 An effective regression pipeline in continuous integration (CI) runs in phases:

 
 
- Fast pre-commit suite (deterministic & heuristic): 50 synthetic tests check JSON schemas, regex validation and latency within seconds on a local mock or fast model variant.
 
- Nightly evaluation suite (semantic & golden dataset): The full golden dataset is run through the target system. Evaluation models calculate scores and compare the aggregated results with the baseline of the previous release.
 
- Diff analysis: If overall accuracy stays the same (94%, for example), but specific edge cases that passed yesterday suddenly fail today, the system flags a silent regression and the build is blocked.
 

 Setting up such evaluations systematically forms the foundation of reliable release management. How this fits into the overall life cycle of software implementations is covered in the overview on [from pilot to production: avoiding pitfalls in AI projects](https://consultancy.llmnet.nl/en/pilot-naar-productie).

 
## Analyzing and categorizing failing tests (error analysis matrix)

 When a non-deterministic test fails, the cause is rarely obvious. Is it a faltering retrieval step, a hallucination by the model, an overly strict evaluation prompt or an outdated reference answer in the test suite? Without structured categorization, teams get bogged down in endless discussions about individual test cases.

 
 
 
 
 Error category | 
 Symptom | 
 Cause | 
 Solution | 
 

 
 
 
 Retrieval error | 
 Groundedness is low; model gives a generic answer | 
 Relevant source documents are missing from the top-k search results | 
 Adjust chunking, add hybrid search (BM25 + vector) | 
 

 
 Model hallucination | 
 Groundedness is low; answer contains invented details | 
 Model ignores context when no clear instruction is given | 
 Tighten the system prompt with negative restrictions | 
 

 
 Format error | 
 JSON parsing fails; markdown tags in payload | 
 Model loses structure on long context or special characters | 
 Force JSON mode / response_format through API parameters | 
 

 
 Evaluator bias | 
 Human approves, LLM judge rejects (or the other way around) | 
 Evaluation rubric is too vague or the model has a length preference | 
 Refine the rubric with few-shot examples; recalibrate | 
 

 
 Outdated benchmark | 
 Model gives the correct new policy, test expects the old policy | 
 Golden dataset has not been updated after a policy change | 
 Tie dataset version control to releases of company documents | 
 

 
 
 

 
## Governance and auditing of test results

 Acceptance testing does not only serve to reassure developers; it is also a legal and internal accountability record. In regulated sectors, an organization has to be able to demonstrate that an AI system was systematically checked for accuracy and bias before it went into use. For the policy preconditions, see the overview on [AI governance for SMEs and responsible implementation without bureaucracy](https://consultancy.llmnet.nl/en/ai-governance-mkb).

 
## Continuous quality monitoring in production

 Acceptance tests do not stop the moment a release is pushed to production. Because user questions in the real world change dynamically and models are subject to shifts in context, continuous monitoring on production data is necessary.

 Because it is too expensive to run every live interaction through a heavy evaluation model, production architectures work with statistical sampling. A fixed percentage of interactions (2% to 5%, for example) is forwarded asynchronously to an evaluation queue. The same semantic checks as in the CI/CD test suite run on it. If the trend over a 24-hour period deviates significantly from the acceptance values, the monitoring system raises an alert.

 For a detailed assignment of tasks and ownership around this quality control after go-live, see the file on [management after go-live and operational ownership of AI systems](https://consultancy.llmnet.nl/en/beheer-na-go-live-wie-is-eigenaar-van-een-ai-toepassing-in-productie).

 
## A practical acceptance framework for deployment

 To determine whether a model configuration or prompt change is ready for deployment, you can work through the decision tree below. Each level acts as a hard gate: only once a level passes is the next level evaluated.

 
 
- Level 1: Format & safety (100% required)
 
 Do 100% of the test cases meet the JSON schema or the agreed structure?
 
- Is all output 100% free of detected PII and blocking terms via regex/heuristics?
 
 
 
- Level 2: Statistical semantic thresholds (dataset-wide)
 
 Does the golden dataset reach the minimum threshold for Groundedness (for example ≥ 95%)?
 
- Is the average embedding cosine similarity above the set baseline (≥ 0.90)?
 
- Is there no regression on critical edge cases compared with the previous release?
 
 
 
- Level 3: Performance and cost limits
 
 Does p95 latency stay within the agreed service level (for example < 2.5 seconds)?
 
- Does token usage per transaction stay within the budgeted maximum?
 
 
 

 By automating this three-stage acceptance and recording it in code, you turn the unpredictability of language models into a manageable, measurable and auditable software engineering process. That makes non-deterministic output just as reliable for business-critical applications as traditional deterministic software.