# Monitoring LLM performance and latency in production [Skip to content](#lm-inhoud)Network/NL[EN](/en/)[Hubhub.llmnet.nlCompare models by task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs robust in software: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlIntroducing AI in an organization, from pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlDevelopments in AI, interpreted for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, for your own tasks.](https://benchmark.llmnet.nl/en/)[Jobsvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, from beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for people who build their own.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/)[](https://x.com/intent/post?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fmonitoring-en-latency-van-taalmodellen-in-productie-bewaken&text=LLM-Monitoring%20en%20Latency%20in%20Productie%20Bewaken)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fmonitoring-en-latency-van-taalmodellen-in-productie-bewaken)[](https://www.reddit.com/submit?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fmonitoring-en-latency-van-taalmodellen-in-productie-bewaken&title=LLM-Monitoring%20en%20Latency%20in%20Productie%20Bewaken)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fmonitoring-en-latency-van-taalmodellen-in-productie-bewaken&text=LLM-Monitoring%20en%20Latency%20in%20Productie%20Bewaken)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fmonitoring-en-latency-van-taalmodellen-in-productie-bewaken)[](https://www.reddit.com/submit?url=https%3A%2F%2Fconsultancy.llmnet.nl%2Fmonitoring-en-latency-van-taalmodellen-in-productie-bewaken&title=LLM-Monitoring%20en%20Latency%20in%20Productie%20Bewaken)[](#) # Monitoring latency and language models in production By Ivo Donker — compiled with AI assistance (Claude & Gemini) · August 20, 2026 When a language model makes the move from an isolated experimental environment to a business-critical production environment, the management question changes drastically. Where traditional microservices show predictable response times in milliseconds and fail unambiguously through HTTP status codes, large language models (LLMs) introduce a dynamic of asynchronous generation, non-deterministic answers, variable queue times, and complex dependencies. Slow prompt processing does not merely frustrate end users; it also disrupts automated back-office flows, drives up operational compute costs, and masks creeping quality faults. Monitoring production models reliably therefore calls for a specific observability architecture. In this dossier we work through measurement methods for the various latency dimensions, the analysis of tail latency and outliers, distributed tracing for composite pipelines, detection of semantic quality drift, and the design of sharp Service Level Objectives (SLOs). The goal is an operational management chain with which teams spot and repair disruptions proactively, before end users are affected. ## The anatomy of LLM latency: four measurable phases In classic web architectures a single central performance metric often suffices: the total round-trip time of an HTTP request. With language models, however, such a single number conceals the real bottlenecks. A call to a model consists of two fundamentally different compute steps on the GPU: the prefill phase (in which the input prompt is processed in parallel and converted into key-value pairs in GPU memory) and the decode phase (in which new tokens are generated sequentially, one by one). To optimize latency in a targeted way, four specific metrics have to be recorded continuously. Metric | Definition | Primary influencing factors | Target value in production | TTFT (Time to First Token) | The time between sending the request and receiving the very first generated token. | Length of the input prompt, document injection in RAG, network latency to the inference cluster, GPU KV cache state, and any queue time. | Interactive: < 800 msAsynchronous: < 3,000 ms | ITL (Inter-Token Latency) | The average time between two consecutive generated tokens during an active stream. | Model size (number of parameters), degree of quantization (e.g. FP16 vs INT4), GPU memory bandwidth, and concurrency on the same inference engine. | Interactive: 20 – 50 ms(corresponding to 20–50 tokens/sec) | TPS (Tokens Per Second) | The total number of generated output tokens divided by the total generation time after the first token has been received. | Hardware architecture (tensor cores), batch size of the inference server, and optimizations such as speculative decoding or vLLM PagedAttention. | Single: 30 – 80 tpsBatch aggregate: 500+ tps | E2E (End-to-End Latency) | The total processing time from sending the initial request to receiving the very last token and closing the connection. | The sum of TTFT and (number of generated tokens × ITL), plus any overhead from parsing, safety guards, and network serialization. | Depends on the required output length and task complexity. | Operational priority differs strongly per type of application. With interactive customer contact systems or AI writing aids, the user experiences the system as immediately responsive as soon as the first words appear on screen within a second and the ITL keeps pace with normal reading speed. With background processing — such as extracting contract data from thousands of PDF files overnight — TTFT is by contrast secondary, and the operations team steers primarily on maximum aggregate throughput (TPS) at minimum cost per token. ## Dissecting tail latency: why p95 and p99 explode in inference Reporting only average response times (the p50 or median) creates a dangerous blind spot in AI operations. LLM inference naturally shows a long-tailed distribution (tail latency). An application can show a respectable average response time of 1.5 seconds while the 95th percentile (p95) climbs to 8 seconds and the 99th percentile (p99) hits 25 seconds. For business-critical processes that means one in twenty transactions stalls or times out. In practice, extreme latency spikes arise through three specific technical mechanisms: - Memory pressure and KV cache fragmentation: Modern inference servers store intermediate results from the attention mechanisms in a key-value memory (KV cache). When GPU memory fills up under peak traffic, active requests have to be paused (preemption) or moved to slower RAM, which leads to abrupt delays. - Queue build-up at the API gateway: Both with commercial cloud providers and on internal clusters, congestion appears as soon as the number of concurrent requests exceeds the available compute cores. Requests sit in a buffer for seconds before the prefill phase even starts. - Unbounded generation loops: When a prompt is insufficiently bounded or when the model gets caught in repeating patterns, generation continues until the absolute token limit of the context window is reached. That drives both E2E latency and token usage up needlessly. To neutralize these outliers in the tail distribution effectively, engineering teams can apply hedging patterns. The article on [hedged requests against tail latency](https://api.llmnet.nl/en/hedged-requests-inzetten-om-llm-tail-latency-te-verlagen) explains in detail how redundant parallel requests bypass slow server responses automatically. ## Distributed tracing and span instrumentation for LLM chains A modern generative AI application rarely talks to just one model. A common pipeline runs a semantic vector search, reorders documents (reranking), injects context through prompt templates, executes a model call, and validates the resulting JSON output against a data model, one after the other. When an end user experiences a slow response, the monitoring platform has to point immediately to which link in the chain is failing. Using OpenTelemetry standards and semantic conventions for generative AI, distributed traces are built in which every subtask is recorded as a separate measurement span. That keeps operations teams from groping in the dark about whether the database, the network, or the inference engine caused the delay. # Gestructureerde span-attributen volgens OpenTelemetry GenAI-conventies { "trace_id": "8a3f9e2b1c4d0e7f8a9b0c1d2e3f4a5b", "span_id": "c1d2e3f4a5b6c7d8", "parent_span_id": "00f067aa0ba902b7", "name": "gen_ai.client.generate_content", "attributes": { "gen_ai.system": "anthropic", "gen_ai.request.model": "claude-3-5-sonnet", "gen_ai.request.temperature": 0.1, "gen_ai.request.max_tokens": 1024, "gen_ai.usage.input_tokens": 2840, "gen_ai.usage.output_tokens": 192, "gen_ai.response.finish_reasons": ["stop"], "llm.latency.ttft_ms": 420.5, "llm.latency.total_duration_ms": 1380.2, "llm.cache.hit": true, "rag.retrieval_duration_ms": 85.4, "rag.documents_retrieved": 5 } } By recording these spans structurally, an administrator can analyze immediately whether an increase in latency was caused by retrieving too many context documents from the vector database or by server congestion at the model provider. See the step-by-step guide to [an observability dashboard for LLM calls](https://api.llmnet.nl/en/een-observability-dashboard-voor-je-llm-calls) for instructions on configuring OpenTelemetry collectors and visualization dashboards. ## Quality monitoring and detecting semantic drift A model that generates an unusable answer at blazing speed performs flawlessly technically but fails functionally. In production environments, quality degradation regularly occurs without any traditional error codes being generated. This phenomenon is known as drift and has two main causes: changes in model behavior after provider updates (model drift) and shifts in the language or intent of users (data drift). Detecting quality deterioration requires automated, asynchronous evaluation streams that run continuously on production data: - LLM-as-a-Judge sampling: An independent, advanced evaluation model continuously analyzes a representative sample (for instance 2% to 5% of all incoming requests). It tests the output for groundedness (is the answer traceable to the context supplied?), relevance, and concision. - Schema conformity validation: When a model produces structured data (such as JSON or XML), the monitoring system records every parse error or field violation. A sudden rise in the percentage of validation errors signals immediately that the model is following instructions less strictly. - Embedding distance and domain shift: By comparing text embeddings of user questions with historical clusters, the system signals when users suddenly ask about topics the model was not instructed for. For an in-depth analysis of statistical methods and evaluation frameworks, the benchmark dossier on [measuring drift in production](https://benchmark.llmnet.nl/en/drift-meten-in-productie-modelprestaties-over-tijd-volgen) offers detailed formulas and reference architectures. Quality standards should moreover be secured up front with automated test suites. Consult the step-by-step guide to [acceptance tests for non-deterministic output](https://consultancy.llmnet.nl/en/acceptatietests-inrichten-voor-niet-deterministische-output) to draw up robust acceptance criteria before a release to production. ## Error margins, HTTP status codes, and automated fallbacks Depending on external AI infrastructure means applications have to be able to handle temporary outages, rate limits, and capacity problems. Categorizing errors forms the basis for targeted automatic recovery actions (self-healing patterns). Error category | Symptom / code | Direct operational impact | Automated mitigation | Rate limiting | HTTP 429 (Too Many Requests / TPM / RPM) | Temporary blocking of new requests during traffic peaks. | Exponential backoff with random jitter; dynamic switching to a secondary API key or fallback provider. | Context length exceeded | HTTP 400 (Context Window Exceeded) | The request fails permanently because input documents are too large. | Automatic prompt compression, shortening context chunks, or switching to a model variant with a larger context window. | Provider server outage | HTTP 500, 502, 503, 504 | Structural or incidental unavailability of the inference host. | Immediate routing through a multi-model gateway with a built-in circuit breaker to an alternative cloud region or vendor. | Safety & content filters | Output refusal / model filter triggers | The model refuses to formulate an answer despite legitimate context. | Input sanitization, adjustment of system instructions, or local validation up front to catch false positives. | When a failure cannot be resolved automatically by the gateway, the operations team has to know immediately which escalation lines apply. See the guidance on [an incident protocol for language model errors](https://consultancy.llmnet.nl/en/een-incidentenprotocol-opstellen-voor-fouten-door-taalmodellen) for formally recording roles, notification flows, and recovery procedures. ## Setting up token allocation and financial observability With traditional server software, operational costs are largely static and tied to fixed hardware instances. With language models, costs are by contrast variable and tied directly to the number of tokens processed in both the prompt and the generated answer. Without active financial monitoring, a programming error in an autonomous agent loop or an uncontrolled batch job can burn through thousands of euros of API credit within a few hours. Effective monitoring therefore ties token statistics directly to organizational entities: - Allocation per feature and tenant: Break token usage down per API key, department, or specific use case (e.g. email summarization vs. code analysis) to gain insight into the marginal cost per business transaction. - Monitoring cache efficiency: Record the ratio of cached prompt tokens to uncached tokens. A drop in the cache hit ratio points to dynamic changes in prompt prefixes that raise cost and latency needlessly. - Automated budget ceilings: Set hard thresholds at hourly, daily, and monthly level. On exceeding 80% of expected daily usage the system sends a warning; at 100% it switches automatically to stricter rate limits or cheaper models. The dossier on [controlling unexpected operational costs of language models](https://consultancy.llmnet.nl/en/onverwachte-operationele-kosten-van-taalmodellen-beheersen) covers concrete calculation methods, cost allocation models, and savings techniques extensively. ## Service Level Objectives (SLOs) and alerting Setting up alerts requires careful balance. Thresholds that are too tight lead to alert fatigue among operations teams, while tolerances that are too loose result in unnoticed outages. Steering on Service Level Indicators (SLIs) and concrete Service Level Objectives (SLOs) creates a transparent management framework. Domain | Service Level Indicator (SLI) | Production SLO target | Alerting condition (pager / alert) | System availability | Percentage of successful requests without uncaught 5xx or 429 errors. | 99.5% availability measured over a rolling window of 30 days. | Error rate > 1% over a period of 5 minutes or 3 consecutive circuit breaker activations. | Interactive responsiveness | p95 Time to First Token (TTFT) for synchronous user queries. | p95 TTFT < 1,200 ms under normal load. | p95 TTFT > 2,500 ms for more than 10 consecutive minutes. | Streaming stability | p95 Inter-Token Latency (ITL) during text rendering. | p95 ITL < 40 ms per token. | p95 ITL > 80 ms for 5 minutes (results in stuttering on-screen rendering). | Schema integrity | Percentage of answers that pass automated JSON schema validation. | > 99.8% error-free payload structures. | Validation errors > 1% in a rolling window of 100 transactions. | The operations team should not only respond to acute incidents but also watch the error budget burn rate. When a slight drop in quality or rise in latency consumes the monthly error budget too fast, planned model updates are frozen immediately until the cause has been removed. ## Synthetic monitoring and proactive health probes Passive monitoring — analyzing incoming user traffic — only flags problems once real users have already run into them. It also gives a distorted picture at night or during off-peak hours because of low transaction volume. Synthetic monitoring solves this by continuously sending automated test requests through the entire chain. A well-considered synthetic measurement setup runs three types of probe every five minutes: - The deterministic baseline probe: A fixed prompt with an expected, unambiguous output. This test verifies that the API gateway, authentication, and basic inference are operational. - The RAG integration probe: A query that has to retrieve and summarize a specific test document from the vector database. This checks that the semantic index and context injection are functioning correctly. - The structured output probe: A task that forces a complex JSON structure. This verifies that the parser and any function models have remained intact after a provider update. By plotting the response times and success rates of these synthetic probes separately from production traffic, network degradations and provider outages become visible before the first employee or customer logs in. ## Ownership and operational management after go-live A technically advanced monitoring dashboard has little operational value if the organization has not established who is responsible for interpreting and following up on signals. Monitoring language models touches three different disciplines: software engineering (for latency, caching, and network errors), data science / AI engineering (for model drift, prompt optimization, and evaluation scores), and the functional process owner (for substantive correctness and process costs). Consult the organizational framework on [management after go-live and ownership of AI systems](https://consultancy.llmnet.nl/en/beheer-na-go-live-wie-is-eigenaar-van-een-ai-toepassing-in-productie) to assign tasks, management roles, and decision-making authority structurally within the organization. ## Operational checklist for go-live Before opening an AI application to large-scale end-user traffic, the checklist below helps verify that all management layers are operationally in place: Checkpoints for reliable LLM observability: - Latency breakdown enabled: TTFT, ITL, and E2E latency are recorded separately in the metrics platform. - Distributed tracing set up: Every model call carries trace IDs with metadata on token counts, model versions, and sub-span response times. - Fallback and circuit breaker logic tested: When simulating HTTP 429 and 503 errors, the gateway switches to a backup model or secondary region within 500 ms. - Financial alerts and budget ceilings active: Automated notifications are configured for unusual token peaks, along with hard spending limits per API key. - Synthetic health checks operational: Periodic test transactions check the full chain from vector storage to structured output 24/7. - Drift and quality measurement in place: A periodic sample evaluation (LLM-as-a-Judge or schema check) runs to flag semantic quality deterioration early. By integrating monitoring, latency control, and quality assurance into the software architecture from the design phase onward, organizations build a robust foundation for scalable, cost-controlled, and reliable AI services.