AI engineer interviews
LLM system design interview questions
An LLM system design interview asks you to plan a product built on a language model: a support assistant, a search feature, a coding helper. The model is only one part. Most of the hour goes to speed, cost and what happens when something fails.
What they test
What interviewers look for.
- You ask for the numbers first: how many requests, how fast the answer must come, how much each one may cost.
- You know where the time goes. You stream the answer, cache what repeats and run slow steps side by side.
- You know where the money goes. Shorter prompts, a smaller model for easy requests and caching all cut the bill.
- You plan for failure: timeouts, a backup model, and a safe answer when nothing works.
- You say how you would measure quality before and after launch.
Questions
Speed questions.
How to make an LLM feature feel fast, and how to find what makes it slow.
Design an assistant that starts responding quickly while searching documents. A complete answer can take five seconds. How would you improve perceived latency without streaming unsupported claims?
A good answer covers: separate time to first useful output from total completion latency; gate factual answer generation on sufficient retrieved evidence; support cancellation and measure latency distributions with answer quality.
Your async server uses Python's asyncio event loop, but some requests stall for several seconds. Investigation shows a blocking network client is called during request handling. How would you diagnose and fix this issue?
A good answer covers: Blocking calls block the entire event loop, stalling all coroutines; Identify and isolate blocking code paths in async handlers; Replace blocking clients with async-compatible or run in executor.
After a cache miss, your AI service receives a burst of identical requests, all querying the same backend. What concrete mechanism would you add to prevent backend overload and reduce redundant work?
A good answer covers: Single-flight or request coalescing to deduplicate backend calls; Locking or coordination on cache miss; Verification that only one backend call occurs per cache miss.
Your system retries API requests after timeouts, but sometimes the original request is still in flight. What design would you use to avoid duplicate work and wasted provider capacity?
A good answer covers: Detect and cancel in-flight requests before retrying; Use unique request IDs to deduplicate at provider; Verify no duplicate processing occurs during retries.
An audio streaming pipeline starts lagging and eventually crashes with memory errors. You discover that the audio buffer grows without bounds when downstream playback is slow. What would you check to diagnose why backpressure is not limiting the buffer size?
A good answer covers: Mechanism for monitoring buffer occupancy; Backpressure signaling from playback to upstream; Enforcing buffer size limits with flow control.
An approximate nearest neighbor (ANN) search API is combined with a selective filter. Users report that too few results are returned, even when more matches exist in the database. What concrete change would you make to address this?
A good answer covers: Candidate pool size tuning; Filter application order; Recall vs. precision tradeoff.
A retrieval service has a default timeout of 5 seconds, but the end-to-end user request deadline is only 2 seconds. How would you diagnose and verify the impact of this timeout mismatch?
A good answer covers: Compare component timeouts to overall deadline; Trace request path to find bottlenecks; Simulate requests to observe late completions.
Your assistant responds in two seconds on average, but one in twenty requests takes fifteen seconds. How would you investigate?
A good answer covers: trace latency per stage and inspect p95 or p99; separate queueing, retrieval and generation time; set timeouts, budgets and measured fallbacks.
Your service issues parallel LLM requests, but sometimes gets errors about exceeding a provider's token-per-minute quota. How would you diagnose and verify the cause, and what concrete change could you make?
A good answer covers: Instrument and sum total tokens used per minute; Correlate errors with request/token volume; Implement global token rate limiting across requests.
You are designing a server-sent events (SSE) client that parses streamed JSON messages. Sometimes, the parser treats incomplete JSON fragments as complete, causing errors. What design change would you make to robustly handle partial messages?
A good answer covers: Buffer incoming data until a full JSON object is detected; Validate JSON completeness before parsing; Handle fragmented messages across multiple SSE events.
While building a new embedding index, you observe that live query latency spikes and sometimes times out. How would you isolate the source of contention and verify whether index building is affecting live queries?
A good answer covers: Resource profiling and monitoring; Workload isolation or throttling; Correlation of build activity with query latency.
Your embedding service batches incoming requests to improve throughput. However, users with infrequent queries experience long delays. What concrete design change would you make to improve latency for low-volume traffic?
A good answer covers: Adaptive or timeout-based batching; Latency vs. throughput tradeoff; Early flush for small batches.
After users start attaching longer documents, time to first token rises sharply while time between subsequent tokens stays roughly unchanged. How would you locate the bottleneck before choosing an optimization?
A good answer covers: separate queueing and prompt prefill from autoregressive decode in the trace; compare prompt length and concurrency against prefill duration and queue wait rather than blaming decode alone; measure output quality alongside changes such as reducing redundant context, prefix caching or scheduling prefill work.
After weight quantization, a model loads comfortably on one GPU. It still runs out of memory when several users submit long prompts. Explain what memory you would measure and how you would bound capacity.
A good answer covers: distinguish model weights from per-sequence KV cache, activations and runtime workspace; measure memory versus concurrent sequence lengths and inspect the actual KV cache dtype and allocation policy; set admission or token limits from measured headroom and compare cache quantization or offload with quality and latency costs.
Your assistant receives a burst of ten times normal traffic and its model provider has a fixed concurrency limit. Design overload handling while preserving useful service.
A good answer covers: apply admission control bounded queues and request deadlines; prioritize or degrade work with explicit user-visible failure modes; measure queue time provider latency and saturation under realistic load.
Your streaming API disconnects a client, but the upstream text generation job continues running, consuming resources. How would you confirm this leak, and what safeguards would you add to prevent such orphaned jobs?
A good answer covers: Trace upstream job lifecycle after client disconnect; Log or monitor generation job termination on disconnect; Implement cancellation or cleanup hooks on disconnect event.
Your voice assistant streams partial responses as they are generated. Sometimes, the model later refuses the request, but the user has already heard misleading partial output. What system design changes could reduce user confusion in this scenario, and how would you verify users are less misled?
A good answer covers: Buffering or gating early outputs until intent is clear; Mechanism for retracting or correcting misleading audio; Verification with user feedback or confusion metrics.
A serving team proposes speculative decoding with a small draft model to reduce response latency. Explain the mechanism and design an experiment that can tell whether it helps your workload.
A good answer covers: describe drafting several tokens and verifying them with the target model, with rejection or correction rather than unconditional acceptance; measure accepted draft tokens, drafting and verification overhead across representative request types; compare end-to-end latency, throughput and distribution or quality guarantees under the chosen algorithm and decoding settings.
A model fits on one GPU, but the team wants to use a second GPU to reduce interactive latency. Compare tensor parallelism with two replicas and propose a benchmark to choose between them.
A good answer covers: distinguish splitting one model's computation across devices from serving independent requests on replicas; account for inter-device communication, batch size and whether the objective is per-request latency or throughput; compare tail latency, throughput and memory under representative concurrency on the actual interconnect.
You are designing a streaming voice assistant that must detect when a user's turn is finished before responding. Sometimes, users pause to think, but the system treats these as turn completions, interrupting them. Under strict latency and user experience constraints, what design could reduce these errors, and how would you verify it works for real users?
A good answer covers: Dynamic adjustment of pause thresholds based on context; Incorporating prosody or semantic cues for turn detection; User feedback or correction mechanisms for false positives.
A search system's full-text query suddenly causes high latency, with logs showing a scan of nearly all documents. Indexes appear healthy and the query is not obviously malformed. How would you isolate the failure's root cause?
A good answer covers: Examine query plan to confirm index usage; Check for query patterns bypassing indexes; Validate index coverage for all query fields.
Questions
Cost questions.
How to keep the price per request under control without hurting quality.
Design context management for a chat assistant whose costs grow with conversation length. Users expect it to remember decisions from earlier turns.
A good answer covers: retain explicit user decisions and source references in structured state; budget recent turns retrieved evidence and output tokens separately; test recall contradictions and task quality after compression.
Your LLM provider offers a prompt cache: cached input tokens are billed at 50% the normal rate, not zero. Design an accounting system to ensure users are charged the correct rate for both cached and uncached tokens on each request.
A good answer covers: Identify and separate cached vs. uncached tokens per request; Apply correct billing rate to each token type; Aggregate and record mixed-rate token charges for user billing.
A model router picks which model to use based on each model's self-reported confidence score. There is no calibration. What are the risks of this approach, and how would you improve reliability?
A good answer covers: Self-reported confidence can be poorly calibrated or misleading; Calibration aligns reported confidence with actual accuracy; Alternative routing signals (e.g., validation set performance) can improve model selection.
Your LLM bill triples after a prompt update while traffic stays flat. How would you find and reduce the cause?
A good answer covers: measure tokens and calls per completed task; inspect growing context, retries and model routing; compare quality before accepting a cheaper configuration.
Design routing between a cheap model and an expensive model for support questions. The goal is lower spend while preserving quality for difficult cases.
A good answer covers: define a measurable quality constraint and cost baseline by task slice; route or escalate using calibrated signals rather than self-reported confidence alone; evaluate routing errors latency and total cost including retries.
A per-user token quota is enforced by a check before each request. However, some users exceed their quota by submitting many requests in parallel. How would you diagnose and fix this vulnerability to ensure quotas cannot be bypassed?
A good answer covers: race conditions in quota enforcement; need for atomic or transactional quota checks; testing with concurrent requests for bypasses.
A quota system estimates token usage using a different tokenizer than the deployed model, sometimes allowing requests that exceed the model’s real limits. How would you diagnose and resolve this issue?
A good answer covers: Compare tokenizer outputs for sample inputs; Align quota checks with model’s tokenizer; Test boundary cases for over-limit errors.
You cache LLM answers for 24 hours to reduce costs. However, some knowledge (e.g., sports scores) changes hourly. How would you design the caching policy to avoid returning stale information while controlling cache costs?
A good answer covers: Align cache TTL with the expected freshness of the underlying data; Categorize queries by their update frequency or volatility; Implement selective cache invalidation or TTLs based on content type.
Usage data from your model provider often arrives after the session has closed and the user has left. How would you design the system to ensure accurate per-session cost accounting, even with delayed usage reports?
A good answer covers: asynchronous reconciliation of delayed usage; temporary session state retention or soft closure; handling late-arriving data for finalized sessions.
A semantic cache is used to reuse responses for similar queries. You notice that requests like 'enable notifications' and 'disable notifications' are treated as equivalent, leading to incorrect outputs. How would you diagnose and address this?
A good answer covers: Analyze how semantic similarity is computed and what features it uses; Test cache behavior with minimal pairs of opposite intent; Refine similarity thresholds or incorporate intent labels in cache keys.
Your fallback routing chain tries several models for each unresolved task. Billing shows multiple models are charged per user request. How would you diagnose and reduce these unintended costs?
A good answer covers: Trace the execution path to see when and why multiple models are invoked; Review early stopping and success detection logic in the fallback chain; Analyze logs to distinguish between resolved and unresolved requests.
A provider caches LLM completions using the entire prompt as the cache key. Your team frequently changes the prompt prefix (instructions), causing cache misses and higher costs. Under strict cost constraints, how would you redesign prompts or caching to mitigate this?
A good answer covers: Prompt structure changes can invalidate provider-side caches; Stabilizing the prompt prefix or isolating user content reduces cache churn; Client-side caching or prompt versioning can help preserve cache hits.
Questions
Production questions.
What to do when a live LLM system starts to misbehave.
An assistant follows an instruction in short conversations but ignores it in long ones. What would you inspect before changing the model?
A good answer covers: inspect the actual assembled messages and truncation policy; separate context loss from conflicting or untrusted instructions; reproduce with controlled history lengths and instrument token budgets.
A schema requires a 'date_of_birth' field, but upstream data sometimes lacks this information. The system currently fabricates placeholder dates to satisfy the schema. How would you diagnose and address the risks of this behavior?
A good answer covers: Identifying fabricated data via audit or logs; Schema evolution to support missing values; Assessing downstream impact of placeholders.
During a rolling deployment, some requests use a new prompt version while others use an old parser, causing failures. How would you diagnose and resolve the version mismatch, ensuring safe transitions in future deployments?
A good answer covers: Trace deployment strategy and request routing; Verify prompt-parser compatibility at each stage; Design atomic or coordinated rollout for dependent components.
Your trace sampling configuration is set to capture only the first N requests per minute. You discover that failed requests, which are often slower, are rarely sampled. What concrete change would you make to ensure failures are still captured for diagnosis?
A good answer covers: Sampling based only on arrival order misses rare/slow failures; Need to prioritize or guarantee sampling of failed/slow requests; Implement adaptive or outcome-based sampling logic.
Your observability system captures and stores raw prompts for debugging, but recent audits show some prompts are retained past configured limits. How would you diagnose and resolve this retention policy violation?
A good answer covers: Retention policies must be enforced at storage and ingestion; Diagnose by auditing storage and comparing to policy; Implement automated data expiration or deletion mechanisms.
A crucial event in a user-submitted video is missed by the model, which samples frames at a fixed interval. The event is very brief. How would you diagnose the sampling logic to ensure such events are detected?
A good answer covers: Fixed interval sampling can skip short-duration events; Review frame selection policy for temporal coverage gaps; Test with videos containing brief events at different positions.
Your service lets users provide URLs to fetch resources. Some URLs resolve to private IPs (e.g., 192.168.x.x), which could expose internal systems. What design would you use to block access to private network addresses?
A good answer covers: Resolve hostnames to IP addresses before fetching; Check resolved IP against private/reserved ranges; Block requests to any private or local addresses.
A parser in your input pipeline throws errors for certain malformed messages. The system retries these messages indefinitely, clogging the pipeline. What system design would you implement to handle persistent failures safely?
A good answer covers: Limit retry count for failed messages; Route unrecoverable messages to a dead-letter queue; Log and monitor dead-lettered messages for investigation.
You are tasked with fine-tuning a model to improve answer quality, but your only available feedback signal is answer length. How would you design your system to avoid optimizing for verbosity at the cost of correctness?
A good answer covers: Identify risk of proxy objective misalignment; Incorporate correctness-focused signals into fine-tuning; Validate outputs for both quality and length constraints.
You've improved both retrieval and fine-tuning, but can't tell which is responsible for recent accuracy gains. What steps would you take to attribute improvements to each component?
A good answer covers: Design controlled ablation tests for each component; Run evaluation with only retrieval or only fine-tuning enabled; Compare performance metrics across ablations.
Users report worse answers after a deployment, but your aggregate quality dashboard is unchanged. What would you investigate?
A good answer covers: slice by user cohort, task and data version; compare traces and changes to prompts, models and indexes; reproduce failures and canary or roll back.
Design recovery when your primary model provider becomes unavailable midway through a conversation. A backup model supports different features. Describe how you would decide whether to continue.
A good answer covers: check backup capability and privacy compatibility before routing; persist accepted user input and prevent duplicate actions across retries; test degraded behavior and communicate unavailable features explicitly.
Your API expects a structured object in responses, but some requests return a refusal message object instead. This breaks downstream consumers expecting the object. How would you isolate and diagnose the cause of these unexpected refusal responses?
A good answer covers: Differentiating refusal and data objects in logs or traces; Tracing input conditions that trigger refusals; Verifying contract and error handling in provider logic.
A model provider returns an empty list for both 'no matches found' and 'provider failure'. Consumers cannot distinguish between these cases. How would you design the output contract to make this distinction explicit?
A good answer covers: Explicit signaling of error vs. empty result; Schema design for disambiguating cases; Testing consumer handling of new contract.
Your extraction system assigns a confidence score to each extracted value, but there is no rule for abstaining when confidence is low. How would you redesign the pipeline to calibrate scores and set appropriate abstention thresholds?
A good answer covers: Calibrate confidence scores against labeled data; Define and tune abstention thresholds for extractions; Monitor downstream impact of abstentions.
A model alias is updated to point to a new model version in the registry, but the application is not redeployed. Users report inconsistent responses. How would you diagnose and resolve this issue?
A good answer covers: alias resolution timing; cache invalidation or refresh; explicit model version pinning.
Your model service logs exception messages, but you discover some logs expose candidate user or model-generated text. Under compliance constraints, how would you redesign logging to avoid exposure while retaining enough data for debugging?
A good answer covers: Log redaction or filtering for sensitive fields; Structured error logging with minimal context; Verification that logs meet compliance requirements.
A vision model sometimes describes objects that are not present when given a blank or nearly empty image (e.g., all white). What design steps would help prevent such hallucinations or detect them before use?
A good answer covers: Blank image detection as a pre-filter; Thresholding on visual content entropy or features; Verification step to suppress output on low-content images.
Your service receives user-uploaded images and determines their type (e.g., JPEG, PNG) by reading the file extension from the filename, which is not verified. What design change would you propose to prevent security or data integrity issues?
A good answer covers: File extensions are not a reliable indicator of actual file content; Content-based MIME type detection is necessary for untrusted uploads; Reject or quarantine files with mismatched extension and content type.
You notice that certain non-ASCII identifiers are silently altered in your logs and outputs after processing. The input files are sometimes malformed UTF-8. What diagnostic steps would you take to isolate where and how the corruption occurs?
A good answer covers: Malformed UTF-8 may be replaced or dropped during decoding; Silent errors occur if decoding failures are not surfaced; Trace identifier transformation through each pipeline stage.
How to answer
A simple plan for the hour.
- Ask and set numbers
Who uses it, how often, how fast it must be and what a wrong answer costs.
- Sketch the path of one request
From the user’s message to the answer: checks, search, the model call, tools, the reply.
- Go deep on the hardest part
Pick the step that decides quality, speed or cost and work through it in detail.
- Cover failure and measurement
Say what breaks, what the user sees then, and which numbers tell you the system is healthy.
Questions.
How is LLM system design different from classic system design?
You still need storage, queues and caching. On top of that, one step is slow, costs money per call and can give a different answer each time. So you spend more time on speed, cost, evals and safe fallbacks.
How do I make an LLM feature faster?
Stream the answer so the user sees words early. Cache repeated work. Send easy requests to a smaller model. Cut the prompt. Run independent steps at the same time.
Who gets LLM system design interviews?
Mostly AI engineers and senior engineers on teams that ship LLM features. Some general loops now swap the classic design round for one.
Can I practice these questions with Orin?
Yes. The AI Engineer mock interview asks questions like these by voice, with follow-ups based on what you say. You also fix real code in a live editor. After it you get a report with quotes from your answers and what to practice next. Your first interview is free.
Practice LLM system design with follow-ups.
An AI engineer mock interview by voice, with a design question for a real product. First interview free.
Start free interview