AI Engineer track
AI engineer interview questions and practice
AI engineer interviews test how you build with LLMs in real products: retrieval, agents, evals, speed and cost. You will be asked to find what broke, fix code and design a system. Practice all of it in one mock interview.
Questions
Interview questions and answers.
Real questions from this track, from easier to harder, with what a good answer covers. In the mock interview, each one comes with follow-ups based on what you say.
Your support assistant cites an outdated policy even though the current document is indexed. How would you isolate the cause?
A good answer covers trace the returned document ID and version.
An LLM feature looks excellent in demos. What evidence would you require before shipping it to customers?
A good answer covers build representative labeled cases including failures.
An assistant follows an instruction in short conversations but ignores it in long ones. What would you inspect before changing the model?
A good answer covers inspect the actual assembled messages and truncation policy.
Design an assistant that starts responding quickly while searching documents. A complete answer can take five seconds. How would you improve perceived latency without streaming unsupported claims?
A good answer covers gate factual answer generation on sufficient retrieved evidence.
Design context management for a chat assistant whose costs grow with conversation length. Users expect it to remember decisions from earlier turns.
A good answer covers retain explicit user decisions and source references in structured state.
Your agent tool is supposed to only read files from a designated workspace directory, but logs show it sometimes accesses files outside this path. What steps would you take to isolate the root cause and prevent future escapes?
A good answer covers Validation against the workspace root.
A product team lowers sampling temperature to zero. Its assistant now gives the same incorrect historical date on every request. Explain what changed and propose a way to improve factual correctness.
A good answer covers separate sampling variability from the model's learned conditional distribution.
A manual task took five minutes. An AI draft takes 30 seconds, but reviewing and correcting it takes five minutes. The dashboard reports a 90% time saving using generation time alone. How would you assess the result and decide the next experiment?
A good answer covers measure total human effort and end-to-end completion rather than generation time alone.
A support assistant finds the right manual but gives incomplete installation steps. The steps span two chunks. How would you diagnose and fix this?
A good answer covers inspect retrieved chunks against the complete source passage.
You are launching a support assistant with 100 labeled customer questions and no production history. Design a release check and explain what passing it would and would not establish.
A good answer covers define task-specific correctness groundedness and failure criteria.
A RAG system splits documents into overlapping chunks. Sometimes the same sentence appears multiple times in the context window. How would you diagnose the cause and verify a fix?
A good answer covers Trace chunk boundaries and overlap logic.
A user was removed from a sensitive group, but their RAG query still returns documents accessible only to that group for several minutes after removal. How would you isolate the cause?
A good answer covers Check if retrieval index snapshots group membership at query time.
A retrieval cache is keyed only by content hash. Some documents are public, others private. How would you redesign the cache to prevent users from seeing private content via public cache hits?
A good answer covers Cache keys must include access scope (e.g., public/private).
Your RAG pipeline uses a candidate generator with a service account, then reranks using the end user's identity. What is one risk of this setup, and how would you mitigate it?
A good answer covers Risk of retrieving unauthorized candidates.
Your RAG assistant often cites a passage that is topically related but does not actually support the generated answer. What concrete design step would you implement to ensure citations are tightly linked to the specific claims made in answers?
A good answer covers semantic alignment check between claim and passage.
Your RAG system displays answer confidence based on retrieval similarity scores. Explain the flaw in this approach and describe a safer method for estimating answer correctness.
A good answer covers Distinguish retrieval similarity from factual correctness.
A durable agent workflow resumes after a crash and re-executes a read step that already finished, causing duplicate reads. How would you isolate whether this is due to missing or incorrect persisted state?
A good answer covers Inspect persisted workflow checkpoint for step completion markers.
A worker sends an approval request and crashes before the user responds. What design would ensure the approval request is not lost and can be recovered after a restart?
A good answer covers Persisting approval requests before sending.
An agent asks an external service to create a support ticket. Its request times out, the UI reports uncertainty, and a success callback may arrive later. The service accepts an idempotency key and offers a status lookup. Design the local state transitions so retries and late callbacks converge on one ticket without claiming a timeout proves failure.
A good answer covers Represent timeout as an unknown outcome when an external side effect may have committed.
An agent checks user approval for a file deletion, then waits for a plan step to finish. When it finally deletes the file, the user's approval may have been revoked. How would you diagnose and prevent this authorization gap?
A good answer covers risk of stale approval between check and use.
You notice your model's evaluation accuracy is suspiciously high. Later, you find that near-duplicate paraphrases of some evaluation examples exist in your training data. What risk does this pose, and how would you design splits to ensure valid evaluation?
A good answer covers risk of data leakage from paraphrase overlap.
Your retrieval-augmented QA model achieves unusually high accuracy on a held-out test set. Review shows some retrieval sources contain verbatim answers to test questions. How would you determine if this is inflating your evaluation results?
A good answer covers Check for answer leakage by comparing test answers with retrieval sources.
Two models are compared using both macro and micro averages, but the preferred model changes depending on the metric. What causes this, and how would you select the appropriate metric?
A good answer covers Metric choice affects ranking in imbalanced datasets.
Your system's success metric counts tool actions executed, but does not check if the user's goal was actually achieved. What specific metric or design change would you implement to ensure evaluation reflects real user success?
A good answer covers Define metrics based on user goal completion, not just actions.
You observe a sample-ratio mismatch: the number of users assigned to each experiment arm is not as expected. How would you diagnose the cause and verify correct experiment assignment?
A good answer covers assignment logic and randomization audit.
A retrieval-augmented LLM uses table cells as context. An attacker inserts a prompt injection payload into a cell. How would you design the system to mitigate this risk while preserving retrieval utility?
A good answer covers Input sanitization or filtering of retrieved content.
Your async server uses Python's asyncio event loop, but some requests stall for several seconds. Investigation shows a blocking network client is called during request handling. How would you diagnose and fix this issue?
A good answer covers Blocking calls block the entire event loop, stalling all coroutines.
After a cache miss, your AI service receives a burst of identical requests, all querying the same backend. What concrete mechanism would you add to prevent backend overload and reduce redundant work?
A good answer covers Single-flight or request coalescing to deduplicate backend calls.
Your system retries API requests after timeouts, but sometimes the original request is still in flight. What design would you use to avoid duplicate work and wasted provider capacity?
A good answer covers Detect and cancel in-flight requests before retrying.
An audio streaming pipeline starts lagging and eventually crashes with memory errors. You discover that the audio buffer grows without bounds when downstream playback is slow. What would you check to diagnose why backpressure is not limiting the buffer size?
A good answer covers Backpressure signaling from playback to upstream.
An approximate nearest neighbor (ANN) search API is combined with a selective filter. Users report that too few results are returned, even when more matches exist in the database. What concrete change would you make to address this?
A good answer covers Candidate pool size tuning.
A retrieval service has a default timeout of 5 seconds, but the end-to-end user request deadline is only 2 seconds. How would you diagnose and verify the impact of this timeout mismatch?
A good answer covers Compare component timeouts to overall deadline.
Your LLM provider offers a prompt cache: cached input tokens are billed at 50% the normal rate, not zero. Design an accounting system to ensure users are charged the correct rate for both cached and uncached tokens on each request.
A good answer covers Identify and separate cached vs. uncached tokens per request.
A model router picks which model to use based on each model's self-reported confidence score. There is no calibration. What are the risks of this approach, and how would you improve reliability?
A good answer covers Self-reported confidence can be poorly calibrated or misleading.
A schema requires a 'date_of_birth' field, but upstream data sometimes lacks this information. The system currently fabricates placeholder dates to satisfy the schema. How would you diagnose and address the risks of this behavior?
A good answer covers Schema evolution to support missing values.
During a rolling deployment, some requests use a new prompt version while others use an old parser, causing failures. How would you diagnose and resolve the version mismatch, ensuring safe transitions in future deployments?
A good answer covers Verify prompt-parser compatibility at each stage.
Your trace sampling configuration is set to capture only the first N requests per minute. You discover that failed requests, which are often slower, are rarely sampled. What concrete change would you make to ensure failures are still captured for diagnosis?
A good answer covers Sampling based only on arrival order misses rare/slow failures.
Your observability system captures and stores raw prompts for debugging, but recent audits show some prompts are retained past configured limits. How would you diagnose and resolve this retention policy violation?
A good answer covers Retention policies must be enforced at storage and ingestion.
In a fine-tuning run, user prompts are much longer than assistant replies. Training loss falls, but response quality on held-out tasks barely changes. The trainer currently scores every non-padding token. How would you investigate the training objective?
A good answer covers inspect how prompt versus response tokens contribute to the reported loss.
An analytics assistant joins orders to order_items and then sums orders.total by month. Orders with more items contribute their full total several times. Explain the issue and propose a fix that also works when two orders have identical totals.
A good answer covers identify the one-to-many join as changing the aggregation grain.
What it covers
Topics.
How it works
How the mock interview runs.
A short warm-up about your work.
A broken system to look into: find the cause and fix it.
A coding task in a live editor.
A design question for a real product.
Your report scores technical correctness, problem-solving & tradeoffs, evaluation & production, communication clarity, with quotes from your answers as proof.
Questions.
What questions are asked in an AI engineer interview?
Expect questions about RAG, agents, evals, prompt and model choices, speed and cost. Many rounds give you a broken system and ask you to find the cause.
Do AI engineer interviews include coding?
Yes. You often fix or extend real code, like an agent loop or a retrieval step, and explain your changes.
How does an Orin mock interview work?
You talk with Orin by voice, like a video call. It asks questions, listens and asks follow-ups based on what you said. Coding happens in a live editor and design on a live whiteboard. After the interview you get a report with quotes from your answers, what cost you points and what to practice next.
Is it free?
Your first interview is free: a 15-minute round with a short coding exercise. No card needed.
More practice.
Practice the real thing.
A ai engineer mock interview by voice, with follow-ups and feedback you can use. First interview free.
Start free interview