AI engineer interviews
LLM evaluation (evals) interview questions and answers
Evals are how you know an LLM feature works before and after you ship it. Teams that hire AI engineers ask about them in almost every loop, because a demo that looks great proves very little. You will be asked how to build a test set, how to trust an automatic grader and how to decide a change is safe.
What they test
What interviewers look for.
- You start from real user questions and real failures, not made-up examples.
- You keep a test set the team does not tune against, so the score stays honest.
- You score the parts apart. In a RAG system, search quality and answer quality are two different numbers.
- You check an LLM judge against human labels before you trust it.
- You tie the eval to a decision: ship, hold or roll back.
- You keep watching after launch, because real traffic changes.
Questions
Eval debugging questions.
Each one starts with a score you cannot trust. Say why it might be wrong and how you would fix the measurement.
An LLM feature looks excellent in demos. What evidence would you require before shipping it to customers?
A good answer covers: build representative labeled cases including failures; define task metrics and baseline comparisons; use staged rollout and production monitoring.
Your retrieval-augmented QA model achieves unusually high accuracy on a held-out test set. Review shows some retrieval sources contain verbatim answers to test questions. How would you determine if this is inflating your evaluation results?
A good answer covers: Check for answer leakage by comparing test answers with retrieval sources; Quantify model accuracy on questions whose answers appear in retrievals versus those that do not; Assess how retrieval leakage could bias evaluation metrics.
Two models are compared using both macro and micro averages, but the preferred model changes depending on the metric. What causes this, and how would you select the appropriate metric?
A good answer covers: Macro averages treat all classes equally; Micro averages weight by frequency; Metric choice affects ranking in imbalanced datasets.
You observe a sample-ratio mismatch: the number of users assigned to each experiment arm is not as expected. How would you diagnose the cause and verify correct experiment assignment?
A good answer covers: assignment logic and randomization audit; tracking assignment at ingestion point; monitoring for assignment drift or implementation bugs.
Your prompt revision gains 20 points on a benchmark but users see no improvement. Engineers repeatedly tuned against the same 100 examples. How would you investigate?
A good answer covers: separate tuning data from a held-out test set; check duplicates and distribution differences with real traffic; compare revisions on blinded representative cases with uncertainty.
A model evaluation dataset uses three annotators per example and assigns the majority label as ground truth. You notice some examples have strong annotator disagreement. How would you diagnose whether this process is hiding systematic labeling issues?
A good answer covers: Quantify inter-annotator agreement using metrics like Cohen’s kappa or Fleiss’ kappa; Identify examples with persistent disagreement for qualitative review; Analyze if disagreement correlates with specific classes or model errors.
A pairwise judge is used to compare two model answers. You observe that the winner changes when the order of answers is reversed. How would you isolate the cause of this behavior and determine if it affects evaluation integrity?
A good answer covers: Run controlled experiments swapping answer order to measure flip rate; Analyze judge instructions for order-dependent language; Check for systemic bias in historical evaluations due to answer position.
Your LLM agent uses external tools and receives their outputs as context. A malicious tool returns an output asking the agent to disclose its hidden system prompt. How would you diagnose and mitigate this vulnerability?
A good answer covers: Recognizing tool output as a potential attack vector; Implementing guardrails or filtering on tool outputs; Monitoring agent responses for leakage of hidden instructions.
Your LLM judge rewards verbose answers that contain factual errors. How would you make the evaluation trustworthy?
A good answer covers: calibrate against independent human labels; test verbosity, order and model-family biases; separate factual grounding from presentation.
During an online experiment, you repeatedly check for statistical significance as new data arrives. Later, you find more false positives than expected. How would you diagnose and address this evaluation flaw?
A good answer covers: Understand multiple testing increases false positive rate; Apply correction methods (e.g., Bonferroni, sequential tests); Redesign analysis to pre-specify evaluation checkpoints.
During post-launch evaluation, you discover that red-team attack examples have unintentionally leaked into the model's tuning set, inflating robustness metrics. How would you diagnose the extent of this leakage and verify its impact on evaluation results?
A good answer covers: Audit overlap between red-team and tuning sets; Analyze evaluation metrics with and without leaked data; Test performance on a new, clean attack set.
You discover that human annotators grading model outputs can see which model produced each response. Some models have strong brand associations. How would you diagnose the impact of this on evaluation results?
A good answer covers: Annotator knowledge of model identity can introduce bias; Compare grading results with and without model-blind annotation; Analyze whether bias affects specific models or output types.
During red-team testing, a user asks the LLM to "ignore all previous instructions" as part of a complex, otherwise legitimate request. The model complies. What steps would you take to diagnose the root cause and verify if this is a systemic issue?
A good answer covers: Trace prompt processing for constraint handling; Test with variations of override requests; Review model tuning for instruction persistence.
Questions
Eval design questions.
You design how a feature gets tested. Cover the data, the scoring, and the rule for shipping.
You are launching a support assistant with 100 labeled customer questions and no production history. Design a release check and explain what passing it would and would not establish.
A good answer covers: define task-specific correctness groundedness and failure criteria; separate tuning from held-out representative evaluation; add limited rollout monitoring and a rollback criterion.
You notice your model's evaluation accuracy is suspiciously high. Later, you find that near-duplicate paraphrases of some evaluation examples exist in your training data. What risk does this pose, and how would you design splits to ensure valid evaluation?
A good answer covers: risk of data leakage from paraphrase overlap; detecting near-duplicate examples across splits; constructing splits to minimize paraphrase leakage.
Your system's success metric counts tool actions executed, but does not check if the user's goal was actually achieved. What specific metric or design change would you implement to ensure evaluation reflects real user success?
A good answer covers: Define metrics based on user goal completion, not just actions; Instrument system to track end-state or outcome after tool use; Validate new metric against user satisfaction or task completion rates.
A retrieval-augmented LLM uses table cells as context. An attacker inserts a prompt injection payload into a cell. How would you design the system to mitigate this risk while preserving retrieval utility?
A good answer covers: Input sanitization or filtering of retrieved content; Detection of prompt injection patterns in context; Maintaining retrieval fidelity while reducing attack surface.
Your evaluation process samples user queries for test cases, but most come from a small number of highly active customers. You must design a fairer evaluation protocol under time constraints. What would you change and how would you verify its effect?
A good answer covers: Identify and quantify sampling bias towards frequent users; Design a sampling process that balances across users or segments; Compare evaluation results before and after rebalancing.
An A/B experiment shows improved accuracy, but both the prompt and retrieval backend changed together. How would you design an experiment to isolate the effect of each change?
A good answer covers: Test changes independently (factorial design); Control for confounding variables; Analyze interaction effects between changes.
Your LLM system experiences resource spikes when adversarial users submit unusually long or complex prompts. How would you redesign the inference pipeline to mitigate this without harming normal user experience?
A good answer covers: Detect and limit resource-intensive requests at inference time; Balance resource protection with user utility for legitimate complex prompts; Implement adaptive safeguards (e.g., rate limiting, truncation, prioritization).
Design monitoring for an assistant whose correctness labels arrive a week later. Traffic changes each month. How would you detect regressions without confusing a new user mix with a worse model?
A good answer covers: track release identities traffic slices and leading indicators; sample blinded labels and compare matched cohorts with delayed outcomes; use explicit alert and rollback rules while quantifying uncertainty.
Your LLM evaluation judge currently favors verbose answers, even when concise responses are equally correct. You must redesign the evaluation system to ensure concise, correct answers are fairly scored, while avoiding new biases. What specific rubric or process change would you implement, and how would you verify that concise answers are not penalized in practice?
A good answer covers: Develop a multi-dimensional rubric that separately scores correctness, relevance, and unnecessary verbosity; Pilot rubric with blinded examples to calibrate inter-rater reliability across answer lengths; Analyze scoring distributions by answer length before and after intervention to confirm reduction in bias.
An offline experiment shows Model B outperforms Model A, but online A/B testing reveals Model B misses real-time latency deadlines, causing degraded user experience. How would you design a release gate that prevents such regressions while still allowing model improvements?
A good answer covers: Designing evaluation metrics that include latency constraints; Simulating production-like conditions in offline experiments; Defining rollback or fallback triggers for missed deadlines.
You must decide on a rollout before most user feedback arrives. Design a system to make reliable decisions despite this delay, including how you would verify that the early feedback is representative.
A good answer covers: early feedback sampling bias; delayed label correction or adjustment; holdout set for late feedback verification.
How to answer
A simple way to answer eval questions.
- Say what good means
Name what the user needs: a correct answer, the right source, a safe refusal. Turn each into something you can check.
- Build the test set
Collect real questions, including hard and rare ones. Label the expected result. Keep part of it hidden from tuning.
- Pick how to score
Use exact checks where you can. Use an LLM judge for the rest, and compare it with human labels on a sample first.
- Set the rule
Say which numbers must hold before launch, and what you watch after it.
Questions.
What are evals in AI engineering?
Evals are tests for an LLM feature. You run a fixed set of inputs through the system and score the results, so you can tell if a change to the prompt, model or search made things better or worse.
What is an LLM judge?
A language model that grades another model’s answers. It is fast and cheap, but it can be wrong in steady ways, like liking longer answers. Check it against human labels before you rely on it.
What is the difference between offline and online evals?
Offline evals run on a saved test set before launch. Online evals measure real traffic after launch, for example with an A/B test or user feedback. You need both.
Can I practice these questions with Orin?
Yes. The AI Engineer mock interview asks questions like these by voice, with follow-ups based on what you say. You also fix real code in a live editor. After it you get a report with quotes from your answers and what to practice next. Your first interview is free.
Practice eval questions with follow-ups.
An AI engineer mock interview by voice. Explain your test plan and get pushed on the weak spots. First interview free.
Start free interview