Questions for roles building on large language models: RAG, evals, agents, tool calling, security, and the cost and reliability engineering around them.
RAG retrieves relevant documents at query time and puts them in the model's context, so answers are grounded in your data without changing the model's weights. Choose RAG when the knowledge changes frequently, must be permissioned per user, needs to be citable, or is too large to bake into a model — updating an index is cheap and instant, retraining is neither. Fine-tuning is the better tool for changing behavior rather than knowledge: output format, tone, domain-specific reasoning patterns, or distilling a large model's behavior into a smaller one. The approaches compose — a fine-tuned model over a retrieval index is common — and the interview follow-up is usually about RAG's real costs: retrieval quality becomes your accuracy ceiling, and chunking, indexing, and relevance ranking are where those systems actually fail.
An embedding model maps text to a dense vector such that semantically similar texts land near each other; retrieval is then a nearest-neighbor search — typically cosine similarity or dot product — over a precomputed index of document vectors. At scale, exact search is too slow, so vector databases use approximate nearest-neighbor structures (HNSW graphs, IVF partitions) that trade a little recall for large speedups. The practical judgment interviewers probe: embedding similarity is not relevance — it misses exact identifiers, negations, and keyword matches — so production pipelines usually run hybrid search (vector plus BM25-style lexical) and often a cross-encoder reranker on the top candidates. Chunking strategy matters as much as the index: chunks must be small enough to embed coherently but large enough to answer with, and the embedding model version must match between indexing and query time.
Treat context as a budget you allocate deliberately, not a buffer you append to until it overflows. Standard techniques: summarize or compact older conversation turns while keeping recent ones verbatim, retrieve rather than retain (store history externally and pull back only what is relevant to the current turn), and structure prompts so stable content — system instructions, tool definitions — stays byte-identical at the front, which also enables prompt caching. Two failure modes to name: silent truncation, where the important early instruction falls out of the window, and degraded attention over very long contexts, where models weight the middle of the context less reliably than the ends — both argue for curating context rather than maximizing it. Measure token usage per component so you know what to cut when the budget tightens.
Offline first: build an eval set of real or realistic inputs with graded expected outputs, and score candidate changes against it before deploy — this is what makes prompt and model changes safe to iterate on. Grading methods, in order of trustworthiness: exact or programmatic checks where the task allows (structured outputs, retrieval hit rate), human review for a sample, and LLM-as-judge for scale. Online, complement with A/B tests on real traffic and product metrics (task completion, user corrections, thumbs-down rate), because offline sets drift from real usage. The mistakes to call out: evaluating on the examples you developed the prompt against, eval sets too small to detect the effect sizes you care about, and treating a single aggregate score as the goal rather than reading the failures.
LLM judges are scalable but biased in known, measurable ways: they prefer longer and more confident answers, favor outputs whose style resembles their own (including self-preference when judging their own model family), and show position bias in pairwise comparisons — swapping the order can flip the verdict. Mitigations: calibrate the judge against a human-labeled sample and report agreement before trusting it; randomize or run both orderings in pairwise setups; use narrow rubrics with concrete criteria instead of a single 'rate 1-10' prompt; and prefer grading against a reference answer over open-ended scoring where possible. Keep the judge model and prompt versioned like code, because judge drift silently invalidates historical comparisons. And keep humans in the loop for the failure review — the judge tells you scores moved, not why.
Layer defenses, because no single one suffices. Grounding: give the model the facts via retrieval and instruct it to answer only from the provided context, with an explicit permission to say 'I don't know' — models hallucinate more when the prompt implies an answer must exist. Constraining: narrow the task, use structured outputs, and validate anything checkable (IDs, dates, citations) against the source of truth programmatically — citation verification catches fabricated references cheaply. Detecting: evals that specifically measure faithfulness to the provided context, not just answer quality. Product design carries the rest: show sources so users can verify, calibrate the feature's claims to its reliability, and route high-stakes outputs through human review. The honest framing interviewers respect: you reduce and contain hallucination; you do not eliminate it.
Tool calling means the model, given tool schemas (name, description, typed parameters), emits a structured request to invoke one; your code executes the tool and returns the result to the model, which continues — the model never executes anything itself. An agent is this in a loop: model decides, harness executes, results feed back, until the task completes or a stop condition hits. Architecture decisions that matter: tool design (few, well-described, orthogonal tools with clear error messages beat many overlapping ones), loop guards (iteration limits, budgets, timeouts), state management (what persists across turns versus what is recomputed), and checkpointing so a long run can fail partway without losing everything. Say the empirical truth: most agent failures are context and tool-description failures, not model-capability failures, and single-agent-with-good-tools should be exhausted before reaching for multi-agent designs.
MCP is an open protocol standardizing how AI applications connect to external tools and data sources. Without it, every assistant-to-integration pairing is custom glue code — an M×N problem; with it, a tool vendor ships one MCP server and any MCP-capable client can use it. The architecture: an MCP server exposes tools (functions the model can call), resources (data the client can read), and prompts (reusable templates) over a defined transport — stdio for local servers, HTTP-based transports for remote ones — and the client (the AI application) discovers capabilities at connect time rather than compile time. The engineering trade-offs worth naming: every connected server's tool definitions consume context tokens, and third-party servers extend your trust boundary — a malicious or compromised server's tool descriptions and outputs are injection surface, so servers need the same vetting as any dependency.
Prompt injection is untrusted content — a retrieved document, a webpage, an email, a tool result — containing instructions the model follows as if they came from the developer or user. It is the signature vulnerability of LLM systems because models cannot fully separate instructions from data in a single token stream, which means no prompt-level defense is complete. Defend in depth: privilege separation (the model only gets the permissions and tools the current task needs — an email summarizer needs no send-email tool), treating all retrieved and tool-returned content as untrusted, human confirmation gates on irreversible or sensitive actions, output validation, and detection layers for known injection patterns. The design principle to state: assume the model can be persuaded, and make sure a persuaded model still cannot do serious damage. Mention indirect injection specifically — attacks arriving through content the user never sees are the dangerous class.
Measure per-request token economics first — input tokens, output tokens, and calls per user action — because intuition about where the money goes is usually wrong. The standard levers: route by difficulty (a smaller, cheaper model for the common easy cases, escalating to a larger one on failure or classified complexity), prompt caching (keep the stable prefix byte-identical so providers can reuse it — this cuts both cost and time-to-first-token), trimming context to what the task needs, capping and tightening outputs since generated tokens dominate latency, and batching anything offline onto cheaper asynchronous tiers. For perceived latency, streaming matters more than raw speed: users tolerate a long generation they can watch. Set budgets and alerts per feature — LLM cost scales with usage and prompt drift, and regressions arrive silently with the next prompt edit.
In order of reliability: use the provider's native structured-output or schema-enforcement feature where available — constrained decoding guarantees syntactically valid output by construction; use tool/function calling with a typed schema, which most APIs enforce well; or fall back to prompt-and-parse with a schema in the prompt, a robust parser, and a repair-retry loop for the failures. Regardless of mechanism, validate semantically after parsing: syntactically valid JSON can still contain wrong enums, hallucinated IDs, or out-of-range values, so run it through schema validation and business-rule checks before use. Design the schema for the model, too — flat structures with descriptive field names and enums outperform deeply nested cleverness, and letting the model emit a reasoning field before the answer fields often improves the answer at modest token cost.
Start from the task's actual requirements — quality bar, latency budget, cost ceiling, context length, modality, and any deployment constraint like on-prem or data residency — and then let your own evals decide, because public leaderboards rank general capability, not your task. The standard method: build the eval set first, prototype with a top-tier model to establish what is possible, then test cheaper and smaller models against the same set and take the cheapest one that clears your quality bar, keeping the router option (small model default, large model escalation) when the traffic mix justifies it. Weigh the non-benchmark factors that bite in production: rate limits, reliability track record, tool-calling and structured-output quality, and open-weight versus API trade-offs (control and privacy versus operational burden). Re-run the decision periodically — the model landscape shifts fast enough that last quarter's choice is stale.
Log the full request lifecycle — prompt version, model and parameters, retrieved context, tool calls with results, final output, token counts, latency, and user feedback signals — with a trace ID linking multi-step agent runs, because 'which step went wrong' is the question you will actually ask. On top of raw traces: dashboards for cost, latency percentiles, error and refusal rates per feature; drift monitoring on input distributions; and continuous evals — scoring a sample of live traffic with the same graders you use offline — so quality regressions surface without waiting for user complaints. Treat prompts as versioned, deployable artifacts so every output is attributable to an exact prompt+model pair; without that, incidents are undebuggable. And handle the data carefully: traces contain user content, so redaction, access control, and retention policies are part of the observability design, not an afterthought.