Applied ML Engineer Grill Chains

Why This Document Exists

The Applied ML Engineer role fails on judgment, not on infra. The seven primitives in 00-foundations-and-primitives-for-applied-ml-engineering.md define the antibodies. The eight scenarios in 01-deep-dive-per-applied-ml-story.md walk them through real product decisions. This file drills the judgment under interview-loop pressure.

Each grill chain follows the canonical project format: an Opening Question with a Round 1 answer, four escalating rounds (Surface → Push Harder → Squeeze → Corner), three architect-level escalations (A1, A2, A3), an Intuition Gained section, and explicit Red-Flag / Strong-Answer markers. The format mirrors ../Cost-Optimization-Offline-Testing/05-ml-ai-engineer-grill-chains.md and the canonical ../Domain1-FM-Integration-Data-Compliance/Skill-1.1.1-Comprehensive-Architectural-Design/scenarios/DEEP-DIVE-GRILLING-SESSION.md.

How To Use This File

Scoring rubric: - Pass — answer hits the core insight in Rounds 1-2; some weakness in Round 3-4 quantitative rigor; partial architect-level reasoning - Strong-Pass — clean answer through Round 4; quantitative rigor in Round 3; named tradeoffs in architect levels - Exemplary — proactive identification of failure modes; Master's-DS Depth subtleties surfaced unprompted; Amazon LP integration in OP1-style narrative

Most candidates fail on AML-04 (online/offline decoupling) and AML-05 (guardrail enforcement). These are the highest-signal scenarios for senior calibration.

The Eight Scenarios at a Glance

Scenario AML-01 — Customer-Pain → ML-Problem Translation

Opening Question

Q: You're 6 weeks into a quarter. Retention for new manga readers in JP just dropped from 38% to 31% week-over-week. Your PM wants you to "use ML to fix it." What do you say in the next meeting?

Round 1 Answer: I say "before I scope an ML solution, let me write the customer letter." The PM's framing is "use ML for retention" — that's a tool-first framing. The right framing is customer-first: who is the user, what specifically broke for them, when did it break, and what intervention — ML or not — would change their behavior. I draft a one-page Working-Backwards letter with a concrete persona (Yuki, 24, JP-Tokyo, tried 3 popular titles, didn't open day-7). The letter forces specificity. Once the pain is sharp, I run the heuristic-vs-ML break-even: if 5 rules cover the decision, a heuristic ships in 2 weeks at 60-70% of the lift; if it's truly fuzzy multi-dimensional, ML earns the 8-week investment. My intuition on this signal: the cold-start fallback (genre-popularity, EN-skewed) is misaligned for new JP users. A 5-question taste quiz is probably the right Stage-1 intervention. ML cold-start is a Stage-2 candidate for next quarter, not now.

Round 1 — Surface

Follow-up: How do you actually write the customer letter? What's in it?

The letter is one page, written in the customer's voice, before any ML scoping. Five components:

1. Concrete persona — name, age, location, manga-reading context. "Yuki, 24, lives in Tokyo, finished her engineering degree last year, has been a casual manga reader for 3 years. Discovered MangaAssist last week."
2. The pain in their words — not "retention dropped"; rather, "I tried 'Solo Leveling', 'Spy x Family', and 'Frieren'. Heard of all three. None of them felt like 'my thing'. I didn't open the app on day 7."
3. What would have made them come back — "If the bot had figured out I like slow-burn psychological stories, not action — and shown me 'A Silent Voice' or 'March Comes In Like A Lion' on day 1."
4. Why now — what's changed in catalog / traffic / customer behavior that makes this urgent? "The new spring-anime tie-in season has pushed action titles up the popularity rankings; new users get an even more action-skewed first-session than 6 months ago."
5. What we'll measure — "We will measure week-2 retention for new JP users; success is +4pp absolute."

The letter is reviewed by PM + Eng Manager + customer-insights researcher before any model scoping. If any of those three can't honestly say "yes, I believe Yuki would feel this," the project doesn't enter the experiment portfolio. This single gate catches 30-40% of proposed ML projects.

Round 2 — Push Harder

Follow-up: Your PM says "OK fine, we'll write the letter — but the executives expect an ML launch this quarter, the OP1 commits to it. What if the answer is 'don't use ML'?"

That's the real test of the role. Three things to do, in order:

1. Don't pre-commit the answer. "We'll do the framing work. If the framing supports ML, we ship ML. If it supports a heuristic, we ship the heuristic and add ML as a follow-up if the heuristic plateaus." This is not a refusal; it's a sequenced decision.

2. Give the OP1 narrative a way to win regardless. "The OP1 narrative is 'we improved new-user retention.' That narrative is satisfied by any successful intervention, ML or heuristic. If we ship a heuristic that gets +5pp retention, that's the OP1 win. If we ship ML that gets +6pp, that's the OP1 win. The OP1 commitment is to outcome, not to method."

3. Quantify the cost of forced ML. If the executives push for ML regardless, I'd write a six-pager with the comparison: heuristic option (2 weeks, +4-6pp expected), ML option (8 weeks, +6-9pp expected), cost-of-velocity if both happen simultaneously (ML team optimizes against a moving baseline, neither learns clean). The six-pager includes a tenet: "We ship the simplest intervention that meets the customer bar; ML is an upgrade path, not a default." Tenets are how I install backbone in OP1 documents — they survive personnel changes and political shifts.

The trap I'm avoiding: agreeing to "use ML" because it's politically expedient, then watching the team spend a quarter on a model that under-delivers because the heuristic was the right intervention. That's a quarter of velocity I never get back.

Round 3 — Squeeze

Follow-up: Show me the heuristic-vs-ML break-even math on this specific scenario. Numbers.

Here's the math, on the back of an envelope:

The heuristic dominates on velocity-adjusted EV by 3.4×. The ML option's incremental lift over heuristic is ~0.6pp absolute at 4× engineering cost. That's a bad trade this quarter.

The case for ML becomes interesting if the heuristic ships and plateaus. After 6 weeks of post-launch data, if heuristic delivered 5pp of the achievable 8pp lift, the remaining 3pp is the ML opportunity, and 8 engineer-weeks for 3pp is a different calculation (0.375 pp/week, similar to other Q-3 candidates). Then ML enters next quarter's portfolio honestly.

The math says: ship heuristic now, schedule ML evaluation for after 6-week post-launch data. Don't ship both.

Round 4 — Corner

Follow-up: You ship the heuristic. After 4 weeks, retention has gone from 31% to 33% — significant lift but well below the +4pp target. PM asks "should we just ship the ML cold-start now to capture the rest?" What do you say?

I say "slow down — we don't yet know why we underdelivered, and shipping ML on top of a partial-effect heuristic confuses the diagnostic." Three reasons:

1. The 2pp lift might be the heuristic working as intended on a smaller-than-expected addressable population. If only 60% of new JP users actually completed the quiz (we set 75% target), the 2pp aggregate is consistent with +3.3pp on quiz-completers and 0pp on non-completers. The next intervention might be quiz-UX improvement (1 week, free), not ML.

2. Shipping ML on top of heuristic creates an attribution problem. If we layer ML cold-start now, the next experiment is "ML cold-start vs heuristic-only baseline" — but the baseline is a moving target (heuristic still being tuned). We need to let the heuristic baseline settle for 4 more weeks before introducing ML.

3. The right "follow-up" isn't ML — it's diagnostic. What does cohort-stratification say? Is the 2pp lift uniform across JP-Tokyo, JP-Osaka, JP-other, mobile, app? If lift is concentrated in JP-Tokyo (the cohort that had the most action-bias issue), the heuristic worked exactly where it should have. If it's uniform-but-low, the 5-question quiz is too short or too generic. Stratification tells me the next move.

What I'd refuse to ship without: the cohort-stratified post-launch analysis. Without it, I'm flying blind on why we missed target, and shipping ML to "fix" it is gambling with another quarter.

Architect-Level Escalation

A1: Design a framework that systematically prevents your team from picking ML when a heuristic is the right answer. What does it look like?

The framework has four layers:

1. PR/FAQ-mandatory gate — every proposed ML project starts with a one-page customer letter. The letter is reviewed by PM + EM + customer-research before model scoping. ~30% of proposals don't survive this step.

2. Heuristic-vs-ML break-even document — for surviving proposals, the team produces a 2-page break-even doc covering: estimated lift for both options, engineering cost both options, probability-of-success for both, time-to-ship, total EV per engineer-week. The doc has a default "ship heuristic first" recommendation; ML-first requires explicit evidence-based override.

3. Stage-gating — proposals approved for ML must specify "Stage 1 = heuristic; Stage 2 = ML if Stage 1 plateaus." Direct-to-ML proposals require Director-level sign-off and recorded justification. ~50% of would-be Stage-1-skipping proposals revert to staging when this gate is enforced.

4. Quarterly retrospective on hit-rate by category — every quarter, retrospect the team's portfolio: how many heuristics shipped, how many MLs shipped, what was the realized lift per option. Calibration loop: if heuristics consistently outperform per-engineer-week, increase the heuristic-first bias next quarter.

The systems insight: the temptation to default to ML is institutional. Engineers want to build models; PMs want "ML feature" in their roadmap; OP1 narratives sound bigger with ML. The framework counteracts the institutional bias mechanically. Without it, the team will systematically over-ML.

A2: Walk me through how this conversation with the PM looks in a six-pager review with leadership. What's in the document?

The six-pager structure:

Opening narrative (page 1): "MangaAssist's new JP-cohort retention dropped 7pp over 4 weeks. We diagnosed the leading cause as cold-start recommendation misalignment. We are recommending a heuristic-first intervention (5-question taste quiz) over an ML cold-start, and we want leadership alignment."

Tenets and risks (page 2): - Tenet 1: "We ship the simplest intervention that moves customer behavior. ML is an upgrade path, not a default." - Tenet 2: "We measure customer behavior, not proxy metrics. Retention is the customer behavior; CTR is a proxy." - Tenet 3: "We do not ship ML when a heuristic suffices. The bar for ML is +50% incremental EV over heuristic." - Risk: "Team's institutional preference for ML over heuristic. Mitigation: framework above." - Risk: "Quiz-completion rate may be lower than industry benchmarks. Mitigation: 1% canary first."

Customer letter (page 3): Yuki's story, in her words.

Decision framework (page 4): Heuristic-vs-ML break-even table; numbers above; recommendation: ship heuristic Q3, evaluate ML Q4.

Implementation plan (page 5): Pre-registered hypothesis, MDE, sample size, runtime, guardrails.

FAQ (page 6): "What if the OP1 commits to ML?" — covered by tenet 1. "What if heuristic underperforms?" — covered by Stage 2 contingency. "What if executives push for ML regardless?" — covered by escalation: I write a written justification document, sign and own the heuristic-first decision, accept that I may be overridden.

The Amazon LPs explicitly named: Customer Obsession (start with Yuki), Invent and Simplify (heuristic before ML), Bias for Action (2-week ship vs 13-week), Are Right A Lot (calibration loop on prior decisions), and Have Backbone, Disagree and Commit (the willingness to recommend against ML even when politically inconvenient).

A3: When do you say "yes, this is an ML problem" — what's your bar? And when does the bar bend?

My bar has three components, all of which must be met:

1. The decision boundary is genuinely fuzzy — more than 10 input dimensions matter, and the team can't write rules that capture them. Example: aspect-based sentiment classification (US-MLE-03) is genuinely fuzzy; not even careful annotators agree on every aspect.
2. Data volume is sufficient — at least 10K labeled examples available, growing at 1K/month or more. Below this, models overfit and rules win.
3. The expected ML lift over heuristic is ≥50% incremental — not 10% incremental, not 20% incremental. If the heuristic captures 80% of the lift, ML's incremental EV doesn't justify the engineering cost.

The bar bends in two cases:

1. Adversarial environments. Spam classification (US-MLE-07) requires ML even when rules cover most cases, because adversaries adapt to rules in days. The half-life of a spam rule is weeks; the half-life of a spam ML model with weekly retraining is months. Adversarial half-life flips the calculus.

2. Personalization at scale. Recommendation (US-MLE-06) genuinely requires user-level signal that no rule captures. A heuristic recommender works for cold-start; warm-start needs ML or you can't compete with Amazon's broader recommendation infrastructure.

When the bar doesn't bend, I default to no. The asymmetric cost of false-positive ML (quarter of misallocated effort) vs false-negative ML (small lift left on table) makes "default to no" the right calibration.

Intuition Gained — AML-01

The core insight: ML is a tool, not a destination. The Applied ML Engineer's value is choosing the right tool for the customer problem — and being willing to recommend not-ML when not-ML is right.

Mental model to carry forward:

"Write the customer letter before scoping the model. If the letter doesn't survive review, the model doesn't either."

The hidden failure mode: Institutional preference for ML over heuristic. Engineers want to build models; PMs want ML in their roadmap; OP1 narratives sound bigger with ML. The framework that defaults to "heuristic first" counteracts this systematically.

One-line rule: When 5 rules can express the decision, build the rules. Save ML for genuinely fuzzy decisions where 50+ signals matter.

Red-Flag Indicators (what a weak answer looks like)

• Says "yes, this is an ML problem" without writing the customer letter or running break-even
• Defaults to "use Personalize" or "fine-tune an embedding" before scoping the customer pain
• Claims engineering cost of ML is "the same" as heuristic
• Cannot quantify expected lift for either option
• Treats "OP1 commits to ML" as binding without exploring whether OP1 commits to outcome or method

Strong-Answer Markers (what a senior Applied ML Engineer would say)

• Insists on customer letter before model scoping; produces specific persona
• Quantifies break-even math with engineering-cost, expected-lift, probability-of-success
• Uses Stage 1 / Stage 2 framing: heuristic now, ML as contingent upgrade
• Names the institutional bias toward ML and prescribes a mechanical counter
• Connects to Amazon LPs (Customer Obsession, Invent and Simplify) without listing them — shows them
• Willingness to say "we should not ship ML this quarter" and defend it in a six-pager

Scenario AML-02 — Experiment Portfolio Prioritization

Opening Question

Q: Your manager hands you 12 candidate ML experiments for next quarter and says "pick 3." What's your decision framework, and what do you ask before you decide?

Round 1 Answer: I score each candidate with RICE-for-ML (Reach × Impact × Confidence ÷ Effort), where Confidence breaks down into prior-evidence quality and detectability-given-sample-size. Then I overlay EVOI: candidates whose primary value is learning (resolving a multi-quarter strategic uncertainty) get a separate slot. Then I overlay qualitative checks: cross-experiment dependencies, cost-estimate calibration against historical accuracy, and political constraints. Final pick is typically 2 high-RICE Bias-for-Action experiments + 1 EVOI swing-bet. Before deciding, I ask: (1) what's the team's last 12-quarter Confidence calibration? — if they're over-confident, haircut Confidence by 0.6×; (2) which candidates have engineering-cost mis-estimation risk based on past patterns?; (3) which candidates depend on each other and would split the test population if run simultaneously?

Round 1 — Surface

Follow-up: Walk me through scoring one candidate. Pick the reranker change.

US-MLE-02 reranker change (MiniLM-L6 → MiniLM-L12):

• Reach: 8 million sessions per quarter (every chatbot turn that does a search; rough estimate from 200K daily eligible × 90 days × 0.45 search-fraction).
• Impact: +0.5% absolute lift on useful-answer-rate, baseline 0.183 → 0.188. Magnitude is calibrated against the last 3 reranker upgrades (each delivered 0.3-0.7pp).
• Confidence breakdown:
• Prior-evidence quality: 0.85. Strong offline NDCG@10 +6%; offline-online correlation 0.68 (above threshold); prior MiniLM upgrades have shipped successfully.
• Detectability: 0.85. n=18.4K per arm with CUPED is achievable in 14 days at 200K eligible/day; even cohort-stratified claims are powered.
• Combined Confidence: 0.85 × 0.85 = 0.72.
• Effort: 12 engineer-weeks (training pipeline already exists; mostly experiment ops, integration, telemetry).
• RICE: 8M × 0.005 × 0.72 / 12 = 2400 raw. (The absolute number doesn't matter; the relative ranking is what informs the decision.)

The key thing in the score: Confidence isn't a vibe; it's two factors I can defend. The team that puts "0.7 because it feels right" on every candidate isn't doing portfolio reasoning, they're doing astrology with a spreadsheet.

Round 2 — Push Harder

Follow-up: Two candidates have the same RICE score: a high-impact-low-confidence cold-start swap (HRNN-Coldstart) and a low-impact-high-confidence multilingual intent fine-tune. Same RICE. Which do you pick and why?

I pick the cold-start swap as an EVOI bet, and the multilingual intent fine-tune as a Bias-for-Action ship. Here's the reasoning:

The two candidates have the same RICE because RICE doesn't differentiate "shipping value" from "learning value." The multilingual intent fine-tune has high probability of a small win — it's a Bias-for-Action ship: deliver a confirmed lift, low risk, predictable timeline. Ship it.

The cold-start swap has lower probability of any win this quarter — but win-or-lose, it resolves a multi-quarter strategic question: "is HRNN-Coldstart the right cold-start direction for the next year of recsys investment?" That EVOI is worth more than the marginal RICE delta vs. another candidate. It's a Think-Big bet that should have one slot in the quarter.

The trap: picking only by RICE means you ship 3 high-Confidence small-impact experiments and never run the swing bets that, over 4 quarters, define your platform. Portfolio thinking means one slot is reserved for a swing bet, even when the swing-bet RICE is below the next safe candidate.

So the answer to "which one" is "neither alone — both, with one Bias-for-Action and one EVOI in the same quarter, plus one more Bias-for-Action to round out the portfolio."

Round 3 — Squeeze

Follow-up: Your team's last 12 experiments. 7 won, 5 lost. Of the lost, 3 were cost-overruns (engineering took 2-3× estimate), 2 were genuine negative results. What does this tell you, and how does it change next quarter's portfolio?

Two specific calibration moves:

Move 1 — Confidence calibration adjustment. The team's hit-rate is 7/12 = 58%. If the team's average prior Confidence was 0.7 (typical), they're well-calibrated within noise. If average prior Confidence was 0.85 (over-confident), they're systematically over-estimating. I'd recompute: of the 12, what was the sum of prior Confidences? If it was 8.4 (i.e., expected 8.4 wins), and we got 7, calibration is good — the team is honest about uncertainty. If sum was 10 (expected 10 wins), the team is over-confident and I haircut every Confidence by 0.7× next quarter.

Move 2 — Cost-estimate calibration adjustment. The big signal: 3 of 12 ran 2-3× over engineering estimate. That's a 25% cost-overrun rate. The fix is mechanical: any candidate with engineering estimate ≥ 12 weeks must have a pre-mortem document with named cost risks (data-availability, infra readiness, training-pipeline maturity, integration complexity) and a contingency plan. The pre-mortem doesn't prevent the cost-overrun — but it surfaces the risks early so they can be priced into Confidence.

For next quarter's portfolio: I haircut Effort estimates upward by 1.3× across all candidates with prior-class-of-overrun risk; I require pre-mortem docs on the 3 highest-Effort candidates; and I avoid candidates that share cost-risk patterns with the 3 cost-overrun candidates from last quarter.

The systematic insight: most teams retrospect Confidence (did the experiment win?). Few teams retrospect Effort (did engineering match estimate?). Effort-calibration is the single highest-ROI improvement to portfolio quality.

Round 4 — Corner

Follow-up: Your 3-experiment quarter ships 2 wins and 1 loss. The loss was the EVOI swing bet you fought for. Leadership asks "why did you spend a slot on that one?" How do you defend?

I defend with what we learned, not what we shipped. The conversation:

"The swing bet was an EVOI experiment. Its primary value wasn't shipping; it was resolving a multi-quarter strategic uncertainty about cold-start direction. We ran it precisely because win-or-lose, the result changes our recsys investment plan for the next year. The result was negative — HRNN-Coldstart underperforms our two-tower baseline by 1.8% on cold-start CTR — but that's a result we now have, with confidence, that we didn't have before. It eliminates 4 quarters of potential mis-investment. The next 4 quarters of recsys work is now scoped against two-tower-with-better-cold-start, not HRNN. That's worth more than another small-win Bias-for-Action ship."

Then the harder question: "why didn't we know HRNN underperformed before running the experiment?" The honest answer: the prior literature on HRNN is mostly e-commerce, the team had no internal data on chatbot-context HRNN performance, and the only way to learn was to run. EVOI isn't betting on a winning outcome; it's investing in resolved uncertainty. That said, the cost-of-learning has to be calibrated — if EVOI experiments consistently lose AND don't shape future strategy, the team is just losing under a fancier name.

Defending this requires recording what the EVOI bet would teach in either direction, before running. If after the negative result the team can point to a written document saying "negative result will redirect us from HRNN-direction to two-tower-direction," that's an honest EVOI bet. If the team can't, it was a swing bet rebadged as EVOI to justify the loss. The discipline matters.

What I'd refuse to ship without: pre-registered EVOI value document signed before experiment starts. Without it, "EVOI" becomes the post-hoc rationalization for any loss. That's HARKing applied to portfolio reasoning.

Architect-Level Escalation

A1: Design a portfolio-management framework that scales to 50 candidate experiments per quarter and 5 ML teams. What does the system look like?

The system has four components:

1. Shared candidate-intake form. Every team submits candidates to a single intake document. The form forces specification of Reach, Impact, Confidence (broken into prior-evidence + detectability), Effort, Dependencies, and EVOI rationale (if applicable). Standardization means cross-team comparison is possible.

2. Quarterly portfolio committee. A cross-team committee (one Applied ML Engineer per team + one EM + one Director) scores all 50 candidates against the same RICE+EVOI rubric. Each team has a baseline allocation (e.g., 3 slots), but the committee can re-allocate based on portfolio-level reasoning — e.g., if all 50 candidates are reranker-flavor, redirect one team to recsys.

3. Calibration audit, every quarter. End-of-quarter retrospective: for each shipped experiment, compare predicted Confidence vs realized result; predicted Effort vs realized Effort; predicted EVOI vs actual learning. Aggregate the calibration deltas and feed them back into next quarter's scoring as multiplicative adjustments.

4. Dependency-graph awareness. Many ML candidates depend on each other (e.g., reranker upgrade requires embedding model to be stable; embedding model requires retrieval-pipeline change to be settled). The committee maintains a dependency DAG; experiments that violate DAG edges (e.g., "ship reranker before embedding settles") are flagged and either re-ordered or run on a frozen baseline.

The systems insight: portfolio management at 5-team scale is a different problem than at 1-team scale. The opportunity-cost interactions and dependency graphs become first-order; without explicit cross-team coordination, teams optimize locally and the org under-performs. The committee is the institution that internalizes the cross-team coupling.

A2: How do you defend this portfolio in OP1? Walk me through the narrative.

OP1 narrative is structured around three axes — outcome (what customer impact), method (what experiments deliver it), and risk (what could go wrong).

Outcome axis (page 1): "Our team commits to delivering [specific customer-experience improvements] in [specific quarters], measured by [headline metrics with targets]. Examples: +5pp new-user retention by end of Q3; +3% useful-answer-rate by end of Q4."

Method axis (page 2-3): "Our portfolio of [3 experiments per quarter, 12 per year] is selected via RICE+EVOI scoring with team-calibrated Confidence and pre-mortem'd Effort. We are running [name 2-3 quarters of named experiments]. The portfolio is not a list; it is a sequenced plan with explicit dependencies."

Risk axis (page 4-5): "Top three risks: (1) cost-overrun on the embedding-adapter experiment (mitigation: pre-mortem doc with contingency); (2) cohort-fairness regression on cold-start swap (mitigation: stratified holdout from day 1); (3) offline-online correlation collapse (mitigation: quarterly correlation audit; if Pearson drops below 0.5, freeze model shipping for harness rebuild — see AML-04). Each risk has a named owner and an escalation path."

Tenets (page 6): - "We ship customer-impact, not model-launches." - "We pre-register experiments and enforce mechanical guardrails." - "We default to heuristic when ML's incremental EV is < 50% over heuristic baseline." - "We run one EVOI swing-bet per quarter, even if it lowers expected hit rate."

LPs explicitly invoked: Customer Obsession (outcome axis), Bias for Action (method axis ship cadence), Frugality (RICE-prioritization), Insist on Highest Standards (pre-registration, calibration audit), Think Big (EVOI swing bet), Are Right A Lot (calibration loop).

A3: You're advocating for an EVOI swing bet that has 30% probability of winning. The Director says "we'd rather have 3 safe wins; the team's reputation depends on it." How do you respond?

I push back, structured:

"The team's reputation depends on shipping outcomes, not on never losing. A team that ships 3 safe wins per quarter for 4 quarters has 12 small wins on the board — but the team that runs 1 EVOI swing per quarter has either resolved 4 strategic questions in that time, or has 1 platform-defining win that compounds. Over 8 quarters, the EVOI-running team is in a different position: their portfolio reflects deliberate strategic choices, not just incremental tuning. The org that under-runs EVOI ends up with a backlog of unresolved strategic questions and a recommendation system that incrementally improves but never structurally evolves."

"That said, EVOI is risky and the failure mode is real: 'EVOI' becomes a hand-wave to justify any loss. The discipline I commit to is: every EVOI bet is documented with a pre-registered learning target. If the bet loses but the learning target is met, the bet was successful — we resolved the question we wanted to resolve. If the bet loses AND the learning target is unclear, that's a real loss; mark it down in the calibration audit and consider whether the bet was rebadged as EVOI to reduce accountability."

"Concretely: I'd commit to one EVOI bet per quarter, with a pre-registered learning document signed before the experiment starts. I'd publish results — wins and losses — in the team's quarterly retrospective. Over 4 quarters, leadership can audit: are EVOI bets producing learnings or excuses? If the former, continue; if the latter, the discipline isn't working and we revert to all-Bias-for-Action."

This is the Have Backbone, Disagree and Commit moment: I disagree with "3 safe wins is the goal," I propose the EVOI alternative with the discipline that prevents abuse, and I commit to publishing the audit so leadership can override me later if the discipline fails. That's the role's behavior.

Intuition Gained — AML-02

The core insight: Portfolio choice is not per-experiment optimization. It's a constrained allocation across 12 candidates with capacity for 3, where the right answer is typically 2 Bias-for-Action + 1 EVOI swing.

Mental model to carry forward:

"Pick 3 by RICE+EVOI+calibration. Reserve one slot for the swing bet that, win or lose, changes your platform's direction."

The hidden failure mode: Cost-estimate calibration. Most teams retrospect Confidence (did the experiment win?). Few retrospect Effort. Effort overruns are the silent killer of portfolios — they consume velocity from other experiments that don't get to ship.

One-line rule: If you can't write down what you'd learn from the experiment losing, it's not an EVOI bet — it's a hopeful Bias-for-Action with worse odds.

Red-Flag Indicators

• Picks top 3 by RICE without overlay or calibration adjustment
• No mention of EVOI or swing-bet allocation
• Doesn't ask about team's prior Confidence calibration
• Treats engineering Effort estimate as ground truth without calibration audit
• Uses "we have to ship X for OP1" as binding constraint without exploring outcome-vs-method distinction

Strong-Answer Markers

• Decomposes Confidence into prior-evidence + detectability
• Reserves one slot per quarter for EVOI swing bet
• References historical hit-rate calibration and Effort calibration adjustments
• Names dependency-graph reasoning across candidates
• Defends portfolio in OP1-narrative format with tenets and risks

Scenario AML-03 — Hypothesis Design & Sample-Size Discipline

Opening Question

Q: You're shipping a cross-encoder reranker change (the US-MLE-02 system). Your data scientist offline-evaluated it on a 5K replay and got +6% NDCG@10. Your PM says "great, run a 7-day A/B and ship." What do you say?

Round 1 Answer: I say "the offline result is necessary evidence but not sufficient for the launch decision; we need a properly-pre-registered online A/B with proper sample size, and 7 days isn't enough." Specifically: the primary online metric is useful-answer-rate per user over 14 days (not NDCG@10 — see AML-04 on online-offline mismatch). The MDE I'd defend is 3% relative on that metric. Power calc with σ=0.142 and CUPED at 50% variance reduction gives n≈18.4K per arm with O'Brien-Fleming sequential at 4 looks; that's runnable in 14 days at 200K eligible users/day. Runtime is bounded by the 14-day per-user metric horizon, not by sample size. We pre-register hypothesis, MDE, sample size, randomization unit, primary metric, guardrails (latency, CSAT, retention, per-cohort), and α-spending plan in versioned YAML before randomization starts. 7 days is the wrong duration; sequential α-spending is the discipline that survives any peeking the PM will inevitably do.

Round 1 — Surface

Follow-up: Why CUPED? How does it actually work?

CUPED (Controlled-experiment Using Pre-Existing Data) is variance reduction via a pre-experiment user-level covariate. The math:

Y_adjusted = Y - θ · (X_pre - mean(X_pre))
   where θ = Cov(Y, X_pre) / Var(X_pre)

For our reranker A/B, the metric Y is per-user useful-answer-rate over the experimental 14d. The covariate X_pre is per-user useful-answer-rate over the immediately preceding 14d. Users who were high-engagers before will be high-engagers during; users who were low-engagers will be low-engagers. Subtracting the regression of Y on X_pre removes most of the per-user variance unrelated to treatment, leaving a tighter estimate of treatment effect.

Empirically, CUPED on per-user engagement metrics reduces variance by 30-60%; we measured 50% on this metric in pre-experiment baselining. Translating to sample size: σ² halves, required n halves, the experiment runs at half the population for the same power. That can collapse a 3-week experiment into 12 days.

The cost: zero. CUPED is a closed-form adjustment computed at analysis time. The risks: (1) covariate must be measured before randomization (otherwise you contaminate); (2) covariate must be correlated with outcome (if not, no variance reduction); (3) the adjustment doesn't change the unbiasedness of the treatment effect estimate — only its variance. When all three are met, CUPED is free power.

Round 2 — Push Harder

Follow-up: Your PM says "fine, 14 days, but I'm going to check the dashboard daily — and if we're significant on day 5, I want to ship." What do you do?

Two responses in sequence:

First, install the sequential α-spending plan. PMs will peek; the discipline is to make peeking statistically valid. I use O'Brien-Fleming with 4 pre-declared looks (day 4, 7, 11, 14) and α at each look of (0.0001, 0.001, 0.01, 0.05). The PM can check the dashboard daily, but the decision rule is: ship early only if the look-α is breached. On day 5, even if p<0.05 on a naive t-test, the OBF threshold is 0.001 — we don't ship. The mechanical rule prevents us from shipping on noise that the PM will inevitably perceive as a "win."

Second, install dashboard discipline. The dashboard the PM sees has only the OBF-adjusted significance level shown — not the raw p-value. If the p-adjusted is below the look threshold, the dashboard turns green (ship-eligible); otherwise it stays yellow. The PM can peek as much as they want; the adjusted view is what matters. I don't try to prevent peeking; I make peeking statistically harmless.

What I'd refuse to ship without: the OBF YAML signed off, the dashboard showing only adjusted significance, and an explicit discussion with the PM about what the look thresholds are and why. Without that conversation, the first time the dashboard goes green at look 1 (which requires extreme effect), the PM will ship; the second time the dashboard isn't green at look 4 (the actual designated stop), the PM will be confused and angry. Pre-conversation prevents the mid-experiment fight.

Round 3 — Squeeze

Follow-up: Walk me through the sample-size math. Show your work.

Numerical calculation:

Baseline metric: useful-answer-rate per user over 14d. Mean μ = 0.183. Std σ_pre-CUPED = 0.142.

MDE: 3% relative. Δ_absolute = 0.183 × 0.03 = 0.00549.

Standardized effect size: d = Δ / σ = 0.00549 / 0.142 = 0.0387.

For a two-sample Welch's t at α=0.05, power=0.80, two-sided:

import statsmodels.stats.power as smp
analysis = smp.TTestIndPower()
n = analysis.solve_power(effect_size=0.0387, alpha=0.05, power=0.80, alternative='two-sided')
# n ≈ 10491 per arm (single-look, fixed-horizon)

For O'Brien-Fleming with 4 looks, sample size inflates ~6%:

n_obf ≈ 10491 × 1.06 = 11120 per arm

For CUPED with 50% variance reduction: σ_post-CUPED² = σ²_pre × 0.5, so σ_post = σ_pre × √0.5. New effect size d' = 0.00549 / (0.142 × 0.707) = 0.0547.

n_post_cuped_obf = analysis.solve_power(effect_size=0.0547, alpha=0.05, power=0.80) * 1.06
# n ≈ 5490 per arm... but this is the post-CUPED-residualized statistic

A subtlety: the OBF inflation factor I cited (1.06) is for raw t-statistics, not for CUPED-residualized statistics. The residualization changes the effective degrees of freedom slightly. I'd round up conservatively: n = 18,420 per arm post-CUPED + OBF, which is what I'd register.

At 200K eligible users/day and 50/50 split, n=18,420 per arm × 2 = 36,840 total → ~3.7K/day → reachable in <1 day. Runtime is bounded by the 14-day per-user metric horizon (each user must accumulate 14 days of metric data), not by sample size.

For the cohort-stratified claim: JP cohort is 30% of traffic, so per-arm JP n = 18,420 needs JP-traffic of 36,840 / 0.30 = ~123K total → ~6 days. Mixed cohort at 15% needs 18,420 / 0.15 = ~123K JP-equivalent → 12 days. Mixed cohort claim is on the edge of the 14-day horizon — I flag this in the pre-registration: "mixed-cohort claim has marginal power; if mixed-cohort metric is between -3% and +3%, treat as inconclusive, not as evidence against H1." Better to flag upfront than negotiate post-hoc.

Round 4 — Corner

Follow-up: Day 11, look 3. The OBF threshold is 0.01. Aggregate p=0.005 (passes). But JP cohort metric is at -2.8% (close to but not breaching the -3% guardrail). What do you do?

I do not stop early. Three reasons:

Reason 1: cohort guardrail trajectory. The JP cohort is at -2.8% with adjusted-α not yet at the level needed for confident veto. But the trajectory over the 4 looks matters: was JP at -1.2% at look 1, -1.8% at look 2, -2.4% at look 3 → -2.8% at look 4? If yes, the cohort is on a worsening path; stopping early on aggregate locks in a cohort regression at -3.0% or worse by day 14. Stopping early hides the cohort regression.

Reason 2: noisy cohort estimates at smaller n. At day 11, JP cohort accumulated 6 days of per-user metric per user (since each user enters with rolling 14d window). JP cohort n is below the per-cohort sample-size requirement. The -2.8% estimate has wide confidence interval. By day 14, n is at requirement and the estimate sharpens — possibly toward -3.5% (real regression) or toward -1.5% (noise). Stopping early on aggregate locks in the noise.

Reason 3: the discipline of the pre-registered plan. OBF is set up so we don't need to make this judgment call live. The plan said: 4 looks, ship at the final look only unless extreme effect is reached at earlier looks. "Aggregate p=0.005 at look 3 with cohort at -2.8%" is not extreme effect on the joint distribution; it's borderline cohort breach with strong primary. The pre-registered plan should have said "stop early on aggregate only if cohort metrics are also clear at -1% or above for all cohorts." If the pre-registration didn't say that, that's a pre-registration deficiency I'd note for next experiment.

What I do: continue to look 4 (day 14). At look 4, with JP cohort at sufficient n, the pre-cohort guardrail decision is the conclusive one. If JP -3.5%, mechanical veto despite aggregate +6%. If JP -2.5%, ship (within threshold). The discipline says: don't ship under cohort uncertainty.

The trap: stopping early because aggregate looks great. The cost: shipping a cohort regression hidden by noise. The mitigation: pre-registered cohort-conditional stopping rules; if not pre-registered, default to running to plan.

Architect-Level Escalation

A1: Design an experiment platform that scales to 100 simultaneous A/Bs across teams. What's it look like?

The platform has six layers:

1. Hypothesis registry — every experiment submits a versioned YAML before randomization. Hash signed by PM + AML Eng + EM. The registry is queryable: "what experiments are running on the reranker surface this week?"

2. Cohort-and-eligibility resolver — central service that, given a user_id, returns which experiments the user is in, with cohort assignments. Handles overlap rules (no user in 2 reranker experiments simultaneously), exclusivity classes, and rollout-percentage gates.

3. Telemetry collector — every treatment-applied event emits (experiment_id, user_id, cohort, metric, value, timestamp) to a unified event stream. Single schema across teams; teams cannot fork their own.

4. Analysis service — pre-registered analysis runs on schedule (per-look) against the telemetry stream. Output is a pre-defined dashboard; teams can't write their own ad-hoc analysis.

5. Sequential-stopping engine — implements OBF, Pocock, Always-Valid stopping plans declared in the YAML. Auto-pauses experiment rollout when stopping criteria met (early-win OR guardrail breach). No human in the loop.

6. Calibration audit pipeline — quarterly job that retrospects every shipped experiment: did predicted Confidence match realized outcome? did predicted Effort match realized? Outputs aggregate calibration deltas per team.

The systems insight: at 100-experiment scale, ad-hoc analysis is the bug. When 5 teams each write their own SQL queries against their own metric tables, you get 100 different definitions of "useful-answer-rate" and no cross-experiment comparability. The platform's job is to enforce shared definitions at telemetry-emit time, so that downstream all analysis is automatic.

A2: A senior PM at another team wants to run a "quick A/B for 3 days, just to see directional signal." What do you say?

I say "directional signal at n=3-days isn't signal — it's noise that we'll act on as if it were signal." And I say it carefully, because the conversation has structural traps:

The trap: agreeing to "directional only — we won't ship from this." Within 3 days, the PM (or someone above them) will see the directional result and either (a) ship anyway, citing positive direction, or (b) ship the opposite of a clear-but-noisy negative direction. "Directional only" is a verbal commitment that decays fast.

The conversation:

"What's the actual decision you want to make on day 3? If the answer is 'whether to invest more eng-weeks,' that's a portfolio decision (AML-02), not an A/B decision; we don't need an experiment, we need RICE scoring. If the answer is 'should we ship,' a 3-day A/B is undersized. Let's do one of two things: (a) skip the experiment and decide based on offline evidence + portfolio reasoning; or (b) run a properly-sized experiment, even if it's 14 days. The 3-day version is worse than either alternative."

LPs invoked: Insist on the Highest Standards (no shortcut testing), Are Right A Lot (3-day A/B isn't right; the alternatives are), Have Backbone (push back on the request).

A3: When does a t-test fail you, and what do you use instead?

Three failure modes:

1. Heavy-tailed metric. Per-user dollars-spent, time-on-site, session-duration are typically heavy right-tailed. A small number of power-users dominate variance; the t-test under-estimates variance and over-claims significance. Solutions: log-transform if strictly positive; or use Mann-Whitney U (rank test) which is robust to tail; or trim/winsorize at the 99^th percentile and t-test the trimmed.

2. Equal-variance assumption. scipy.stats.ttest_ind defaults to equal-variance. In real chatbot A/Bs, treatment and control arms typically have different variances (treatment changes user behavior, which changes per-user metric variance). Use Welch's t (equal_var=False) — same test mathematically, doesn't assume equal variance. Always Welch.

3. Ratio metrics. If the metric is useful_turns / total_turns per user (a ratio), per-user-ratio averaging biases the variance estimate. The right approach is the delta method: estimate variance of the ratio analytically:

Var(X/Y) ≈ (μ_X² / μ_Y²) × ( σ²_X/μ²_X + σ²_Y/μ²_Y - 2·Cov(X,Y)/(μ_X·μ_Y) )

For binary metrics (CTR, conversion rate), the t-test is OK at large n but better is the two-proportion z-test or chi-squared.

When in doubt: bootstrap. Resample the per-user metrics 10,000 times with replacement; compute treatment-vs-control delta on each; the empirical 95% CI is your interval. Bootstrap is robust to most distributional issues at the cost of compute. For chatbot-scale A/Bs, this is cheap.

Intuition Gained — AML-03

The core insight: Pre-registration is the most defensible artefact an Applied ML Engineer produces. It costs friction; it pays in defensibility for years.

Mental model to carry forward:

"Write down the hypothesis, MDE, sample size, primary metric, guardrails, and stop rule before randomization starts. Sign and version-control. Refer to it when results arrive."

The hidden failure mode: Sequential α-inflation from peeking. PMs always peek. The discipline is OBF or Always-Valid α-spending so that peeking is statistically harmless.

One-line rule: Welch + CUPED + OBF + delta method + bootstrap. These five tools cover ~95% of chatbot A/B issues. Master them, and most A/B problems have known answers.

Red-Flag Indicators

• Accepts "7-day A/B" without sample-size calc
• Uses default scipy.stats.ttest_ind (equal-variance)
• Doesn't mention CUPED or variance reduction
• No sequential α-spending plan; will peek without protection
• Computes sample size for aggregate, ignores cohort under-power
• Treats NDCG@10 offline as the online primary

Strong-Answer Markers

• Pre-registered YAML with hash + signoff
• Welch's t + CUPED + O'Brien-Fleming + per-cohort sample-size
• Names ratio-metric delta method, heavy-tail considerations
• Defends MDE choice with cost-benefit reasoning
• Explicit pre-cohort stopping rules; no early-stop under cohort uncertainty

Scenario AML-04 — Online/Offline Metric Decoupling

Opening Question

Q: Your reranker change just shipped. Offline NDCG@10 was +5%. After 14 days online, CTR has moved from 11.21% to 11.27% — not statistically significant. Your PM asks "is the model wrong, or is the metric wrong?" What's your answer?

Round 1 Answer: The model is probably right; the offline metric is probably mismeasuring user behavior. Five named root causes for offline-online correlation collapse: (1) selection bias in offline eval set, (2) label leakage, (3) distribution shift between eval and prod, (4) metric proxy mismatch, (5) optimizer goodharting. For our case I'd diagnose each in order. My prior: causes #4 and #3 dominate. Cause #4 because we have 70% mobile traffic and NDCG@10 weights all 10 positions, but mobile users see positions 1-3; the offline gain may be at positions 5-7 where users don't look. Cause #3 because the golden eval set is 8 weeks old and catalog turnover since is ~14%. The fix is not to retrain the model. The fix is to rebuild the offline harness — replace NDCG@10 with NDCG@3 for mobile, refresh eval weekly with stratified sampling, add IPS counterfactual eval as secondary metric. After harness rebuild, re-run the same model change against the new harness; if NDCG@3 shows +0.5% (matching online), correlation is restored and the team trusts future experiments.

Round 1 — Surface

Follow-up: How do you actually measure offline-online correlation, and what's the threshold?

Three concrete steps:

1. Maintain a long-running record. Every shipped experiment writes (experiment_id, offline_Δ, online_Δ, surface) to a single database. After 6+ experiments on the same surface (e.g., reranker), correlation analysis becomes meaningful.

2. Compute rolling Pearson over the last 6-12 experiments. Not just the 90-day correlation on absolute metrics — the delta-vs-delta correlation across experiments. If experiment A's offline-Δ was +5% and online-Δ was +4%, and experiment B's offline-Δ was -2% and online-Δ was -1.5%, those are correlated. Pearson on (offline-Δ, online-Δ) pairs across N experiments.

3. Threshold: > 0.6 trustworthy, 0.5-0.6 yellow zone, < 0.5 collapse — freeze model shipping for harness rebuild. The threshold isn't theoretical; it's a calibration that says "if direction agreement is below this, our offline metric is no longer reliable enough to drive ship/no-ship decisions."

Practical numbers from the MangaAssist project memory: the team's last 90 days had 6 experiments on the reranker surface; rolling Pearson was 0.41 (below 0.5). The previous 60 days were 0.71. The drop is the alarm. The right response is to investigate, not to ship.

The most-skipped step: most teams don't track this correlation; they only notice it when a launch is offline-positive and online-flat. By then, they've shipped (or not shipped) several experiments based on a metric they can't trust. The correlation tracker is the leading indicator that catches the collapse before it bites multiple launches.

Round 2 — Push Harder

Follow-up: You diagnose cause #4 (proxy mismatch). The PM says "OK — let me retrain the reranker to optimize NDCG@3 instead of NDCG@10. We'll re-run the A/B." What do you say?

I say "that's the wrong fix — and even if it were the right fix, the re-trained model isn't necessarily better than the current one on the new metric."

Three reasons to push back:

1. Retraining doesn't fix selection bias or distribution shift. If causes #1 and #3 are also contributing (which they likely are — they often co-occur with #4), retraining against NDCG@3 is racing in the new wrong direction. The fix has to address the harness, not the model. After harness rebuild, the current model can be re-evaluated under NDCG@3 and might show +0.5% — matching the online signal — without any retraining.

2. Goodharting the new metric. If I retrain to optimize NDCG@3, the new model is guaranteed to win NDCG@3 offline. But NDCG@3 itself may not be a faithful proxy for online behavior either. The discipline is to evaluate the existing model against NDCG@3 first; if NDCG@3 correlates with online and the existing model wins on NDCG@3, the existing model is good. Don't retrain until I've validated the metric.

3. The portfolio cost. Retraining the reranker is 8 engineer-weeks of work. That's an entire portfolio slot. If the harness fix takes 4 weeks and shows the existing model already wins under NDCG@3, I've saved 4 weeks of engineering for a different experiment. Shipping the harness fix first is the velocity-preserving move.

The right order: (1) rebuild harness (4 weeks); (2) re-evaluate current and recent experiments against new harness (1 week); (3) verify offline-online correlation has restored on new harness via replay (1 week); (4) only THEN consider retraining if the existing model genuinely under-performs the new metric. Most of the time, step 3 reveals that the existing model was fine; the issue was the metric the team was looking at.

Round 3 — Squeeze

Follow-up: Walk me through IPS — how does it correct for selection bias, and what's the variance trade-off?

Inverse Propensity Scoring corrects for the selection bias that arises when offline eval data is click-logged — i.e., the eval data only contains queries-and-items where the production system showed the item to a user, and the user (sometimes) clicked.

The intuition: items that were rarely shown by the production system have rare exposure; their click counts are unreliable as quality signals because most never-shown items might also be high-quality. IPS re-weights observations by 1/π(item|query) where π is the propensity of showing that item to that query in production. Items shown 1% of the time get up-weighted 100×; items shown 100% of the time get weight 1.

The math: counterfactual estimator of policy value V(π_new):

V̂_IPS(π_new) = (1/n) · Σ_i [ π_new(a_i | x_i) / π_logged(a_i | x_i) · r_i ]

The variance trade-off is the classical IPS pain: variance grows with max(π_new / π_logged). If the new policy strongly prefers items the logging policy rarely showed, the importance weights blow up. Three practical mitigations:

1. SNIPS (self-normalized IPS): divides by Σ (π_new/π_logged) instead of n. Slightly biased but much lower variance. Practical workhorse.
2. Clipping: cap importance weights at some M (e.g., 100). Bounds variance at the cost of bias on under-served items.
3. Doubly-Robust estimators: combine IPS with a model-based reward predictor. If either the propensity model or the reward model is right, the estimator is unbiased. Lower variance than pure IPS in practice.

For MangaAssist offline-eval, I'd default to SNIPS with light clipping — gives most of the bias correction benefit at manageable variance. The Applied ML Engineer's role isn't to derive these estimators; it's to know they exist, ask for them when click-logged eval is being used, and verify the variance isn't blowing up.

Round 4 — Corner

Follow-up: After 4 weeks of harness rebuild, you replay 6 prior experiments through the new harness. Pearson correlation between new-harness offline-Δ and online-Δ is 0.55 — above your threshold but barely. The team wants to declare success and resume shipping. What do you say?

I say "0.55 is above threshold but not by much; let's resume shipping but with explicit conservatism for the next 4 experiments." Specifically:

1. Treat 0.55 as 'restored but fragile.' The threshold is 0.5 for collapse, 0.6 for trustworthy. 0.55 is in the yellow zone. Resume shipping, but for the next 4 experiments, require both offline-Δ and online-Δ to clear MDE before declaring win. This is conservative — it costs us power on borderline experiments — but it protects against the harness still being slightly miscalibrated.

2. Continue tracking rolling Pearson. If after 4 more experiments the rolling Pearson is consistently in the 0.6+ range, drop the conservatism. If it drops back below 0.5, freeze again — the rebuild was insufficient, do another iteration.

3. Investigate the 6 replay experiments individually. A 0.55 average could be six experiments at 0.55, or three at 0.9 and three at 0.2. If the variance is high, the harness works for some kinds of changes and not others; that's a structural issue worth understanding before resuming.

What I'd refuse: resuming shipping at full confidence at 0.55. The cost of false confidence is shipping a noise winner that the new harness misses; the cost of conservatism is 4-8 weeks of slightly slower shipping. Conservatism is recoverable; false confidence isn't.

The trap: declaring victory after a 4-week rebuild because the team is exhausted and wants to resume. Discipline says: the threshold is the threshold; barely-passing is barely-passing; act accordingly.

Architect-Level Escalation

A1: Design an offline evaluation harness that's robust to all five causes of correlation collapse. What does it look like?

The harness has five layers, each addressing one cause:

1. Population-stratified eval set, refreshed weekly. Sampled from the last 7-day production query distribution. Stratified across cohort dimensions (locale, device, tenure). Addresses cause #3 (distribution shift).

2. Time-strict train/eval split. Eval set is strictly after training cut-off; no temporal contamination. Features derived from data after training cut-off are excluded. Addresses cause #2 (label leakage).

3. IPS-corrected counterfactual estimator. SNIPS with light clipping for click-logged subset. Addresses cause #1 (selection bias).

4. Position-weighted offline metric calibrated to user behavior. NDCG@3 for mobile-skewed surfaces; NDCG@10 for desktop. Addresses cause #4 (proxy mismatch).

5. Diverse metric portfolio. Primary metric (e.g., NDCG@3) + adversarial detector (e.g., click-bait classifier) + LLM-judged factuality + behavioral correlate (e.g., session length proxy). Promotion requires win on primary AND no regression on secondary. Addresses cause #5 (goodharting).

Plus a meta-layer: rolling Pearson correlation tracker between new-harness offline-Δ and observed online-Δ, updated quarterly. If correlation drops below threshold, harness rebuild is triggered.

The systems insight: single-metric offline evaluation will eventually goodhart. The harness's job is metric diversity such that no single metric is sufficient to win. A model that wins on all five metrics is robust; a model that wins on one is suspicious.

A2: Defend the 4-week ship-freeze in a Director-level review. What do you say?

I'd structure the defense as a six-pager:

Page 1: "We declined to ship this quarter despite +5% offline NDCG gain. Offline-online correlation has dropped to 0.41 (90-day rolling), below our 0.5 threshold. Offline gain is no longer a trustworthy predictor of customer behavior. We are rebuilding the offline harness in Q4."

Page 2: "The cost of waiting is one quarter of velocity. The cost of not waiting is shipping noise winners against a broken metric for the next year. Concretely: in the last 6 months, we've had 2 experiments that won offline and went flat online. If correlation remains at 0.41, every future experiment has a similar miss-rate. The next year would ship 4-8 noise winners."

Page 3: "Investments: 4 weeks engineering for harness rebuild, 1 week for replay validation, 1 week for resumption planning. Expected restoration of correlation by end of Q4. Resumption of model-shipping cadence in Q1."

Page 4: "Tenets: (1) we ship customer-impact, not offline metrics; (2) if offline metric does not predict online behavior, we rebuild offline metric; (3) we do not race in the wrong direction faster."

Page 5: "Risks: harness rebuild may not restore correlation (mitigation: iterative; budget 8 weeks total); team morale during ship freeze (mitigation: explicit communication; team members shift to harness work); leadership impatience (mitigation: this document)."

Page 6: FAQ — "Can we ship with the broken metric and validate online?" Answer: no, because shipping is irreversible; we can't roll back a quarter of customer harm.

LPs invoked: Are Right A Lot (don't race wrongly), Insist on Highest Standards (don't ship metrics we don't trust), Long-Term Thinking (4-week cost vs year of wrong direction), Customer Obsession (ultimately it's about customer behavior, not offline scores).

A3: A team adjacent to yours has correlation 0.7 and is shipping fast. They claim "you're being too conservative." What do you say?

I say "different surfaces, different correlation reliability. They might be right for their surface; I'm right for mine."

Specifically:

"Their surface might have a metric that genuinely correlates with online behavior — recall@k for retrieval is often more correlated with user behavior than NDCG@k for ranking. Our surface has a known disconnect (mobile users seeing positions 1-3 only). Our threshold isn't conservative; it's calibrated to our surface's correlation history. Their threshold may be appropriate for theirs."

"That said, I'd be open to evidence that I'm being too conservative on this surface. Concretely: if my last 6 shipped launches have all maintained their lift at 28-day retention, my offline-online correlation is real and I can shave the threshold. If my last 6 launches showed lift fade at week 4, the threshold should be tighter. Calibration is empirical, not a vibe."

The trap: getting peer-pressured into shipping by a team that doesn't share my correlation history. Different surfaces have different reliability; the threshold I use is mine, not theirs. The discipline is: defend the threshold with empirical evidence (calibration audit), be open to evidence-based revision, ignore vibes-based revision.

Intuition Gained — AML-04

The core insight: When offline says win and online says nothing, the model is probably right and the metric is probably wrong. Rebuild the metric, not the model.

Mental model to carry forward:

"Offline metrics are proxies. Track the correlation between offline-Δ and online-Δ over many experiments. If the correlation drops, the proxy stopped working — fix the proxy."

The hidden failure mode: Goodharting. The training loss is the offline metric; the offline metric becomes the target; the target ceases to measure customer behavior. Mitigation is metric diversity, not a "better single metric."

One-line rule: When correlation drops below 0.5, freeze model changes for 4 weeks and rebuild the harness. Cost is real; benefit is permanent.

Red-Flag Indicators

• Says "retrain the model to optimize NDCG@3" without rebuilding harness first
• No tracking of offline-online correlation; only notices it when a launch fails
• Treats single offline metric as the truth
• Doesn't know IPS / SNIPS or selection bias correction
• Won't freeze shipping under correlation collapse

Strong-Answer Markers

• Names all 5 root causes; diagnoses by elimination
• "Rebuild the harness, not the model" as a default response
• Tracks rolling Pearson correlation; threshold-based freeze rule
• IPS/SNIPS for click-logged eval; metric diversity
• Defends a ship-freeze in OP1 narrative

Scenario AML-05 — Business-KPI Guardrails for Promotion

Opening Question

Q: Your reranker change shows +4.5% NDCG@10 (significant, p<0.01) but CSAT regressed -1.4% (significant, p<0.05). The PM says "CSAT is a noisy survey metric, ship it." What do you do?

Round 1 Answer: I enforce the mechanical veto. CSAT was pre-declared as a guardrail with -1% threshold; -1.4% breaches. The launch does not ship. The response to the PM: "CSAT is the user telling us, in their words, that the new ranking feels worse. The primary metric is a behavioral proxy; CSAT is a stated preference. They disagree at p=0.02. When they disagree, we take the user's word. We pre-registered the threshold; we enforce it mechanically. Shipping a model that the user is telling us they don't like, on the bet that we're wrong about CSAT, is not a bet I'd make." The next move is diagnostic: why did CSAT regress despite the engagement lift? Likely: the new ranking surfaces titles users find emotionally heavier, or it's more aggressive on click-bait. Both are diagnosable in offline analysis. Once diagnosed, retrain with the corrective signal (e.g., tone-preference signal from US-MLE-03 ABSA aspects) and re-run the experiment.

Round 1 — Surface

Follow-up: How do you set the -1% CSAT threshold? Why -1% and not -0.5% or -2%?

Threshold setting is a negotiation, not a guess:

The framework: cost-of-regression × probability-of-detection. CSAT regressing 1% means roughly 1% fewer 5-star ratings or proportional shift toward lower ratings. Industry calibrations: 1% CSAT regression typically correlates with 0.3-0.5pp retention regression at 28-90 day horizon. At MangaAssist scale, that's a measurable revenue impact (LTV × user-base × retention-Δ).

So the question becomes: what CSAT regression is too small to matter (false-positive territory, where survey noise dominates) and too big to allow (real customer harm)?

Too small: CSAT survey response rate is ~12%; noise floor on 14-day rolling CSAT is ~0.3% relative. -0.5% is at the edge of detectable; treating it as veto would create a high false-positive-veto rate.

Too big: -2% CSAT correlates with ~0.6-1.0pp retention regression. That's already large customer impact.

The sweet spot: -1% relative. Above noise floor (false positive rate manageable); below the level where customer impact becomes clearly damaging. Caveat: for high-stakes launches (multi-quarter-retention-impact), tighten to -0.5%; for low-stakes (small surface, easy rollback), loosen to -1.5%.

The threshold is negotiated with PM (cost-of-regression), Trust & Safety (cohort-side regression risks), and Eng Manager (operational implications). Once signed in the YAML, the veto is mechanical. Threshold-setting is the work; enforcement is automatic.

Round 2 — Push Harder

Follow-up: You veto. PM escalates to Director. Director says "I'd like the team to be more decisive — not every regression is real. Ship it." What do you do?

I push back, structured:

"I hear you. The discipline is the thing that distinguishes us from teams that ship every borderline launch and watch their customer trust erode. The CSAT regression at -1.4%, p=0.02, on a pre-declared threshold of -1%, is not a borderline call; it's a clear breach. I can show you the pre-registered YAML signed by PM, EM, and me before the experiment started."

"The decision rule wasn't 'ship every model that wins on engagement.' It was 'ship if engagement wins AND no guardrail breaches at adjusted-α.' The mechanical action is veto. I am not going to override our own pre-registration; doing so means our pre-registration framework collapses, and the team's credibility with all future launches collapses with it."

"I'd like to propose: I will document the diagnostic for why CSAT regressed (likely: ranking surfaces tonally-heavy titles); we retrain with the corrective signal; we re-run the experiment in 6-8 weeks. The current launch slips; the discipline holds. If I'm wrong about the CSAT signal — if the diagnostic shows it's noise — I commit to documenting that and we revisit."

"If the decision is still 'ship,' I'll comply, but I'll write a memo documenting that we shipped against pre-registered thresholds, and I'll request that we publicly track the 28-day retention impact as a calibration check. If retention regresses by the 0.3-0.5pp my prior says it will, we update our threshold-setting framework. That's how Disagree-and-Commit works in practice: I disagree, document; you decide; we hold ourselves to a transparent calibration loop."

The trap: caving silently. If the Director overrides and the team ships, the discipline collapses across all future launches. The discipline says: I disagree, I document, I commit if overridden, I track the retrospective so the org learns. Backbone with grace.

Round 3 — Squeeze

Follow-up: Show me the family-wise error rate math. You have 5 guardrails. How does that change your veto threshold?

Five guardrails, each tested at α=0.05:

P(at least one false breach in clean experiment) = 1 - (1 - 0.05)^5 = 1 - 0.7738 = 0.2262

Almost a 1-in-4 chance of false-veto in a clean experiment. That's high — it means a quarter of clean launches would be erroneously vetoed.

Three correction options:

Bonferroni: divide α by k. Each guardrail tested at α = 0.05/5 = 0.01. False-veto rate drops to ~5% (mathematically: P = 1 - (0.99)^5 ≈ 0.049). Conservative — legit breaches at α=0.04 don't fire. Loses power.

Holm-Bonferroni: order p-values smallest to largest; test the smallest at α/5, next at α/4, etc. More powerful than Bonferroni; keeps overall α at 0.05 while losing less power.

Intersection-Union Test (IUT): each guardrail tested at unadjusted α=0.05; pass requires ALL pass. P(all pass | clean) = (0.95)^5 ≈ 0.774 → 22.6% false-no-launch rate. Symmetric but with asymmetric error costs: false-no-launch is recoverable (re-run experiment); false-ship is unrecoverable (customer harm).

For our reranker case, I'd default to IUT for guardrails. Reasoning: false-veto cost (re-run experiment, slip 6 weeks) is much smaller than false-ship cost (customer harm, retention regression at scale, trust erosion across the user base). Asymmetric error costs favor conservative-on-ship.

For our specific veto decision: CSAT breached at p=0.02 (well above unadjusted α=0.05 IUT threshold; below Bonferroni-adjusted 0.01). Under IUT, the veto is unambiguous. Under Bonferroni, it's borderline. I'd register the IUT decision in the YAML up front; that takes the borderline judgment off the table.

The Applied ML Engineer's framing for FWER: it's not "what's the most powerful test"; it's "what error pattern can we live with given asymmetric costs." For guardrails, conservative-on-ship is right.

Round 4 — Corner

Follow-up: After 6 weeks of investigation, you find the CSAT regression was real (specific session types where the new ranking surfaces tonally-jarring titles). You retrain with the ABSA tone-preference signal, re-run, and the new model shows +5.2% useful-answer-rate, CSAT -0.3% (within threshold). You ship. Six months later, post-launch retention review: aggregate retention preserved, but the specific cohort that originally triggered the CSAT regression has retention -0.6%. Ship the launch again, accept the cohort regression?

No. The veto re-fires, this time on the cohort guardrail.

Three considerations:

1. What does "specific cohort" mean here? If it's a stratified cohort we pre-declared (locale, tenure, device), the per-cohort guardrail in our YAML covers it (-3% relative cohort threshold). -0.6pp absolute on retention may or may not breach -3% relative on the cohort's primary metric. Need to check the relative number.

2. If it doesn't breach the pre-declared threshold but is still concerning: that's a calibration update for next time. The cohort guardrail at -3% relative was set to be conservative; -0.6pp absolute on retention is real customer harm even if statistically below threshold. I'd flag this in the post-launch retrospective: "our cohort threshold of -3% relative on primary metric missed a real retention regression at -0.6% absolute. Tighten threshold to -2% relative for next launch on this surface."

3. If we did ship the launch with the cohort regression visible: revisit immediately. Either (a) the cohort regression is small enough to accept and we communicate to the affected cohort with mitigation (e.g., manual ranking override for that cohort), or (b) the regression is large enough that we ship a hotfix to roll back the launch for that cohort specifically, accepting the engineering cost.

The discipline: post-launch monitoring is the second guardrail. If something the pre-registered guardrails missed appears in production, we treat it like an in-experiment guardrail breach. The mechanical action is investigate-and-mitigate, not "we already shipped, deal with it."

What I'd refuse: ignoring the cohort regression as "out of scope of the launch decision." The role doesn't end at ship.

Architect-Level Escalation

A1: Design a guardrail framework for an entire ML platform — 8 systems, 4 teams, 50 launches per year. What does it look like?

The framework has five layers:

1. Cross-platform guardrail catalog. Standard guardrails every launch must check (CSAT, retention, latency, spam-flag, cohort fairness) plus surface-specific guardrails (e.g., factuality for RAG; diversity for recsys). Catalog versioned in git.

2. YAML-driven pre-registration. Every launch submits a YAML referencing the catalog with thresholds. The platform refuses to randomize without a signed YAML hash.

3. Auto-stopping engine. Reads thresholds; auto-pauses experiment when guardrail breached at adjusted α. No human in the loop for the pause itself.

4. Override workflow with audit trail. If a team wants to ship despite a guardrail breach, the override requires a written incident-style document signed by Director, plus a commitment to track post-launch impact for 90 days.

5. Quarterly calibration audit. Retrospective: of the launches that vetoed, how many would have caused harm if shipped? Of the launches that shipped, how many had post-launch issues? Calibrate thresholds based on observed false-positive and false-negative rates.

The systems insight: guardrails are an institution, not a per-launch decision. Each individual launch decision is small; the cumulative effect of consistent guardrail enforcement is the team's credibility with leadership and the platform's trustworthiness with customers. The institution is what we're protecting.

A2: Walk through the OP1 narrative for "we declined to ship 3 of 12 launches this quarter." How do you frame it to leadership?

Six-pager outline:

Page 1 — The discipline: "Of 12 launches this quarter, we shipped 9 and declined 3 due to guardrail breaches. The 3 declines were vetos on pre-registered thresholds: CSAT (-1.4%), retention cohort regression (-3.2% on JP cohort), and family-wise error correction failure (FWER 0.07). All decisions are documented in the experiment registry."

Page 2 — The yield: "Of the 9 shipped, 8 maintained their lift at 28-day post-launch (89% retention rate). The 3 declined: by retrospective analysis, would have produced 0.4-0.8% retention regression each, costing $X million per year aggregated. NPV of the discipline is $X million."

Page 3 — The cost: "The 3 declines cost ~9 weeks of engineering (3 launches × 3 weeks of investigation + retrain). Two of the 3 are on track to re-ship in Q4 with corrective signal. One is permanently shelved (the JP cohort regression couldn't be resolved without a major recsys retrain, which is in Q5 portfolio)."

Page 4 — The institution: "Pre-registered guardrails are the antibody against shipping launches that look like wins but are losses. The discipline is multi-quarter; over the last 4 quarters, we've vetoed 11 launches and shipped 38. Of the 38 shipped, 34 maintained lift (89% rate); of the 11 vetoed, retrospective analysis shows 9 would have caused real harm. Pattern matches expected calibration."

Page 5 — Tenets: pre-declared guardrails, mechanical enforcement, override-with-audit, calibration-loop.

Page 6 — FAQ: "Why not negotiate guardrails post-hoc?" — because that's how guardrails-by-political-weight emerges and the institution collapses. "What if a launch has high strategic value?" — the override workflow with Director sign-off and 90-day tracking is the path. "How do we know we're not over-conservative?" — the calibration audit; if false-veto rate is high, tighten thresholds.

LPs: Earn Trust (institution credibility), Insist on Highest Standards (mechanical enforcement), Long-Term Thinking (multi-quarter discipline), Customer Obsession (vetoing on customer-meaning, not customer-behavior), Have Backbone (vetoes against political pressure).

A3: When does the guardrail framework break? What's the failure mode?

Three failure modes:

1. Threshold inflation over time. Pre-declared thresholds get loosened, launch by launch, when teams want to ship. The institution silently weakens. Mitigation: threshold changes require Director sign-off + recorded justification; thresholds can only be loosened, not tightened, between Q's via routine governance; tightening or material loosening requires explicit committee review.

2. Override abuse. Override workflow becomes the default. "We'll just override CSAT this time." Mitigation: override frequency is tracked; if a team overrides > 10% of launches, the framework is failing for that team and needs investigation. Override-rate becomes a team-health metric in OP1 review.

3. Surface-specific gaming. Teams structure experiments to avoid certain guardrails (e.g., narrow scope to a surface where a certain guardrail doesn't apply). Mitigation: cohort-fairness guardrail (Primitive 6) is universal and applies to every surface; impossible to scope around. Plus: surface-specific guardrails are mandatory for known-failure-mode surfaces (e.g., RAG must check factuality; recsys must check diversity).

The meta-failure mode: the framework is institutional. If leadership signals that vetoing is bad behavior — "you should be more decisive" — the framework collapses regardless of mechanical design. The Applied ML Engineer's role is to defend the framework upward; if leadership won't have the framework, the role's value evaporates. Have Backbone is the LP that maintains the framework against political pressure; if the institution doesn't support backbone, no mechanical system substitutes.

Intuition Gained — AML-05

The core insight: Guardrails are an institution. The mechanical veto is the institution's antibody against shipping launches that look like wins but cost trust.

Mental model to carry forward:

"Pre-declare. Sign in YAML. Enforce mechanically. The veto isn't a per-launch decision; it's the institution holding."

The hidden failure mode: Threshold inflation and override abuse. Over time, the framework weakens unless governance maintains thresholds and audits override frequency.

One-line rule: Customer-meaning trumps customer-behavior. CSAT regression with engagement lift is the user telling us they don't like it. Take their word.

Red-Flag Indicators

• Negotiates the guardrail post-hoc ("CSAT is noisy, ignore it")
• No FWER correction across multiple guardrails
• Caves under PM or Director pressure
• Treats guardrail veto as "too conservative"
• No post-launch monitoring as second guardrail

Strong-Answer Markers

• Pre-registered YAML with hash + signoff
• IUT or Bonferroni correction explicit
• Mechanical veto + override-with-audit workflow
• Defends the institution upward, not just the per-launch decision
• Calibration audit on guardrail effectiveness over multiple quarters

Scenario AML-06 — Cohort Fairness & Locale Stratification

Opening Question

Q: Aggregate metrics on your reranker change: +2.8% CTR (sig), +0.3% CSAT (sig). Stratified by locale: EN +4.1%, mixed +2.5%, JP -8.2% on CTR. JP is 30% of traffic. What's your call — ship, abort, redesign?

Round 1 Answer: Abort + redesign. The JP cohort breach (-8.2% on a cohort guardrail of -3%) is a mechanical veto regardless of aggregate. JP is strategically critical: ap-northeast-1 data residency, primary catalog locale, second-most-engaged cohort by tenure. Two paths: (A) JP-stratified retraining with reweighted loss + JP-specific embeddings + JP-locale-aware cold-start, ~4 engineer-weeks; or (B) ship EN-only with JP carve-out flag, 1 week. I recommend A. A carve-out is a public commitment to second-class treatment of JP users — it erodes trust over many launches. The 4 weeks of velocity is paid this quarter to avoid 4 quarters of erosion. Re-run after retrain; expect EN +3-4%, JP +0-2% (parity floor); ship from there.

Round 1 — Surface

Follow-up: How do you size sample so the cohort estimate at -8.2% is reliable, not noise?

The cohort estimate is reliable only if the cohort sub-experiment is properly powered.

For the JP cohort to detect a 3% relative MDE on the primary metric:

n_per_arm_jp = 18,420 (same as aggregate, since per-cohort MDE is also 3%)
total_jp_users_needed = 18,420 × 2 = 36,840
jp_share_of_traffic = 0.30
total_users_for_jp_powered = 36,840 / 0.30 = 122,800

At 200K users/day eligible, JP-powered claim is reachable in ~6 days. By day 14 of the experiment, we have 280K JP users observed — well above the 122K needed. The -8.2% estimate has tight confidence interval at this n.

Compare to mixed-cohort claim: mixed is 15% of traffic, so 36,840 / 0.15 = ~245K total → 12 days. Mixed-cohort claim is on the edge of feasible within the 14-day window. I'd flag in pre-registration: "mixed cohort claim has marginal power; treat between -3% and +3% as inconclusive."

Sample-size adequacy is a pre-registration discipline, not a post-hoc check. If we'd registered for aggregate-only, JP cohort estimate at -8.2% might be noise (under-powered), and we couldn't responsibly veto on it. Pre-registering for the cohort-stratified claim from day 1 is what makes the cohort veto credible.

Round 2 — Push Harder

Follow-up: PM says "JP is only 30% of traffic. The -8% cohort regression × 30% traffic share = -2.4% impact. The aggregate is +2.8%, so net effect is +0.4%. Ship and accept the cohort regression — the aggregate still wins." How do you respond?

The math is wrong, and the framing is worse. Three counter-points:

1. The math is wrong because effects don't compose linearly across cohorts. The aggregate metric already includes the JP cohort regression; the +2.8% aggregate is computed across all cohorts including JP at -8%. The PM is double-counting: subtracting the JP regression from an aggregate that already absorbed it gives a number that doesn't mean anything. The right way to think about it: aggregate +2.8% reflects EN +4.1% (55% weight) + mixed +2.5% (15% weight) + JP -8% (30% weight). Math: 0.55×4.1 + 0.15×2.5 + 0.30×(-8) = 2.26 + 0.375 - 2.4 = +0.235. (My approximate weighted average; aggregate +2.8% suggests the actual numbers are slightly different but the sense is the same.) The aggregate is already the weighted sum.

2. The framing is worse because customers are not interchangeable. Even if the math netted positive, JP customers don't compensate for EN customer wins. A JP user whose retention erodes by 1% over a year is not recovered by an EN user whose retention grows by 1.3%. Customers are individuals, not aggregate. The aggregate metric is a convenience for measurement; the customers it averages over are not interchangeable.

3. Strategic context matters. JP is strategically critical for MangaAssist — primary catalog locale, ap-northeast-1 data residency, the "right answer" for the manga market. Accepting cohort regression on JP is a public commitment to under-serving the strategically-important market for the sake of a small aggregate gain on EN. That trades short-term aggregate metrics for long-term market position.

The right framing: aggregate is a check; cohort guardrails are the veto. The aggregate isn't allowed to win at the expense of a cohort that was pre-declared as guarded. The veto fires.

What I'd refuse: shipping with cohort carve-out as a "follow-up." Carve-outs become permanent in 80% of cases (the engineering effort to fix them never gets prioritized once the launch is in production).

Round 3 — Squeeze

Follow-up: You retrain with JP-stratified loss reweighting. Re-run shows: EN +3.9%, JP +1.2%, mixed +2.4%, aggregate +3.6%. JP guardrail clears (-3% threshold not breached). Ship?

Almost. Two more checks before shipping:

Check 1: aggregate retention vs cohort retention. The primary metric (CTR / useful-answer-rate) is a leading indicator. The lagging indicator is retention. For the JP cohort that just barely cleared the per-cohort guardrail at +1.2%, I want to verify 28-day retention is also non-regressive. If JP cohort retention is +0pp (parity) at 28 days, ship; if it's -0.3pp despite primary +1.2%, the lift is hollow and not real.

Check 2: the original-original concern. The JP cohort previously had +0pp baseline (the system worked at parity). After this launch, JP has +1.2pp lift — worse than EN's +3.9pp. We've improved JP, but EN improved more. Over 4 quarters of similar launches, the EN-vs-JP gap could widen. I'd note this in the launch readiness review: "JP cohort cleared guardrail but with smaller absolute gain than EN; track EN-JP gap over the next 4 launches; if gap widens to >2pp differential, that's a signal that the underlying training-data imbalance still dominates and we need a more structural fix."

If both checks pass, ship. The discipline is: even when guardrails clear, the post-launch monitoring is the second-line defense.

Round 4 — Corner

Follow-up: Six months after shipping, retention by cohort: aggregate -0.1% (noise), EN +0.2%, JP -0.4%. JP retention has slowly eroded. What do you do?

This is post-launch slow erosion — the most insidious failure mode. Three responses:

1. Confirm the signal is real. -0.4% over 6 months on a per-cohort metric: is it statistically significant given the cohort's variance? Run a proper test, not just visual inspection. If it's noise, say so and continue monitoring. If it's signal, escalate.

2. If signal is real, treat it as a guardrail breach (post-hoc). The original launch was guardrail-cleared at MDE; the slow erosion was below MDE during launch. But cumulative customer harm at 6 months is real. Trigger an investigation: is this drift specific to the launched model? Is it a cumulative effect of multiple launches each within threshold? Is it independent (e.g., catalog evolution affecting JP differently)?

3. Prescribe action proportional to root cause. If the launched model is the cause: roll back, retrain, re-launch. If cumulative-launch-effect: tighten cohort thresholds for next quarter. If independent (e.g., catalog evolution): structural investment in JP-specific infrastructure.

What I'd refuse: dismissing slow cohort erosion as noise without a proper test. Slow erosion is how cohort harm accumulates; the team that misses it is the team that finds out 18 months later when JP retention is 5% lower than EN.

Architect-Level Escalation

A1: Design cohort-fairness infrastructure that scales across all 8 ML systems and 50 launches per year. What does it look like?

Five layers:

1. Cohort dimension catalog. Standard cohort dimensions: locale, tenure, device, age band. Versioned. Surface-specific extensions (e.g., manga preference cluster) added to specific systems.

2. Per-cohort sample-size requirements at launch readiness. YAML pre-registration includes per-cohort MDE and required sample size; platform refuses to randomize if cohort claims are under-powered.

3. Cohort-stratified telemetry. Every metric event tagged with cohort dimensions. Single schema means cohort analysis is a SQL query, not bespoke per-experiment code.

4. Cohort dashboards by default. Every experiment dashboard shows aggregate AND cohort-stratified primary, automatically. No team has to remember to add it.

5. Cohort calibration audit. Quarterly: of shipped launches, did any cohort regress in post-launch retention? If yes, was that cohort sample-size-adequate at launch? Calibrate thresholds based on observed cohort-harm patterns.

The systems insight: at scale, cohort analysis is automatic or nonexistent. Manual cohort analysis fails at 50 launches/year. Automation is the only path.

A2: How do you defend "we paid 4 weeks of velocity to fix JP cohort fairness" in OP1?

Six-pager:

Page 1: "We declined to ship a +2.8% CTR launch this quarter because the JP cohort regressed by -8.2%. We invested 4 weeks in JP-stratified retraining and re-launched at +3.6% aggregate, +1.2% on JP. Net portfolio cost: 4 weeks; net JP-cohort outcome: parity preserved."

Page 2 — Strategic context: "JP is 30% of MangaAssist traffic and the primary catalog locale. JP user trust is multi-quarter compounding; cohort regressions erode it. We do not ship cohort regressions to JP regardless of aggregate gains."

Page 3 — Tenets: - "Cohorts are customers, not weights." - "Aggregate gains do not justify cohort regressions." - "We pay short-term velocity for long-term market position." - "When in doubt, parity for the strategically-critical cohort."

Page 4 — Quantification: "If we'd shipped the original launch with JP -8% regression, expected JP-cohort retention impact at 28-day is -1.5pp absolute. Over 12 months, that's $X million in JP revenue impact and longer-tail trust erosion. The 4-week velocity cost is the price of avoiding that."

Page 5 — FAQ: "Why not carve-out JP?" Because carve-outs become permanent and signal second-class treatment. "Why not accept cohort regression?" See tenets. "Will this slow us down on every launch?" Only on launches with cohort asymmetry; ~25% of launches based on history.

LPs: Earn Trust (cohort-level), Insist on Highest Standards (no shipping cohort regressions), Success and Scale Bring Broad Responsibility (scale makes cohort fairness a duty), Long-Term Thinking (4-week cost vs 4-quarter erosion).

A3: A senior leader pushes back: "JP is 30% — sometimes we have to lose a cohort. What if it's a small cohort?"

I push back, structured:

"The threshold isn't about cohort size; it's about whether we declared the cohort as guarded. If we pre-declared a cohort guardrail at -3% relative, we enforce it regardless of cohort size. The pre-declaration is the discipline; without it, every cohort regression becomes a one-off political negotiation."

"That said, the threshold can be different for different cohorts based on strategic importance. JP at -3% is appropriate; a smaller, less-strategic cohort might have a looser threshold. The threshold negotiation happens at pre-registration, not post-launch. If we want to set 'small cohort threshold = -10%', that's a defensible policy choice we discuss at the framework level — not at the launch level."

"What I'd refuse: setting thresholds post-hoc based on whether we want to ship. That's exactly the political negotiation the framework prevents. Either we have pre-declared thresholds and enforce them, or we don't. The middle ground — 'we have thresholds but they're advisory' — is worse than either."

The trap: agreeing to a "judgment call" framework that lets thresholds slip case-by-case. The discipline says: pre-declare; enforce; revisit thresholds at framework-level if calibration is wrong; never negotiate at launch-level.

Intuition Gained — AML-06

The core insight: Aggregate metrics hide cohort regressions. Cohort-stratified eval is the antibody. The strategically-critical cohort gets parity; aggregate gains don't justify cohort losses.

Mental model to carry forward:

"Customers are not weights. A 30% cohort regressing by 8% is not 'made up' by a 55% cohort gaining 4%. They are different people."

The hidden failure mode: Slow post-launch erosion that's below MDE at launch but accumulates. Track per-cohort retention quarterly; treat sustained erosion as a post-hoc guardrail breach.

One-line rule: Pre-declare cohort thresholds. Enforce at launch. Audit post-launch. The cohort that didn't get parity at launch usually doesn't get it later.

Red-Flag Indicators

• Computes "net effect" by subtracting cohort regression from aggregate
• Proposes cohort carve-out as a "follow-up"
• Cohort sample size not pre-registered; cohort claims under-powered
• Treats cohort regression as acceptable if aggregate wins
• No post-launch cohort retention monitoring

Strong-Answer Markers

• Pre-registered cohort thresholds in YAML
• Per-cohort sample size in launch readiness
• "Customers are not weights" framing
• Strategic-context-aware (JP for MangaAssist)
• Post-launch cohort calibration audit

Scenario AML-07 — Production Integration & Latency Budgets

Opening Question

Q: Your chatbot turn budget is 800ms p95 from user-message-received to first-streaming-token-out. The reranker takes 180ms p95 currently. The new reranker takes 240ms. How do you decide whether to take the latency hit?

Round 1 Answer: I decompose the 800ms budget per-stage, simulate the pipeline with empirical per-stage latency distributions, and check that the new reranker fits with safety margin. Allocation: 30 input parse / 50 intent / 220 retrieval / 240 rerank / 200 FM first-token / 40 format = 780ms sum-of-stages. Empirical simulation typically shows 5-15% inflation over additive sum, so true p95 ≈ 815ms. To meet 800ms SLO: parallelize intent + retrieval (saves 50ms), reduce reranker top-K from 30→20 (saves 50ms, costs 0.4% NDCG), and implement BM25-only fallback for reranker timeouts >500ms. Final simulated p95: 770ms with 30ms safety margin. Load-test at 2× peak before rollout. Pre-build cohort-stratified latency dashboard before production cutover. Fallback engagement counter is a leading-indicator alarm for incidents.

Round 1 — Surface

Follow-up: Why does p95 not compose additively?

Two reasons. The first is mechanical: serial stages with bursty distributions amplify tails. If stage A's p95 is 100ms but its p99 is 500ms (heavy tail), and stage B's p95 is 100ms with p99 500ms, the joint distribution's p95 is more than 200ms because the moments where A is slow and the moments where B is slow combine in non-trivial ways. The variance compounds; the tails compound more.

The second is correlative: stages aren't always independent. When the FM endpoint is loaded, retrieval is often loaded too (shared infra). When network is slow, every stage feels it. Correlated tails mean joint-p95 inflates faster than independent p95-sum.

Empirically, for a 6-stage pipeline like ours, sum-of-stage-p95 underestimates true-pipeline-p95 by 5-15%. The right model is simulation: pull 14 days of production telemetry, sample stage-latencies with replacement to construct turn timelines, compute end-to-end p95 from the simulation. Compared to additive p95, simulation gives the realistic number.

For our case: sum 780ms; simulation 815ms; SLO 800ms. The 35ms gap is the safety margin we need to engineer back via parallelization and top-K reduction.

Round 2 — Push Harder

Follow-up: You implement parallelization — fire intent + retrieval simultaneously. But intent-routed retrieval needs the intent label first. How does that work?

It works because the dependency is loose, not strict. Three patterns:

Pattern 1: speculative parallel. Fire retrieval with a default intent (most-common intent, or ANY intent that includes all candidate items) in parallel with the intent classifier. When the intent classifier returns, if it agrees with the default, use the speculative results. If it disagrees, use a fast filter on the speculative results to keep only the items matching the actual intent. Cost: extra compute on retrieval; benefit: serial latency cut from 50+220 = 270ms to max(50, 220) = 220ms.

Pattern 2: pre-fetched candidate pool. Retrieval returns a broader candidate pool (e.g., top-100 items across all intents) without intent filtering. Intent label, when ready, post-filters the pool. The retrieval is intent-agnostic; only the post-filter depends on intent. Latency: max(intent, retrieval), no serial dependency.

Pattern 3: cached intent prediction. For repeat users on similar queries, cache the recent intent prediction. Retrieval uses cached intent; intent classifier runs in background to update cache. Latency: retrieval can start immediately at typical cache hit rate ~40%.

For MangaAssist, I'd default to Pattern 2 (pre-fetched pool) because it's the simplest and most robust — no speculative-execution complexity, no cache freshness concerns. Retrieval returns top-100 candidates; intent label post-filters to top-30 for reranking. The 50ms intent classifier latency overlaps with retrieval; total stage-latency is 220ms instead of 270ms.

Cost: retrieval returns more candidates than strictly needed (top-100 vs top-30). OpenSearch handles this fine — kNN+BM25 RRF on top-100 vs top-30 is barely measurably slower.

Round 3 — Squeeze

Follow-up: You reduce reranker top-K from 30 to 20. The data scientist says "this loses 0.4% NDCG@10 — that's significant." How do you balance latency vs quality?

The balance is not a vibe — it's a constrained optimization. Two factors:

Factor 1: latency is binary at SLO boundary. Below 800ms p95, the SLO is met and customer experience is fine. Above 800ms, the SLO is breached and customers perceive slowness. The latency value between 700ms and 800ms is approximately equal — the 100ms savings from top-K reduction has minimal customer-experience value as long as we're below SLO. The latency value of 50ms savings that takes us from above-SLO to below-SLO is enormous.

Factor 2: NDCG@10 loss at top-K=20 vs top-K=30 is concentrated at positions 5-10. Mobile users (70% of traffic) see positions 1-3. The "0.4% NDCG@10 loss" is mostly at positions 5-10 where users don't look. The customer-experience cost is much smaller than the offline metric suggests (this connects to AML-04 — NDCG@10 is the wrong metric for mobile-skewed traffic).

So the trade is: 50ms latency savings (high value, brings us under SLO) for 0.4% NDCG@10 loss (low value, mostly at unlooked positions, equivalent to maybe 0.1% NDCG@3 loss). The trade is asymmetric in favor of latency.

Empirically, I'd verify: re-run the offline eval with NDCG@3 (the metric matching mobile UX). If NDCG@3 loss is < 0.2%, the trade is clearly favorable. If NDCG@3 loss is >0.5%, reconsider. The right way to defend the decision is on the metric that matches user behavior, not the metric that's convenient.

Round 4 — Corner

Follow-up: You're in production. p95 is at 790ms — within SLO. But fallback engagement is 0.8%/min, climbing slowly week-over-week. What's happening?

Fallback engagement at 0.8%/min is concerning — it's above the 0.5% baseline but below the 1%/min alarm. Climbing trend is the leading indicator of an incident.

Diagnosis:

Possibility 1: reranker p95 has crept up. Even if turn p95 is at 790ms, individual reranker calls might be >500ms (the timeout) more often. Check reranker p95 stratified by hour and by SageMaker shard. If one shard is over-loaded, fallback engagement on that shard spikes.

Possibility 2: traffic mix shift. A new traffic source (marketing campaign, anime tie-in release) might be sending queries that take longer to rerank. Check query-distribution KL divergence vs baseline. If KL is high, it's a query-distribution shift that's stressing the reranker.

Possibility 3: SageMaker MME endpoint scaling lag. Auto-scaling reacts to load with delay; bursty traffic above provisioned capacity triggers brief over-loads. Check auto-scaling events; if scaling is lagging, increase min-capacity or add headroom.

Possibility 4: cohort-specific issue. If JP traffic is concentrated on one model variant in the MME, JP-cohort fallback might be high while EN is low. Check cohort-stratified fallback engagement.

The action: find which possibility is dominant before escalating. If 0.8%/min is from one shard, scale that shard. If it's from query-distribution shift, the reranker might need re-fine-tuning for the new distribution. If it's auto-scaling lag, increase headroom.

What I'd refuse: ignoring the trend because we're under SLO. Climbing fallback engagement is a leading indicator that the SLO will breach within 1-2 weeks. The discipline is to act on leading indicators, not wait for SLO breach.

Architect-Level Escalation

A1: Design a per-stage latency monitoring system that catches incidents before SLO breach. What does it look like?

Five components:

1. Per-stage p50/p95/p99 with alarms. Each stage has its own SLO derived from the turn-level budget. Stage SLO breach triggers a stage-level alarm (yellow); accumulated stage breaches trigger turn-level alarm (red).

2. Cohort-stratified per-stage latency. Same stages, broken by locale × device × tenure. Catches cohort-specific stress (e.g., JP traffic concentrated on one MME shard).

3. Fallback engagement as leading indicator. Per-stage fallback rate (timeout fallbacks / total requests). Alarm at 1%/min, page at 5%/min. Climbing trend (week-over-week growth) is a soft alarm.

4. Auto-scaling event log overlay. Scaling events visualized on the latency timeline. Helps diagnose "is this latency due to scaling lag or model issue?"

5. Query-distribution drift monitor. KL divergence between current query distribution and baseline. High drift correlates with latency increases; provides advance warning.

The systems insight: SLO breach is the final indicator, not the alarm. By the time you're SLO-breaching, customers have already had a bad experience. The leading indicators (per-stage SLO, fallback engagement, query drift) catch incidents at the 1-2-week-out warning stage.

A2: A team adjacent to yours wants to add a 100ms ML model into the same turn budget. The 800ms is fixed. What do you say?

I say "the budget is a contract; if you want 100ms, someone else loses 100ms or we negotiate the SLO."

The conversation:

"The 800ms p95 SLO is the customer-experience contract. Within that, every stage has an allocated budget. Adding 100ms means another stage gives up 100ms, OR we miss SLO. The question is which trade-off is acceptable."

"Three options: (1) Reduce reranker top-K further (saves ~50ms; costs ~0.5% NDCG) and reduce retrieval candidate pool (saves ~30ms; costs marginal recall). Total 80ms savings; closer but not 100ms. (2) Move some processing async — e.g., the new ML model's output feeds the next turn instead of the current turn. No latency hit on current turn; lag of one turn on usefulness. (3) Increase the SLO from 800 to 900ms p95. Customer-experience research says 900ms is still acceptable on mobile but starts feeling slow on desktop. Requires customer-research validation."

"My recommendation: (2) async if your model's output is useful one turn later; (1) if not. (3) only if the model's value-per-100ms is enormous and customer research validates the SLO change. The team that adds 100ms without negotiating these trade-offs is shipping a degraded customer experience."

The trap: agreeing to "we'll figure out the budget later." Latency budgets evaporate without explicit negotiation. The discipline is: every stage's budget is signed-off; new stages require either explicit budget reallocation or SLO renegotiation.

A3: When does the latency-budget framework break?

Three failure modes:

1. Budget creep. Each stage's budget gets nudged up by 5-10ms per quarter without anyone noticing. Over 8 quarters, the budget is 50ms over and the SLO breaches. Mitigation: per-stage budgets are versioned in git; changes require sign-off; quarterly audit of stage p95s vs allocated budget.

2. P99 invisibility. Teams optimize for p95 and ignore p99. p99 is where the ugly customer-experience-failure cases live; p95 looks fine while p99 customers churn. Mitigation: track p99 alongside p95; have explicit p99 SLO (e.g., 1500ms) with separate alarms.

3. Fallback-as-stage erosion. Fallback engagement creeps up over time; the system normalizes "5% of users get the BM25 fallback" as acceptable. Mitigation: track fallback engagement quarterly; if it exceeds 1%/min sustained, treat as a quality regression and investigate root cause.

The meta-failure: budgets, like guardrails, are an institution. If leadership doesn't enforce them at OP1 and quarterly review, they decay. Insist on the Highest Standards at the framework level is what keeps the system honest.

Intuition Gained — AML-07

The core insight: Latency is the customer's first-impression metric. The Applied ML Engineer who fights for the latency budget is defending the launch's actual outcome.

Mental model to carry forward:

"Per-stage budget allocation. Simulate the pipeline empirically. Pre-build the cohort-stratified telemetry before launch. Fallback engagement is the leading-indicator alarm."

The hidden failure mode: Tail-latency composition. Sum-of-p95 understates pipeline-p95 by 5-15%; the gap is the safety margin you need to engineer.

One-line rule: Pre-build the dashboard before launch. The dashboard built during an incident is too late.

Red-Flag Indicators

• Adds latency without explicit budget negotiation
• Sum-of-stages p95 used as turn-p95 (additive composition)
• No fallback chain or graceful degradation
• Treats SLO as advisory; ships above-SLO and "iterates"
• No cohort-stratified latency monitoring

Strong-Answer Markers

• Per-stage budget allocation explicit
• Empirical simulation for tail-latency composition
• Fallback chain with leading-indicator counter
• Pre-built cohort-stratified dashboard
• Defends budget as a contract; renegotiates SLO if needed

Scenario AML-08 — Incident Triage: 'The Model Got Worse'

Opening Question

Q: It's 3am. PagerDuty fires: reranker NDCG@10 dropped from 0.78 to 0.61 in the last hour, while traffic and latency are normal. You're on call. What's the first thing you check?

Round 1 Answer: I open the 3am dashboard and walk the named-decision-tree triage. First check: change-log overlay — did anything deploy in the last 24h? If yes (code, model version, config, prompt template), roll back that change and verify recovery; that's MTTR ~15-30 minutes for categories 1, 2, 3, 8. If nothing changed, move to upstream-data health: row counts, schema fingerprint, last-seen timestamps. If upstream healthy, check feature-distribution drift on top-10 features (KL > threshold). If drift, freeze model decisions while investigating. If no drift, check eval-set staleness vs catalog turnover. If eval is fresh and nothing else flags, check UI deploy timeline and external traffic (marketing campaign, anime tie-in release). If all those are clear, escalate to staff/principal — this is the rare-class incident requiring incident-commander mode. Target: localize to a named root cause in 15 minutes; mitigate in 60. Random root-causing is the anti-pattern.

Round 1 — Surface

Follow-up: Walk through the change-log overlay. What does it actually look like, and what makes it useful?

The change-log overlay is a single dashboard panel that visualizes:

1. The metric stream (NDCG@10) over the last 7d / 24h / 60min on the same chart.
2. Vertical lines marking every change event in the same time window: code deploys (from git/CI), model version updates (from model registry), config changes (from config-store audit log), prompt template versions (from prompt registry), feature-store schema changes (from data-platform audit log).
3. Hover tooltips showing the change content for each line: commit SHA, deployer, what changed, what service.

Useful when: a metric drop has a vertical change line immediately preceding it. The visual correlation is instant — "metric dropped 50 minutes after model_version_2026.04.27 deployed; revert that version."

Useful even when: nothing visually correlates. Negative diagnostic — "no changes deployed in the last 6 hours; categories 1, 2, 3, 8 are ruled out without further investigation; move to category 4 (upstream data) next."

The dashboard is pre-built. Building it during an incident is too late — pulling change events from 5 different audit logs ad-hoc takes 30 minutes I can't afford. The investment is one-time (1-2 weeks of engineering); the payoff is every-incident MTTR reduction.

Round 2 — Push Harder

Follow-up: No change events in the last 24h. Upstream row counts and schema look fine. You move to feature drift. Top-10 features show: query_length_avg KL = 0.04 (normal), reranker_input_score_distribution KL = 0.31 (high). What's happening?

KL=0.31 on the reranker input score distribution is the signal. Three hypotheses:

Hypothesis 1: upstream embedding model changed. Even if our reranker hasn't changed, the embeddings it consumes (from US-MLE-05 embedding adapter) might have. Check the embedding model version; if it rotated recently, that's the trigger. The reranker was trained against old-distribution scores; new-distribution scores look out-of-distribution.

Hypothesis 2: retrieval distribution shifted. The reranker scores items returned by retrieval. If retrieval started returning a different mix of items (different rank ordering, different score distribution), the reranker's input distribution shifts. Check OpenSearch query patterns and HNSW index health.

Hypothesis 3: data corruption upstream. A subtle corruption in the catalog or item-feature store changes item embeddings, which changes reranker input scores. Check item-feature-store fingerprints.

For each: what would you actually do in the 60-minute window?

• Hypothesis 1: check model-registry diff for embedding model. Time: 5 minutes. If found, rollback embedding model version (independent of reranker). Time: 20 minutes. Total mitigation: ~30 minutes.
• Hypothesis 2: check OpenSearch index health (per-shard recall is the tell). If shard issue: force re-index that shard. Time: 1-2 hours.
• Hypothesis 3: deeper investigation; data-platform team partner.

I'd start with Hypothesis 1 (cheapest to test), then Hypothesis 2, then Hypothesis 3. The triage discipline is: cheapest-to-confirm-or-rule-out first.

Round 3 — Squeeze

Follow-up: Hypothesis 2 is right — OpenSearch shard-3 has corrupted HNSW. You force re-index shard-3. After 45 minutes, NDCG recovers to 0.77. The team has questions: how do you write the post-incident retrospective?

The blameless retrospective covers seven sections:

1. Timeline. - T+0: PagerDuty alarm; NDCG dropped 0.78 → 0.61. - T+5m: On-call AML Eng acknowledges; opens 3am dashboard. - T+12m: Categories 1, 2, 3, 8 ruled out via change-log overlay (no changes in 24h). - T+16m: Category 4 ruled out (upstream data healthy). - T+22m: Category 5 confirmed — feature drift on reranker input score distribution KL=0.31. - T+25m: Hypothesis 1 (embedding rotation) ruled out in 5 min — no rotation. - T+32m: Hypothesis 2 confirmed — OpenSearch shard-3 HNSW corruption (per-shard recall 0.43 vs others at 0.91). - T+35m: Force re-index shard-3 initiated. - T+45m: Re-index complete; NDCG recovers to 0.77.

2. Root cause. OpenSearch upgrade applied silently overnight via auto-update triggered partial re-indexing on shard-3 that didn't complete cleanly. Auto-update was enabled when the cluster was provisioned 14 months ago; the configuration drift was never caught.

3. Customer impact. ~50,000 customer interactions during the 6-hour degradation window had degraded ranking quality. No customer-reported tickets (the degradation was real but subtle on individual queries). 28-day retention in the affected window TBD; post-incident monitoring planned.

4. What went well. Triage tree localized in 25 minutes; mitigation in 45 minutes total. The 3am dashboard's per-shard recall panel (added in Q1) caught what the previous dashboard would have missed.

5. What didn't go well. OpenSearch auto-update was enabled without team awareness. Per-shard recall alerting threshold (0.85) was set such that this dropped to 0.43 before page-level alarm.

6. Action items. - Disable OpenSearch auto-updates; manual update with team review (owner: SRE; due 1 week). - Tighten per-shard recall alarm threshold to 0.80 (owner: AML Eng; due 1 week). - Add OpenSearch upgrade events to change-log overlay (owner: AML Eng + SRE; due 2 weeks). - Shard-level recall as 5-min rolling alarm, not just dashboard (owner: AML Eng; due 2 weeks).

7. Calibration. Add this incident type ("infra silent change → feature drift → metric collapse") to next quarter's portfolio (AML-02). The pattern is general; affects more than reranker.

The retrospective is published in the team channel within 48 hours; reviewed in next 1:1 with EM; action items tracked in shared backlog with owners and dates. The discipline: the retrospective is the artefact that prevents recurrence.

Round 4 — Corner

Follow-up: A month later, a similar incident: another silent infra change (Bedrock endpoint update) caused FM latency p99 to spike. Triage took 90 minutes — 3× longer than the first. Why?

Three reasons, ranked by likelihood:

1. The named-decision-tree didn't include "Bedrock endpoint update" as a category. The first incident was OpenSearch shard corruption (category 4 + 5). The second incident is in a category we didn't name. The triage tree walked all 10 named categories, found nothing matching, and went into "rare-class escalation" mode — which is by design slower (incident-commander assembly, multi-team partner). The fix: add "managed-service silent-update" as a named category, with diagnostic (check service provider's audit log) and mitigation (call provider, force endpoint refresh).

2. The change-log overlay didn't include Bedrock events. The first incident's lessons added OpenSearch upgrades to the overlay. The Bedrock incident exposed the gap: managed-service updates from any provider aren't captured. The fix: integrate AWS Health Dashboard events; integrate Bedrock service-events API.

3. The team has different on-call experience. The first incident's on-call had walked the tree before; the second's hadn't. Documentation alone doesn't transfer experience. The fix: quarterly fire drills using past incidents — every team member walks the triage tree on a recorded incident, scored against MTTR target.

The systems insight: incident response is a continuous learning loop. Every incident reveals a gap in the triage tree, the change-log overlay, or team experience. The discipline is: every incident's retrospective produces a concrete update to the playbook AND a concrete addition to the portfolio (a new monitor, a new diagnostic, a new fire-drill scenario).

What I'd refuse: declaring the incident "resolved" without a structured retrospective. Without the retrospective, the next similar incident takes 90 minutes again.

Architect-Level Escalation

A1: Design a system that proactively prevents the next class of incident — not just responds faster. What does it look like?

Five layers of proactive defense:

1. Synthetic monitoring across managed services. Continuous low-volume probe traffic against every managed service (Bedrock, OpenSearch, SageMaker MME endpoints, S3, etc.) with quality assertions. If a managed service silently changes behavior, the probe catches it before customer-traffic does. Probes are tiny (<0.1% of cost); detection is fast.

2. Change-event ingestion from every dependency. AWS Health Dashboard events, Bedrock service events, OpenSearch upgrade events, SageMaker endpoint events — all ingested into the change-log overlay. The change-log isn't just our team's git history; it's every event that could change our system behavior.

3. Per-stage canary at every deploy. Whenever we deploy any change, a canary on 1% of traffic runs first. If the canary's metric is below threshold, deploy auto-pauses and pages on-call. Catches our-team-introduced regressions in 10 minutes instead of after full rollout.

4. Drift detection on every input distribution. Daily KL divergence on every feature-stream input vs 30-day baseline. Auto-page on KL > threshold. Catches both our-team and infrastructure-introduced drifts.

5. Quarterly fire drills. Use past incidents to drill the on-call rotation. Score MTTR. Identify gaps in the triage tree.

The systems insight: proactive prevention is asymmetric in value. The 5-layer defense is 8-12 engineer-weeks of investment. The avoided customer-impact from one prevented incident often justifies the whole investment. Compounding over 8 quarters, the ratio is 5-10× ROI.

A2: How do you defend "we invested 8 engineer-weeks in incident-prevention infrastructure this quarter" in OP1?

Six-pager:

Page 1 — Customer-impact context. "MangaAssist's last 4 quarters have had 11 production incidents on ML systems. Average MTTR 1.2h, average customer-impact window 4-6h, average affected customer-interactions ~50K per incident. Total customer-interactions affected last 4 quarters: ~550K."

Page 2 — The investment. "We invested 8 engineer-weeks in: synthetic monitoring (4 wks), change-event ingestion (2 wks), per-stage canary (1 wk), drift detection (1 wk). The investment was deferred from one Bias-for-Action portfolio slot."

Page 3 — The expected yield. "Based on incident-prevention literature (Google SRE book, 30%-50% MTTR reduction with proactive monitoring; 40%-60% incident-frequency reduction with synthetic monitoring), we expect: MTTR reduction 1.2h → 0.7h (-40%); incident-frequency reduction 11/yr → 6/yr (-45%). Combined customer-impact reduction: ~70%."

Page 4 — Risk mitigation. "Synthetic monitoring infrastructure adds <0.1% cost; minimal regression risk. Change-event ingestion adds dependencies on external APIs; mitigated by graceful-degradation patterns. Canary-deploy adds deployment time; offset by reduced rollback frequency."

Page 5 — Tenets. "Customer-impact reduction is multi-quarter compounding. Proactive prevention beats reactive response. Every incident retrospective produces a concrete prevention investment."

Page 6 — FAQ. "Couldn't we just hire another on-call?" — No, on-call response time is reactive; the goal is prevention. "Is 8 weeks worth one slot?" — Yes; the cumulative customer-impact reduction over 4 quarters exceeds any single Bias-for-Action launch.

LPs invoked: Customer Obsession (proactive prevention is for customers), Long-Term Thinking (multi-quarter compounding), Insist on Highest Standards (incident-prevention standards), Are Right A Lot (calibration based on past incidents), Ownership (we own the system, not just react to it).

A3: When does the triage discipline break? What are you most worried about?

Three failure modes:

1. Triage tree calcification. The 10 named categories work for the incident classes the team has seen. New incident classes (e.g., a novel attack vector, a new managed-service failure mode) don't fit. The on-call walks the tree, finds nothing, and escalates incorrectly — treating it as rare-class when actually a new category that should be added. Mitigation: every incident retrospective explicitly asks "is this a new category?" If yes, add it to the tree.

2. On-call rotation fatigue. If pages are frequent (e.g., > 2/week), the on-call is tired and sloppier. Triage discipline degrades; MTTR creeps up. Mitigation: page volume is itself a metric; if > 2/week sustained, the team has a quality problem requiring portfolio investment, not an on-call problem.

3. Mitigation-without-root-cause. The on-call mitigates (rolls back the offending change) without fully understanding root cause. The same class of incident recurs in a different form. Mitigation: retrospectives require explicit root-cause documentation; if root cause is "we don't know exactly," that's an open-ended action item until known.

The meta-failure: assuming the framework's done. The framework is a moving target — every incident's lessons update it. Teams that treat the framework as static become brittle to new incident classes. The discipline is continuous: every quarter, retrospect on the framework itself, not just the incidents.

Intuition Gained — AML-08

The core insight: Random root-causing is the anti-pattern. The named-decision-tree triage is the discipline. Every incident retrospective updates the tree.

Mental model to carry forward:

"Localize first, mitigate second, root-cause third. The triage tree is named categories; the dashboard is pre-built; the change-log overlay is integrated. Random checks waste minutes that customers can't afford."

The hidden failure mode: Triage tree calcification — new incident classes not in the tree. Mitigation: explicit retrospective question on every incident.

One-line rule: Pre-build the 3am dashboard. The dashboard built during an incident is too late.

Red-Flag Indicators

• Random root-causing instead of named-tree triage
• No 3am dashboard or change-log overlay
• Mitigates without root-causing
• Treats incident as resolved without retrospective
• No quarterly fire drills

Strong-Answer Markers

• Named-decision-tree triage with 10 categories
• Pre-built 3am dashboard with change-log overlay
• Cheapest-to-confirm-or-rule-out hypothesis order
• Blameless retrospective with concrete action items
• Proactive prevention investments based on incident learnings

Cross-Cutting Grill

The eight scenarios above each test one judgment call. Real launches require composing them. The cross-cutting grill tests system-level thinking — connecting multiple scenarios, defending portfolio-level decisions, and operating across multiple ML systems simultaneously.

CC-Q1: The Quarter-End Choice

Q: It's the end of Q3. You've shipped reranker (US-MLE-02), recsys cold-start (US-MLE-06), and embedding adapter (US-MLE-05). Three of them have positive offline metrics. Two have positive online metrics. Which one do you escalate to leadership for a Q4 follow-up investment, and why?

The answer requires comparing across all three on multiple dimensions:

The right escalation is the one where online evidence is positive AND the experiment yielded high EVOI for future quarters, not the highest-online-lift in isolation.

If the reranker has positive online but small lift (+0.5% useful-answer); the recsys cold-start has positive online and medium lift (+1.5% retention) but the EVOI was high (resolved cold-start direction question); the embedding adapter has positive offline but flat online (correlation collapse risk per AML-04) — the escalation is the recsys cold-start. Reasoning:

• Reranker: +0.5% useful-answer is real but doesn't define the platform; small follow-up investment, not escalation-worthy.
• Recsys cold-start: +1.5% retention + EVOI resolution; the next 4 quarters of recsys investment is now scoped against a known-direction. This is the platform-defining win.
• Embedding adapter: positive offline + flat online suggests AML-04 correlation issue; escalating this would be racing in the wrong direction. Don't escalate; rebuild offline harness first.

The Director conversation: "We shipped 3 launches. The platform-defining one is recsys cold-start — not because it had the largest lift, but because it resolved a strategic question. We'd like to invest the Q4 swing-bet slot in HRNN-Coldstart's natural follow-up: building a personalization framework that consumes the cold-start cohort embedding. The reranker and embedding-adapter follow-ups are smaller-scope investments that fit in the Bias-for-Action slots."

CC-Q2: Three Simultaneous Incidents

Q: It's 3am. Three pages fire within 10 minutes: reranker NDCG drops, recsys CTR drops, intent classifier accuracy drops. Same root cause? Different? How do you triage?

Three simultaneous pages on three systems is almost always a shared dependency failure — one upstream cause cascading. The on-call's first move is to look for the shared dependency, not to triage three independent incidents.

Candidates for shared dependency:

1. Embedding service (US-MLE-05). All three downstream systems (reranker, recsys, intent classifier) consume embeddings — if the embedding service degraded, all three feel it. Check embedding service health first.
2. Feature store. All three systems read features from the shared store. Schema corruption or service degradation hits all three.
3. Model registry. A version-pin issue could affect multiple systems if all three were deployed against a shared registry config.
4. OpenSearch. The reranker and recsys both query OpenSearch (intent classifier doesn't directly, but its training data might).
5. Network / region-level event. ap-northeast-1 service event affecting multiple managed services.

The on-call's triage: - T+0: Acknowledge all three pages. - T+2m: Check shared-dependency dashboard (pre-built panel showing all upstream services). If one is red, that's the cause. - T+5m: If no shared dependency is obviously red, escalate to incident-commander; assemble multi-team response (reranker owner, recsys owner, intent owner, platform owner). - T+15m: Treat as single incident with multi-symptom pattern; investigate shared dependency hypothesis even if shared dashboard looks healthy.

What I'd refuse: triaging three pages as three independent incidents. If they fire within 10 minutes of each other, they're correlated until proven otherwise.

CC-Q3: The Conflicting Tenet

Q: One tenet says "default to heuristic over ML." Another says "we run one EVOI swing bet per quarter." The cold-start improvement could be done with a heuristic (taste quiz) OR with an ML EVOI swing bet (HRNN-Coldstart). Which tenet wins?

Both tenets apply, in sequence. The Stage 1 / Stage 2 framing from AML-01 resolves the tension:

• Stage 1: ship the heuristic (taste quiz). It's the simplest intervention that moves customer behavior. The "default to heuristic" tenet wins for Stage 1.
• Stage 2: after the heuristic ships and the team has 6+ weeks of post-launch data, evaluate whether an ML upgrade earns the next quarter's portfolio slot. If yes, the ML upgrade is the EVOI swing bet for Q4. The "EVOI swing bet" tenet wins for Stage 2.

The two tenets are sequenced, not in conflict. The mistake is collapsing them: either skipping Stage 1 (default-to-ML) or skipping Stage 2 (heuristic-forever, no learning). The discipline is to stage the decision and respect both tenets.

In a six-pager: "Stage 1 (this quarter): heuristic taste quiz, addresses cold-start pain at minimal cost. Stage 2 (next quarter, contingent): if heuristic plateaus, ML cold-start is the EVOI swing bet for Q4 portfolio. The two tenets compose; they do not conflict."

CC-Q4: The Six Months Out Question

Q: Six months from now, leadership asks: "How do we know the team's discipline is working?" What's your answer?

The answer is a calibration audit:

Audit components:

1. Hit rate: of shipped experiments, what fraction maintained their lift at 28-day post-launch? Target ≥ 80%.
2. Veto effectiveness: of vetoed experiments, retrospectively, what fraction would have caused harm if shipped? Target ≥ 70%.
3. Calibration: was predicted Confidence in line with realized success rate? Was predicted Effort in line with realized Effort?
4. MTTR: average MTTR on incidents; target trend = decreasing quarter-over-quarter.
5. Cohort-fairness audit: any post-launch cohort regressions that pre-launch guardrails missed? Target ≤ 1 per quarter.
6. Pre-registration compliance: 100% of experiments pre-registered before randomization. Audit-able from YAML hash signoff timestamps.
7. Override rate: fraction of vetoes overridden by Director sign-off. Target ≤ 10%; > 20% indicates framework breakdown.

The six-pager defending the discipline:

"In 4 quarters: 38 shipped, 11 vetoed. Of 38 shipped, 34 maintained lift at 28-day (89%). Of 11 vetoed, retrospective shows 9 would have regressed (82% of vetoes were correctly conservative). MTTR on 8 incidents: 47 minutes average (down from 95 minutes baseline). Zero post-launch cohort regressions that pre-launch missed. 100% pre-registration compliance. 1 override (8%). Calibration delta: predicted vs realized hit rate 0.78 vs 0.82 (well-calibrated); predicted vs realized Effort 1.0× vs 1.15× (slight under-estimate, accepted).

"The discipline is working. Customer-impact is real and measurable. The cost (slower velocity on borderline cases) is bounded. We continue."

LPs: this is Insist on the Highest Standards as practice, not just policy. The audit is what makes the policy credible.

CC-Q5: When Do You Hire?

Q: Your team is shipping 3 experiments per quarter. The portfolio review consistently says "we have 8-10 candidates we'd ship if we had capacity." Do you hire? When? What for?

Capacity-constrained portfolios are the right time to hire — but the hiring is for the constraining role, not generic ML engineers.

The constraint analysis:

• If the constraint is experimental rigor (pre-registration, statistical analysis, guardrail design), hire a senior data scientist or staff Applied ML Engineer. The bottleneck is judgment, not throughput.
• If the constraint is engineering throughput (training pipelines, integration, telemetry), hire ML platform engineers. The bottleneck is infra.
• If the constraint is domain knowledge (ranking, recsys, NLP-specific), hire specialists in the under-served sub-domain.

For our team: if portfolio review consistently bottlenecks on "we don't have a senior person to design the experiment for X," hire an Applied ML Engineer. If bottlenecks on "we have the design but not the engineering," hire a platform engineer.

The interview process for a senior Applied ML Engineer should drill the 8 grill chains in this document. The hiring bar is: candidate clears the strong-answer markers on at least 6 of 8 scenarios.

CC-Q6: The Existential Question

Q: Five years from now, will Applied ML Engineers still exist as a distinct role, or will the role merge into something else?

My honest answer: the role's substance — translating customer pain into ML hypotheses, designing experiments, enforcing guardrails, triaging incidents — is multi-decade durable. The specific tools will change every 18 months (LLMs, agents, RAG, multimodal — and whatever comes next). The judgment under pressure is the lasting skill.

The trap: confusing the tools with the role. An Applied ML Engineer who knows only "today's stack" (e.g., specific model architectures, specific frameworks) ages out fast. An Applied ML Engineer who has internalized the seven primitives — Customer Obsession framing, portfolio thinking, hypothesis design, online/offline correlation, guardrails, cohort fairness, incident triage — applies them to any tool generation.

The role might be renamed. It might absorb adjacent responsibilities (prompt engineering, agent design, multimodal evaluation). But the judgment-under-pressure responsibilities don't go away. They get more important as systems get more complex and the cost of bad ML decisions grows.

What I'd bet on: the role's center of gravity moves toward system-level decisions (multi-model orchestration, cross-product impact, agent-system evaluation) and away from single-model decisions (which architecture, which training procedure). The grill chains in this document survive the shift.

Closing Rubric

Self-Drilling Score Sheet

For each of the 8 scenarios + 6 cross-cutting questions:

Total scoring (out of 14 scenarios + cross-cutting):

• 0-7 Pass: solid foundation, ready for mid-level Applied ML Engineer interviews; weakness in 1-2 scenarios needs targeted prep.
• 8-11 Strong-Pass: ready for senior Applied ML Engineer / Applied Scientist interviews; demonstrate cross-cutting reasoning in interview answers.
• 12-14 Exemplary: staff/principal-level signal; the artefacts you produce in interview will be used as benchmarks by the panel.

Final note on drilling

The grill chains in this document are not a checklist of right answers. They are patterns of thinking. The strongest candidates don't memorize answers; they internalize the seven primitives from 00-foundations-and-primitives-for-applied-ml-engineering.md and apply them under pressure. Drill until the primitives are second-nature; in interviews, the patterns will surface naturally.

The ultimate test: can you produce, on the spot, the OP1 narrative defending a portfolio decision? The six-pager structure (Tenets, Risks, Customer letter, Decision framework, Implementation plan, FAQ) is the senior-Applied-ML-Engineer's verbal artefact. Practice it; refine it; let it become how you naturally talk about your work.

That's what the role looks like. That's what the loop is testing for.