Applied ML Engineer User Stories — MangaAssist Chatbot (Amazon-Scale)

Overview

This directory is the Applied ML Engineer / Product Engineer for ML lens on the MangaAssist chatbot. Where the sibling ML-Engineer-User-Stories/ folder owns the platform lens (training pipelines, drift hub, model registry, retrain cadence, embedding re-index), this folder owns the product lens — translating a customer pain into an ML/AI hypothesis, designing the experiment with statistical discipline, integrating the model into the chatbot turn pipeline, and defending it under business-metric guardrails.

The role distinction matters because the two roles fail in different ways:

• ML Engineer failure mode: drift detection lags, retraining is brittle, the registry is wrong, online serving p99 spikes. Symptoms are infrastructural.
• Applied ML Engineer failure mode: shipped a model that won offline by 5% NDCG and lost 1.2% retention online; ran a 7-day A/B on an effect that needed 21 days to detect; promoted on aggregate CTR while JP cohort regressed by 8%; declared victory before novelty effect washed out. Symptoms are judgment failures, not infra failures.

This folder is the playbook for not making those judgment failures.

The user's framing for this artifact: "Applied ML is product engineer for ML projects where I am taking a business use case and then building a solution with ML and AI for that" — combined with experiment-selection rigor (data-science master's discipline) and Amazon product-engineering DNA (Working Backwards, customer obsession, six-pager culture).

What's In Here

Stories live inside 01-deep-dive-per-applied-ml-story.md (not as separate BDD files). The pattern mirrors Cost-Optimization-Offline-Testing/04-scenario-deep-dives-per-cost-story.md: one section per story, full BDD framing inside the section.

Story Roster

The eight stories cover the full lifecycle of a product-applied ML decision: should we build it (AML-01, 02) → how do we test it (AML-03, 04) → when do we ship it (AML-05, 06) → how does it run (AML-07) → what do we do when it breaks (AML-08).

Dependency & Sequencing Graph

The eight Applied-ML stories sit on top of the platform-ML stories. AML stories do not own training, registries, or drift hubs — they consume them. The dependency graph below shows AML stories (top) anchored to US-MLE platform stories (bottom).

graph TB
    subgraph Applied[Applied ML Engineer Lens — Product Decisions]
        AML01[AML-01<br/>Customer pain<br/>→ ML problem]
        AML02[AML-02<br/>Experiment<br/>portfolio]
        AML03[AML-03<br/>Hypothesis<br/>& sample size]
        AML04[AML-04<br/>Online/offline<br/>decoupling]
        AML05[AML-05<br/>Business-KPI<br/>guardrails]
        AML06[AML-06<br/>Cohort fairness]
        AML07[AML-07<br/>Production<br/>integration]
        AML08[AML-08<br/>Incident triage]
    end

    subgraph Platform[ML Engineer Lens — Platform Foundations]
        MLE01[US-MLE-01 Intent]
        MLE02[US-MLE-02 Reranker]
        MLE05[US-MLE-05 Embedding]
        MLE06[US-MLE-06 Recsys]
        MLE08[US-MLE-08 Cover-art]
        DRIFTHUB[Drift Hub<br/>+ Model Registry]
    end

    AML01 -->|frames the problem for| AML02
    AML02 -->|picks experiments for| AML03
    AML03 -->|sample-size feeds into| AML07
    AML04 -->|correlation gate before| AML05
    AML05 -->|veto signal for| AML07
    AML06 -->|cohort holdout for| AML03
    AML07 -->|telemetry feeds| AML08
    AML08 -->|root-cause feeds back into| AML01

    AML03 -.consumes.-> MLE02
    AML03 -.consumes.-> MLE06
    AML04 -.consumes.-> MLE05
    AML06 -.consumes.-> MLE08
    AML07 -.consumes.-> MLE01
    AML08 -.consumes.-> DRIFTHUB

    classDef appl fill:#9cf,stroke:#333,stroke-width:2px
    classDef plat fill:#fd2,stroke:#333
    class AML01,AML02,AML03,AML04,AML05,AML06,AML07,AML08 appl
    class MLE01,MLE02,MLE05,MLE06,MLE08,DRIFTHUB plat

Reading paths:

1. First time through — read 00-foundations-and-primitives-for-applied-ml-engineering.md end-to-end. Then pick a scenario in 01-deep-dive-per-applied-ml-story.md that matches a real product decision you face. Then drill yourself with the matching grill chain in 02-applied-ml-engineer-grill-chains.md.
2. Interview prep (Amazon Applied Scientist / Applied ML Engineer loop) — go through grill chains AML-01 → AML-08 in order, scoring yourself against the red-flag and strong-answer markers. Most candidates fail on AML-04 (online/offline decoupling) and AML-05 (guardrails) — those are the two stories with the highest signal-to-noise for senior-level evaluation.
3. Working a real launch — start with AML-02 (portfolio prioritization) to defend that this experiment is the right one to run; then AML-03 (hypothesis & sample size); then AML-05 (guardrails); then AML-07 (integration). The other stories are diagnostic, not prescriptive.

Owner Mapping

The Applied ML Engineer holds the cross-cutting product-decision contract: every model promotion ultimately requires their sign-off, and the artefacts in this folder are the evidence base for that sign-off.

Unified KPI Rollup

A senior leader (Eng Manager, Director, GM) should be able to scan this table and know what the role is accountable for on each story.

These are the metrics the Applied ML Engineer is on the hook for. They are deliberately not the same metrics the ML Engineer (US-MLE-XX) is on the hook for; the two roles share systems but own different metric surfaces.

Cross-Cutting Concerns Inherited by Every AML Story

These are non-negotiable defaults for every Applied ML Engineer artefact in this folder. Deviations require explicit Engineering Manager + PM sign-off documented in the experiment record.

Amazon Leadership Principle Map

The Applied ML Engineer role lives at the intersection of Customer Obsession, Dive Deep, and Insist on the Highest Standards. Each story illustrates a specific LP application:

This mapping is the lens the Applied ML Engineer uses when defending decisions in a six-pager review or operational meeting (OP1/OP2).

Relationship to Other Folders

This folder is part of an interconnected set:

• ML-Engineer-User-Stories/ — platform lens (owns the systems this folder makes product decisions about)
• Cost-Optimization-User-Stories/ — FinOps lens (cost-side decisions on the same systems)
• Cost-Optimization-Offline-Testing/ — offline-eval lens (the format this folder borrows for deep-dives + grill chains)
• Ground-Truth-Evolution/ — drift lens (why ground truth moves; AML-04 and AML-08 anchor here)
• POC-to-Production-War-Story/ — failure-narrative lens (AML-08 incident vignettes anchor here)
• RAG-MCP-Integration/ — architecture lens (where the models live in the chatbot turn pipeline)
• Domain1-FM-Integration-Data-Compliance/ — architectural-design lens (foundational design-space discussions)

When working on a real product decision: pull the platform owner from ML-Engineer-User-Stories/, the FinOps owner from Cost-Optimization-User-Stories/, and the Applied ML Engineer (you) from this folder. The three artefacts together are the launch readiness package.

How These Stories Were Built

Each story is grounded in a real architectural surface from the project memory: the manga catalog, the JP/EN bilingual traffic, the OpenSearch + Titan + reranker retrieval stack, the WebSocket streaming chatbot turn, the seven production catastrophes from the POC-to-Production War Story. None of the scenarios are abstract textbook examples; every one names the specific component, the specific metric, and the specific failure mode that an Applied ML Engineer at Amazon Japan running MangaAssist would actually face.

The framing — master's-degree data-science depth + Amazon product-engineering DNA — surfaces in two callouts inside every deep-dive section:

• Master's-DS Depth Callout — the statistical or methodological subtlety a non-DS engineer would miss (e.g., why a t-test is wrong here, why CUPED reduces variance, why pre-experiment covariates matter)
• Amazon Product-Lens Callout — the LP framing or six-pager-style reasoning a non-Amazon practitioner would miss (e.g., the customer letter, the input/output metric distinction, the OP1 narrative)

Together they encode the role: rigour without ivory tower, product instinct without hand-waving.