03. Tradeoffs & Decision Frameworks — How Metrics Drove Every Choice

"In production ML, there are no 'right' answers — only tradeoffs with different cost functions. My job was to make those tradeoffs explicit, measurable, and reversible. Every decision I made was backed by a metric, not an opinion."

Decision Framework: How I Structured Tradeoff Analysis

For every significant model inference decision, I used a consistent framework:

graph TD
    A[Identify the Tradeoff] --> B[Define Options<br>2-4 concrete alternatives]
    B --> C[Select Decision Metric<br>What number drives the call?]
    C --> D[Run Experiment<br>A/B test, shadow mode, or offline benchmark]
    D --> E[Measure Outcome<br>Quantify each option]
    E --> F[Make Decision<br>Document rationale]
    F --> G[Set Reversal Trigger<br>Under what conditions do we revisit?]

The reversal trigger was key — every decision had a documented condition under which we'd revisit it. This prevented both premature optimization and "set and forget" technical debt.

All Reversal Triggers Dashboard

One-stop reference: when do we revisit each decision?

Reusable Decision Template

Use this framework for any new inference tradeoff:

1. IDENTIFY THE TRADEOFF
   What are we trading? (latency vs quality, cost vs accuracy, etc.)

2. DEFINE 2-4 CONCRETE OPTIONS
   Each with measurable pros/cons.

3. SELECT A DECISION METRIC
   A single compound number that captures the tradeoff.
   E.g., Quality-adjusted latency, Cost per marginal accuracy point.

4. RUN AN EXPERIMENT
   A/B test, shadow mode, or offline benchmark. Never decide on intuition.

5. MEASURE AND DECIDE
   Show the math. Document the rationale.

6. SET A REVERSAL TRIGGER
   Under what specific, measurable condition do we revisit?

Tradeoff 1: Latency vs. Quality (Model Size)

The Options

Decision Metric

Quality-adjusted latency: (Response Quality Score × 20) - (P99 Latency in seconds × 10)

This penalized both low quality AND high latency, forcing a balance.

Decision

Option B — Model tiering. Route 40% of messages to templates (no LLM), 15% to Haiku, 45% to Sonnet based on intent complexity.

Reversal Trigger

Revisit if: (a) Haiku quality drops below 3.5/5.0 on routed queries, or (b) a new model class offers Sonnet quality at Haiku latency.

Tradeoff 2: Cost vs. Accuracy (Classifier Model Selection)

The Options

Decision Metric

Cost per marginal accuracy point (CMA): For each model upgrade, how much does one additional percentage point of accuracy cost?

Decision

DistilBERT (Option B) paired with rule-based fast path. The jump from rules to DistilBERT was highly cost-effective ($1,277/point). The marginal gain from DistilBERT to RoBERTa ($3,704/point) wasn't justified given our budget.

What the Accuracy Gap Actually Meant in Production

The 2.7% accuracy gap between DistilBERT (92.1%) and RoBERTa (94.8%) translated to:

500K messages/day × (100% - 40% rule-handled) = 300K BERT-classified messages
300K × 2.7% accuracy gap = ~8,100 additional misclassifications/day

Of those 8,100 misclassifications:
- ~60% still produced acceptable responses (wrong intent but LLM compensated)
- ~25% produced degraded but harmless responses
- ~15% produced noticeably wrong responses = ~1,215 bad experiences/day

Cost to fix: $10K/month additional infra
Cost of bad experiences: ~1,215 × $0.50 estimated support cost = ~$607/day = ~$18K/month

The math showed that the bad experience cost (~$18K/month) was close to the infra cost (~$10K/month). But the DistilBERT misclassifications were largely caught by the LLM layer (which could reason about intent independently), making the effective impact smaller. We decided the gap was acceptable.

Reversal Trigger

Revisit if: (a) Misclassification-driven escalation rate exceeds 3%, or (b) Inferentia support for RoBERTa reduces cost gap.

Tradeoff 3: Precision vs. Recall for Guardrails

The Problem

The post-generation guardrails pipeline validated LLM responses before sending them to users. But every guardrail check had a precision-recall tradeoff:

• High recall (catch all bad responses) → many false positives → good responses blocked → user sees "I can't help with that" more often → poor UX.
• High precision (only block truly bad responses) → some bad responses slip through → user sees hallucinated prices, fabricated products → trust damage.

The Options by Guardrail Type

ASIN Validation Guardrail:

Hallucination Score Guardrail:

Decision Metric

Net quality impact: (Recall × Severity of missed bad responses) - (False Positive Rate × Cost of blocking good responses)

For the hallucination guardrail: - Severity of a hallucination reaching users: High (trust damage, potential financial liability for wrong prices) - Cost of blocking a good response: Medium (user sees fallback, slightly worse UX but no harm)

This asymmetry favored higher recall: it's worse to let a hallucinated price through than to occasionally block a correct response.

Decision

• ASIN Validation: Strict (block if any ASIN invalid). Then surgically remove only the invalid ASIN from the response and regenerate that portion — rather than blocking the entire response.
• Hallucination Score: 0.5 threshold for alerting, 0.7 threshold for automated blocking. The 0.5 threshold feeds into a weekly review queue where a human validates and either confirms or dismisses.

Reversal Trigger

Revisit if: (a) guardrail block rate exceeds 5% of all responses, or (b) user complaints about "I can't help with that" responses increase.

Tradeoff 4: Fine-Tuned Model vs. Prompt Engineering

The Problem

For improving the LLM's manga-specific knowledge, we had two approaches:

Decision Metric

Time-to-value adjusted quality: Quality improvement per week of investment.

Decision

Prompt engineering + RAG for the MVP and V2. Reasons: 1. 2x better quality-per-week of investment. 2. Fully reversible — a bad prompt can be rolled back in seconds; a bad fine-tune requires retraining. 3. Bedrock limitations — fine-tuning Claude on Bedrock was not yet available at our scale. 4. RAG improvements compound — better retrieval improves all intents, not just the fine-tuned ones.

We reserved fine-tuning as a V3 option if prompt engineering plateaued.

Reversal Trigger

Revisit if: (a) prompt engineering quality plateaus below 90% user satisfaction, or (b) Bedrock supports easy fine-tuning with fast iteration cycles.

Tradeoff 5: RAG Chunk Size vs. Retrieval Quality

The Options

Decision Metric

Effective retrieval quality: Recall@3 × Precision@3 — a single number that penalizes both missed documents and noisy documents.

Decision

Variable chunking by source type. Product descriptions: 256 tokens, policies: 512 tokens, review summaries: 128 tokens. Different content types have different information density — one size doesn't fit all.

Reversal Trigger

Revisit if: (a) a new embedding model handles long chunks better (reduces precision penalty for large chunks), or (b) context window sizes grow enough that chunk count can increase from 3 to 5+.

Tradeoff 6: Real-Time vs. Batch Inference

The Problem

Not every model needed real-time inference. Some components could run asynchronously without impacting user experience.

Decision

Hallucination scoring moved to an async pipeline (SQS → Lambda), saving ~50-100ms from the critical path. This was the only component where real-time vs. batch made a meaningful latency difference without sacrificing quality.

The risk: if a hallucinated response was sent and scored after the fact, the user already saw it. Mitigation: for high-severity hallucinations (wrong prices), the validation was still synchronous (price cross-check was fast, ~10ms). Only the sophisticated entailment-based scoring was async.

Tradeoff 7: Temperature Tuning — Creativity vs. Accuracy

The Options

Decision

Intent-specific temperature — because different intents had different accuracy requirements:

Metrics That Validated This Decision

Summary: Decision Log

Key Takeaways for Interviews

1. "Every decision was backed by a metric" — I didn't choose DistilBERT because it "felt right." I chose it because the cost per marginal accuracy point showed diminishing returns for more expensive models.

2. "I defined reversal triggers" — Every decision had a documented condition under which we'd revisit. This shows intellectual honesty: I acknowledge decisions might be wrong and plan for that.

3. "Asymmetric costs drive asymmetric thresholds" — Guardrail precision vs. recall isn't 50/50. The cost of a hallucinated price reaching a user is 10x the cost of blocking a correct response. This asymmetry should be reflected in the threshold.

4. "Compound metrics over single metrics" — Quality-adjusted latency, effective retrieval quality (R@3 × P@3), cost per quality point — these compound metrics capture tradeoffs that single metrics miss.

5. "Not every component needs real-time inference" — Moving hallucination scoring to async saved 50-100ms. Knowing which components can tolerate latency is an architectural insight.

Related Documents

• 02-data-scientist-collaboration.md — Area 9 (Cost-Quality Tradeoff Analysis) — the CPQ framework
• Challenges/real-world-challenges.md §9 — Cost Management — Detailed cost breakdown and optimization strategies
• 15-tradeoffs-challenges.md — High-level tradeoff summary