US-02: LLM Model Tiering — Quality vs Cost vs Latency

graph TD
    subgraph "Cost Team Position"
        C["Use Haiku for everything.<br/>Sonnet is 15x more expensive.<br/>At 1M messages/day, that's<br/>$250K/month difference."]
    end

    subgraph "Performance Team Position"
        P["Haiku is 3x faster TTFT.<br/>Sonnet prefill kills our<br/>sub-second target.<br/>Template path is instant."]
    end

    subgraph "Inference Team Position"
        I["Sonnet's quality is measurably<br/>superior for recommendations<br/>and multi-turn reasoning.<br/>Haiku hallucinates 3x more<br/>on complex queries."]
    end

    C ---|"Agree on<br/>light queries"| P
    P ---|"Disagree on<br/>complex queries"| I
    I ---|"Disagree on<br/>budget"| C

    style C fill:#f9d71c,stroke:#333,color:#000
    style P fill:#4ecdc4,stroke:#333,color:#000
    style I fill:#ff6b6b,stroke:#333,color:#000

graph TD
    A["User Message"] --> B["Intent Classifier"]
    B --> C{"Complexity<br/>Classifier"}

    C -->|"Tier 1: Trivial<br/>Greeting, thanks, order status,<br/>simple promo lookup"| T1["Template Response<br/>⚡ 10-50ms | 💰 $0 | 🧠 N/A"]

    C -->|"Tier 2: Simple<br/>FAQ, product lookup,<br/>single-turn factual"| T2["Claude 3.5 Haiku<br/>⚡ 200-400ms | 💰 $0.001 | 🧠 0.78"]

    C -->|"Tier 3: Complex<br/>Recommendation, multi-turn,<br/>reasoning, comparison"| T3["Claude 3.5 Sonnet<br/>⚡ 800-1500ms | 💰 $0.015 | 🧠 0.92"]

    T1 --> D["Response"]
    T2 --> D
    T3 --> D

    style T1 fill:#2d8659,stroke:#333,color:#fff
    style T2 fill:#fd9644,stroke:#333,color:#000
    style T3 fill:#eb3b5a,stroke:#333,color:#fff

graph LR
    A["User Message<br/>+ Intent<br/>+ Entities"] --> B{"Rule-Based<br/>Pre-filter"}
    B -->|"High confidence<br/>trivial"| T1["→ Template Tier"]
    B -->|"Uncertain"| C{"Lightweight<br/>Complexity Model"}
    C -->|"score < 0.4"| T2["→ Haiku Tier"]
    C -->|"0.4 ≤ score < 0.7"| T2B["→ Haiku Tier<br/>(with quality monitoring)"]
    C -->|"score ≥ 0.7"| T3["→ Sonnet Tier"]

    style T1 fill:#2d8659,stroke:#333,color:#fff
    style T2 fill:#fd9644,stroke:#333,color:#000
    style T2B fill:#fd9644,stroke:#333,color:#000
    style T3 fill:#eb3b5a,stroke:#333,color:#fff

graph TD
    subgraph "Clear Template (30%)"
        CT["Greetings, order lookups,<br/>simple promos<br/>✅ No debate"]
    end

    subgraph "Clear Sonnet (20%)"
        CS["Multi-turn recommendations,<br/>complex comparisons,<br/>reasoning chains<br/>✅ No debate"]
    end

    subgraph "Gray Zone (50%)"
        GZ["Single-turn FAQ<br/>Product questions<br/>Simple recommendations<br/>⚠️ THIS IS WHERE THE<br/>TRADEOFF LIVES"]
    end

    GZ --> D{"Decision<br/>Framework"}
    D -->|"Default: Haiku"| H["Save money, accept<br/>slight quality drop"]
    D -->|"Upgrade: Sonnet"| S["Spend more, guarantee<br/>quality"]

    style CT fill:#2d8659,stroke:#333,color:#fff
    style CS fill:#eb3b5a,stroke:#333,color:#fff
    style GZ fill:#f9d71c,stroke:#333,color:#000

sequenceDiagram
    participant User
    participant Router as Complexity Router
    participant Haiku as Claude Haiku
    participant QM as Quality Monitor
    participant Sonnet as Claude Sonnet

    User->>Router: "What's a good manga for beginners?"
    Router->>Haiku: Tier 2 (simple recommendation)
    Haiku-->>Router: Response (quality_estimate=0.72)
    Router->>QM: Log response + quality estimate

    Note over QM: quality_estimate < 0.75 threshold

    QM->>Router: Trigger re-generation with Sonnet
    Router->>Sonnet: Same query + context
    Sonnet-->>Router: Higher quality response
    Router-->>User: Sonnet response delivered

    Note over QM: Latency penalty: +800ms<br/>Cost penalty: +$0.014<br/>Quality gain: +0.18

gantt
    title Model Tiering Rollout
    dateFormat  YYYY-MM-DD
    section Phase 1
    Implement template tier for chitchat + order     :a1, 2026-01-01, 14d
    Measure baseline quality per intent              :a2, 2026-01-01, 14d
    section Phase 2
    Deploy complexity classifier                     :b1, after a1, 14d
    Route clear-simple queries to Haiku              :b2, after b1, 7d
    section Phase 3
    Expand Haiku to gray zone (with monitoring)      :c1, after b2, 14d
    Implement auto-escalation pipeline               :c2, after b2, 14d
    section Phase 4
    Tune thresholds based on QACPI                   :d1, after c1, 21d
    Ongoing monthly review                           :d2, after d1, 30d