US-06: Cache Aggressiveness — Freshness vs Speed vs Cost

graph TD
    subgraph "Cost Team"
        C["Cache everything with long TTLs.<br/>Every cache hit saves an API call<br/>and an LLM invocation.<br/>At 60% hit rate, we save<br/>$120K/month."]
    end

    subgraph "Performance Team"
        P["We agree on caching for speed.<br/>But invalidation latency matters.<br/>If cache invalidation is slow,<br/>we serve stale data AND<br/>the user blames us."]
    end

    subgraph "Inference Team"
        I["Stale cached recommendations<br/>are WORSE than no cache.<br/>If someone bought Volume 1<br/>and we keep recommending it,<br/>we look stupid. Freshness<br/>is non-negotiable for<br/>personalized responses."]
    end

    C ---|"Freshness<br/>concern"| I
    I ---|"Invalidation<br/>complexity"| P
    P ---|"Cache cost vs<br/>savings"| C

    style C fill:#f9d71c,stroke:#333,color:#000
    style P fill:#4ecdc4,stroke:#333,color:#000
    style I fill:#ff6b6b,stroke:#333,color:#000

graph LR
    subgraph "Never Cache"
        NC["Prices<br/>Inventory counts<br/>Cart state<br/>Payment info"]
    end

    subgraph "Short TTL (1-5 min)"
        ST["Product details<br/>Availability status<br/>Search results"]
    end

    subgraph "Medium TTL (15-60 min)"
        MT["Recommendations<br/>Promotions<br/>FAQ answers"]
    end

    subgraph "Long TTL (1-24 hours)"
        LT["Review summaries<br/>Trending lists<br/>Editorial content<br/>User preferences"]
    end

    NC -->|"Freshness<br/>critical"| ST
    ST -->|"Moderate<br/>freshness"| MT
    MT -->|"Low<br/>freshness need"| LT

    style NC fill:#eb3b5a,stroke:#333,color:#fff
    style ST fill:#fd9644,stroke:#333,color:#000
    style MT fill:#f9d71c,stroke:#333,color:#000
    style LT fill:#2d8659,stroke:#333,color:#fff

graph TD
    A["User asks: 'What should I read next?'"] --> B{"Cache Hit?"}
    B -->|"Hit"| C["Return cached reco<br/>⏱️ 5ms | 💰 $0"]
    B -->|"Miss"| D["Full pipeline<br/>⏱️ 1,500ms | 💰 $0.015"]

    C --> E{"Is it stale?"}
    E -->|"No — user hasn't<br/>bought anything new"| F["✅ Good response"]
    E -->|"Yes — user just bought<br/>Volume 1 of recommended series"| G["❌ Bad response<br/>Recommends what they just bought"]

    style F fill:#2d8659,stroke:#333,color:#fff
    style G fill:#eb3b5a,stroke:#333,color:#fff

sequenceDiagram
    participant User
    participant OrderService as Order Service
    participant SNS as SNS Topic
    participant Invalidator as Cache Invalidator
    participant Cache as ElastiCache

    User->>OrderService: Purchase Demon Slayer Vol 1
    OrderService->>SNS: Publish purchase event
    SNS->>Invalidator: Trigger invalidation
    Invalidator->>Cache: DELETE reco:{user_id}:*

    Note over Cache: Next recommendation request<br/>will compute fresh results<br/>that exclude the purchased item

    Note over Invalidator: Target: < 60 seconds<br/>from purchase to invalidation

graph TD
    A["'What's a good horror manga?'"] --> B["Embed query"]
    B --> C{"Semantic similarity to<br/>cached query > 0.95?"}
    C -->|"Yes"| D["Return cached LLM response<br/>⏱️ 20ms | 💰 $0"]
    C -->|"No"| E["Full LLM pipeline<br/>⏱️ 1,500ms | 💰 $0.015"]

    F["'Recommend me some horror manga'"] --> B
    G["'Best horror manga series?'"] --> B

    style D fill:#2d8659,stroke:#333,color:#fff
    style E fill:#eb3b5a,stroke:#333,color:#fff

graph TD
    subgraph "Tier 1: FAQ Cache (Aggressive)"
        T1["Generic, non-personalized<br/>Similarity threshold: 0.92<br/>TTL: 1 hour<br/>Expected hit rate: 40%"]
    end

    subgraph "Tier 2: Product Info Cache (Moderate)"
        T2["Product-specific, not user-specific<br/>Similarity threshold: 0.95<br/>TTL: 15 min<br/>Expected hit rate: 25%"]
    end

    subgraph "Tier 3: Recommendation Cache (Conservative)"
        T3["User-specific cache key<br/>Similarity threshold: 0.98<br/>TTL: 15 min<br/>Expected hit rate: 10%"]
    end

    subgraph "Never Cached"
        T4["Multi-turn conversations<br/>Cart-dependent queries<br/>Price/availability queries"]
    end

    style T1 fill:#2d8659,stroke:#333,color:#fff
    style T2 fill:#fd9644,stroke:#333,color:#000
    style T3 fill:#f9d71c,stroke:#333,color:#000
    style T4 fill:#eb3b5a,stroke:#333,color:#fff

graph TD
    A["Cache Failure Modes"] --> B["Cache Stampede"]
    A --> C["Stale Serve"]
    A --> D["Cache Pollution"]
    A --> E["Memory Pressure"]

    B --> B1["Mitigation: Probabilistic<br/>early expiry + mutex lock<br/>for cache rebuild"]
    C --> C1["Mitigation: Event-driven<br/>invalidation + version tags<br/>+ TTL as safety net"]
    D --> D1["Mitigation: Eviction policy<br/>(allkeys-lru) + cache<br/>quality scoring"]
    E --> E1["Mitigation: Memory alerts<br/>at 70% + tiered eviction<br/>(evict low-value first)"]

    style B fill:#eb3b5a,stroke:#333,color:#fff
    style C fill:#fd9644,stroke:#333,color:#000
    style D fill:#f9d71c,stroke:#333,color:#000
    style E fill:#ff6b6b,stroke:#333,color:#000

sequenceDiagram
    participant R1 as Request 1
    participant R2 as Request 2
    participant R3 as Request 3
    participant Cache as ElastiCache
    participant Origin as Origin Service

    R1->>Cache: GET product:ASIN123
    Cache-->>R1: MISS
    R1->>Cache: SET lock:product:ASIN123 (TTL=5s)
    R1->>Origin: Fetch product data

    R2->>Cache: GET product:ASIN123
    Cache-->>R2: MISS
    R2->>Cache: SET lock:product:ASIN123
    Cache-->>R2: Lock already held
    Note over R2: Wait 50ms, retry cache

    R1->>Cache: SET product:ASIN123 (data, TTL=5min)
    Origin-->>R1: Product data

    R2->>Cache: GET product:ASIN123
    Cache-->>R2: HIT (fresh data)

    Note over R1,R3: Only one request hits origin.<br/>Others wait for cache population.