Scenarios and Runbooks: Resilient FM Systems

SCENARIO: [Title]
├── SITUATION: What happened and why
├── WHAT WENT WRONG: Root cause analysis
├── DETECTION: How the issue was (or should have been) detected
├── IMPACT: Business and user impact
├── RESPONSE RUNBOOK: Step-by-step mitigation
├── RESOLUTION: How to fix permanently
├── PREVENTION: How to prevent recurrence
├── EXAM ANGLE: What the AIP-C01 exam tests from this scenario
└── KEY TAKEAWAY: One-sentence lesson

Timeline:
  7:22:00 PM  Bedrock starts throttling (~30% of requests)
  7:22:01 PM  All 24 ECS tasks detect ThrottlingException
  7:22:01 PM  All 24 tasks begin retry attempt 1 (100ms delay)
  7:22:02 PM  All 24 tasks fire retry attempt 1 simultaneously
  7:22:02 PM  Bedrock throttle rate increases to 60% (overloaded by correlated retries)
  7:22:03 PM  All 24 tasks begin retry attempt 2 (200ms delay)
  7:22:04 PM  Retry storm: 24 tasks x 200ms = synchronized wave
  7:22:06 PM  All 24 tasks begin retry attempt 3 (400ms delay)
  7:22:10 PM  Retry attempt 4 (800ms delay) — Bedrock now throttling 90%
  7:22:18 PM  Retry attempt 5 (max 10s delay) — all retries exhausted
  7:22:18 PM  All 24 tasks switch to Haiku fallback simultaneously
  7:22:19 PM  Haiku now overwhelmed by 24 tasks' worth of redirected traffic
  7:22:25 PM  Haiku also starts throttling
  7:22:30 PM  Circuit breakers open across fleet — cache-only mode
  7:23:05 PM  Bedrock recovers, but circuit breakers remain open for 30s
  7:23:35 PM  Circuit breakers transition to HALF_OPEN
  7:23:36 PM  Probe requests succeed — circuit breakers close
  7:23:40 PM  Normal operation resumes

Without Jitter (the problem):
  Task 1:  |==|    |====|        |========|
  Task 2:  |==|    |====|        |========|
  Task 3:  |==|    |====|        |========|
  ...
  Task 24: |==|    |====|        |========|
           ↑ All fire at same instant ↑

With Decorrelated Jitter (the fix):
  Task 1:  |=|      |=======|         |==========|
  Task 2:  |===|  |====|        |===============|
  Task 3:  |==|       |=====|     |============|
  ...
  Task 24: |====|        |===|           |========|
           ↑ Spread across time window ↑

RUNBOOK: Exponential Backoff Storm Mitigation
SEVERITY: P2 (High — service degradation, not outage)
ON-CALL TEAM: MangaAssist Platform Engineering

STEP 1: CONFIRM THE INCIDENT (0-2 minutes)
  ├── Check CloudWatch dashboard: MangaAssist-FM-Health
  │   └── Look for: ErrorRate spike + TierCount shift away from Sonnet
  ├── Check X-Ray service map for error segments
  │   └── Filter: fault = true AND service("Bedrock Runtime")
  ├── Check Bedrock Service Health Dashboard
  │   └── https://health.aws.amazon.com/health/status
  └── Confirm: Is this a Bedrock regional issue or MangaAssist-specific?

STEP 2: IMMEDIATE MITIGATION (2-5 minutes)
  ├── If Bedrock is the source:
  │   ├── Enable "storm brake" — temporarily disable retries fleet-wide
  │   │   └── Set environment variable: RETRY_MAX_ATTEMPTS=0
  │   │   └── ECS will pick up on next health check (30s)
  │   ├── Force all traffic to Haiku (skip Sonnet entirely)
  │   │   └── Set environment variable: PRIMARY_MODEL=haiku
  │   └── If Haiku also affected: enable cache-only mode
  │       └── Set environment variable: FM_MODE=cache_only
  ├── If MangaAssist-specific (our retry logic):
  │   ├── Scale ECS tasks to 0 temporarily, then back to 12 (half fleet)
  │   │   └── aws ecs update-service --desired-count 0
  │   │   └── Wait 10 seconds
  │   │   └── aws ecs update-service --desired-count 12
  │   └── This breaks the synchronized retry pattern
  └── Monitor: Watch ErrorRate metric for improvement

STEP 3: VERIFY RECOVERY (5-10 minutes)
  ├── Confirm Sonnet invocations returning to normal
  ├── Check circuit breaker states: all should be CLOSED
  ├── Verify cache serve rate dropping back to baseline (~4%)
  ├── Check user-facing error rate back below 0.1%
  └── Confirm p99 latency below 3s SLA

STEP 4: POST-INCIDENT (within 24 hours)
  ├── Collect X-Ray traces from the incident window
  ├── Analyze retry patterns: Were they correlated?
  ├── Document: Timeline, impact, mitigation steps
  └── Schedule: Post-mortem meeting

Timeline:
  5:58:00 PM  Viral tweet posted (6.2M followers)
  5:58:30 PM  Traffic begins rising: 800 → 1,200 rps
  5:59:00 PM  Traffic: 1,800 rps (within limits)
  5:59:30 PM  Traffic: 3,200 rps (within burst, consuming tokens fast)
  5:59:45 PM  Burst tokens exhausted: 4,000 tokens consumed
  5:59:46 PM  Rate limiter begins returning 429 responses
  5:59:46 PM  Legitimate customers see: "You're sending messages too fast"
  6:00:00 PM  Traffic: 4,200 rps — 2,200 rps being throttled (52%)
  6:00:30 PM  Customer complaints spike on social media
  6:01:00 PM  Operations team alerted via CloudWatch alarm
  6:02:00 PM  Manual rate limit increase: 2,000 → 5,000 rps
  6:02:15 PM  Throttle rate drops to 0%
  6:05:00 PM  Traffic stabilizes at 3,800 rps
  6:15:00 PM  Traffic returns to 1,500 rps (above normal but manageable)
  6:30:00 PM  Normal operations

RUNBOOK: Rate Limiter Blocking Legitimate Traffic
SEVERITY: P1 (Critical — revenue-impacting, customer-facing)
ON-CALL TEAM: MangaAssist Platform Engineering + Customer Success

STEP 1: CONFIRM ORGANIC SPIKE vs. DDoS (0-2 minutes)
  ├── Check WAF metrics: Is traffic from diverse IPs or concentrated?
  │   └── Diverse IPs = organic | Few IPs = potential DDoS
  ├── Check geographic distribution: Mostly Japan? (Expected for manga)
  ├── Check request patterns: Normal chat queries or repetitive patterns?
  ├── Check social media: Is there a viral event driving traffic?
  └── Decision: If organic → increase limits | If DDoS → keep limits, enable WAF rules

STEP 2: INCREASE RATE LIMITS (2-5 minutes)
  ├── Global rate limit:
  │   └── aws apigateway update-usage-plan --usage-plan-id UP_ID \
  │       --patch-operations op=replace,path=/throttle/rateLimit,value=5000
  ├── Burst capacity:
  │   └── aws apigateway update-usage-plan --usage-plan-id UP_ID \
  │       --patch-operations op=replace,path=/throttle/burstLimit,value=8000
  ├── Scale ECS tasks:
  │   └── aws ecs update-service --desired-count 48 (double fleet)
  └── Verify: Monitor 429 rate dropping to 0%

STEP 3: PROTECT BEDROCK (concurrent with Step 2)
  ├── Check Bedrock service quotas: Are we near account limits?
  │   └── aws service-quotas get-service-quota --service-code bedrock ...
  ├── If near Bedrock limits:
  │   ├── Route 50% of traffic to Haiku (cheaper, faster)
  │   ├── Enable aggressive caching (extend TTLs temporarily)
  │   └── Consider: Bedrock on-demand throughput increase request
  └── Monitor: Bedrock throttle rate should remain below 5%

STEP 4: FIX USER-FACING MESSAGES (5-10 minutes)
  ├── Update throttle message for global rate limit:
  │   └── FROM: "You're sending messages too fast"
  │   └── TO: "We're experiencing high demand — you're in a queue!
  │            Estimated wait: ~30 seconds. While you wait, browse
  │            our Kaiju No. 8 collection directly."
  ├── Enable queue mode: Accept and queue requests rather than reject
  └── Add banner to UI: "High demand for Kaiju No. 8 — please be patient"

STEP 5: POST-SPIKE NORMALIZATION (30-60 minutes)
  ├── Monitor traffic returning to baseline
  ├── Gradually reduce rate limits back to normal
  ├── Scale ECS tasks back to standard count
  ├── Review: Were any customers permanently lost?
  └── Document: What threshold triggered the issue?

Timeline:
  Sat 9:00 AM   Sale begins — all prices show 30% discount
  Sat-Sun        Cache populated with sale-price responses (TTL: 4 hours)
  Sun 11:59 PM   Sale ends — DynamoDB prices updated to normal
  Mon 12:00 AM   Cache contains: mix of stale (sale) and fresh (normal) prices
  Mon 3:00 AM    Most sale-price cache entries expired (4hr TTL)
  Mon 3:30 AM    Cache warm-up job runs — repopulates with correct prices
  Mon 10:15 AM   Bedrock throttling event begins (3 minutes)
  Mon 10:15 AM   Fallback cascade: Sonnet fail → Haiku fail → Cache
  Mon 10:15 AM   PROBLEM: Some cache entries were refreshed at 9:00 AM
                  during the sale's "lingering" in one region, containing
                  sale prices that were manually re-added in error
  Mon 10:15 AM   ~180 users receive responses with incorrect (sale) prices
  Mon 10:18 AM   Bedrock recovers — fresh responses resume
  Mon 10:45 AM   Customer support reports: "Users saying prices differ
                  from what chatbot told them"
  Mon 11:00 AM   Investigation confirms stale cache served sale prices
  Mon 11:15 AM   Cache manually flushed for all pricing-related entries

Cache Entry Problem:

  User asks: "How much is One Piece Volume 105?"

  Fresh response (correct):
    "One Piece Volume 105 is ¥528 (regular price)."

  Stale cached response (incorrect):
    "One Piece Volume 105 is ¥369 (30% off — flash sale!)."
    ← This response was cached during the weekend sale
    ← Served during the Monday Bedrock throttle event

RUNBOOK: Stale Cache Serving Incorrect Data
SEVERITY: P2 (High — financial impact, customer trust)
ON-CALL TEAM: MangaAssist Platform Engineering + Customer Success

STEP 1: CONFIRM STALE DATA (0-5 minutes)
  ├── Identify affected cache entries:
  │   └── redis-cli KEYS "manga:cache:*" | sample 20 entries
  │   └── Check created_at timestamps — are any from before the sale ended?
  ├── Compare cached prices to DynamoDB source of truth:
  │   └── For each cached product_id, query DynamoDB for current price
  │   └── Flag entries where cached price != DynamoDB price
  ├── Estimate scope: How many cache entries are stale?
  └── Check: Is the issue ongoing or has the Bedrock throttle resolved?

STEP 2: FLUSH AFFECTED CACHE ENTRIES (5-10 minutes)
  ├── Option A: Flush ALL pricing-category cache entries
  │   └── redis-cli EVAL "local keys = redis.call('keys', 'manga:cache:*')
  │       for i, key in ipairs(keys) do
  │         local entry = redis.call('get', key)
  │         if string.find(entry, '\"category\":\"pricing\"') then
  │           redis.call('del', key)
  │         end
  │       end
  │       return 'done'" 0
  ├── Option B: Flush ALL cache entries (nuclear option)
  │   └── redis-cli FLUSHDB
  │   └── WARNING: This increases Bedrock load temporarily
  └── Verify: Spot-check that pricing queries now return fresh Bedrock responses

STEP 3: IDENTIFY AFFECTED CUSTOMERS (10-30 minutes)
  ├── Query CloudWatch Logs for cache-served pricing responses:
  │   └── Filter: tier="cached" AND category="pricing" AND
  │       timestamp BETWEEN "2024-XX-XX 10:15" AND "2024-XX-XX 10:18"
  ├── Extract user_ids from affected traces
  ├── For each user: What price were they quoted? What is the correct price?
  ├── Generate report: user_id, product, quoted_price, actual_price, delta
  └── Send report to Customer Success team for remediation

STEP 4: CUSTOMER REMEDIATION (same day)
  ├── Decision: Honor incorrect prices or issue apology + coupon?
  │   └── If delta < $10 per customer: Honor the quoted price
  │   └── If delta > $10: Contact customer with apology + 15% coupon
  ├── Send proactive email to affected customers:
  │   └── "We noticed a brief pricing display issue on [date]..."
  └── Update FAQ: Add note about the incident for customer support agents

STEP 5: PREVENT RECURRENCE (within 1 week)
  ├── Implement content-aware TTLs (pricing: 15 min max)
  ├── Add event-driven cache invalidation on DynamoDB price updates
  ├── Add staleness warning for cached pricing responses
  └── Add automated price-accuracy check for cached entries

Timeline:
  2:45 PM    New Bedrock error response begins appearing
  2:45 PM    Error rate increases from 0.1% to 0.4%
  2:45 PM    X-Ray captures 5% of all traces (including errors)
  2:45 PM    Of 0.3% error traffic, only 0.015% captured in X-Ray
  3:00 PM    CloudWatch alarm triggers: ErrorRate > 0.3%
  3:05 PM    On-call engineer checks X-Ray: finds 2 error traces
  3:05 PM    Engineer: "Looks like a rare transient — only 2 traces"
  3:05 PM    Engineer dismisses as transient (wrong conclusion)
  3:30 PM    Error rate persists at 0.4%
  4:00 PM    More customer complaints: "Error when searching manga with kanji"
  4:15 PM    Second investigation: CloudWatch Logs searched directly
  4:15 PM    FOUND: 4,300 error events in 1.5 hours (not 2!)
  4:20 PM    Root cause identified: New Bedrock error format for Unicode
  4:30 PM    Fix deployed: Updated error handler to recognize new format
  4:35 PM    Error rate returns to baseline

The Sampling Math Problem:

  Total traffic:  11.6 rps (1M messages/day)
  Error rate:     0.3% = 0.035 errors/second = ~125 errors/hour
  X-Ray sampling: 5%
  Captured errors: 0.05 x 125 = ~6 error traces/hour

  But: X-Ray groups samples by trace, not by error.
  The engineer saw only 2 error traces in a 15-minute window.
  Conclusion: "This is rare and transient" (WRONG)
  Reality: 125 errors/hour x 1.5 hours = ~188 errors before detection

  With 100% error sampling:
  The engineer would have seen 31 error traces in the same 15-minute window.
  Conclusion: "This is a consistent pattern affecting Unicode queries" (CORRECT)

RUNBOOK: X-Ray Sampling Missing Critical Errors
SEVERITY: P2 (High — errors reaching users, investigation delayed)
ON-CALL TEAM: MangaAssist Platform Engineering

STEP 1: DO NOT RELY SOLELY ON X-RAY FOR ERROR INVESTIGATION
  ├── ALWAYS cross-reference X-Ray with CloudWatch Logs
  ├── CloudWatch Logs: Complete record (no sampling)
  │   └── Log Insights query:
  │       fields @timestamp, error_code, error_message, user_query
  │       | filter level = "ERROR"
  │       | stats count(*) by error_code
  │       | sort count desc
  ├── X-Ray: Use for distributed trace context AFTER identifying the pattern
  └── Rule: If X-Ray shows few errors but CloudWatch alarm fired,
            trust CloudWatch — X-Ray may be undersampled

STEP 2: IDENTIFY ERROR PATTERN (0-10 minutes)
  ├── Run Log Insights query for error distribution:
  │   └── fields @timestamp, error_code, @message
  │       | filter level = "ERROR"
  │       | stats count(*) as error_count by bin(5m)
  │       | sort @timestamp desc
  ├── Look for: New error codes, changed error formats, specific patterns
  ├── Correlate with user input: Is there a common pattern in failing queries?
  └── Check: Did AWS release any changes to Bedrock API recently?

STEP 3: INCREASE X-RAY ERROR SAMPLING (immediate)
  ├── Create high-priority error sampling rule:
  │   └── aws xray create-sampling-rule --cli-input-json '{
  │         "SamplingRule": {
  │           "RuleName": "MangaAssist-AllErrors",
  │           "Priority": 1,
  │           "FixedRate": 1.0,
  │           "ReservoirSize": 100,
  │           "ServiceName": "MangaAssist",
  │           "ServiceType": "*",
  │           "Host": "*",
  │           "HTTPMethod": "*",
  │           "URLPath": "*",
  │           "ResourceARN": "*",
  │           "Version": 1
  │         }
  │       }'
  └── This captures 100% of traces going forward (useful for ongoing investigation)

STEP 4: FIX THE ERROR (once root cause identified)
  ├── Update error handler to recognize new error format
  ├── Add the new error code to the retryable set (if retryable)
  ├── Deploy fix to ECS tasks (rolling update, no downtime)
  └── Verify: Error rate returns to baseline

STEP 5: REVERT X-RAY SAMPLING (after incident resolved)
  ├── Remove the temporary 100% sampling rule
  ├── Ensure permanent error sampling rule exists:
  │   └── Priority: 50, FixedRate: 1.0, ReservoirSize: 10
  │   └── Filter: Errors only (via annotation or HTTP status)
  └── Verify: Normal sampling rate for success, 100% for errors

Graceful Degradation Message (version 1 — problematic):

  "I'm having a bit of trouble right now, but I'm still here!
   Could you try asking again in a moment? In the meantime,
   you can browse our manga catalog directly."

Problems with this message:
  1. "Try again in a moment" — Users spam retry, increasing load
  2. "I'm still here" — Implies the bot is working, just slow
  3. No estimated recovery time — Users don't know if it's 1 min or 1 hour
  4. "Browse our manga catalog directly" — Link not provided
  5. No acknowledgment that there IS a problem
  6. Same message for every query — "Are you ignoring my specific question?"

Timeline:
  8:30:00 PM  Bedrock outage begins
  8:30:05 PM  Circuit breakers open across ECS fleet
  8:30:05 PM  Cache serves 60% of queries (cache hit)
  8:30:05 PM  40% of queries receive graceful degradation message
  8:30:30 PM  Users retry repeatedly (message says "try again in a moment")
  8:31:00 PM  Retry traffic: +40% above normal (users following the message)
  8:31:00 PM  Social media: "MangaAssist is broken but pretending it's fine"
  8:32:00 PM  Support tickets: "The bot keeps saying try again but nothing works"
  8:33:00 PM  Some users interpret "I'm still here" as chatbot being sentient
  8:35:00 PM  Operations team adds maintenance banner to website
  8:37:00 PM  Updated degradation message deployed (version 2)
  8:42:00 PM  Bedrock recovers
  8:42:30 PM  Normal service resumes
  8:45:00 PM  Post-incident: 2,100 support tickets about "confusing bot messages"

User Experience Comparison:

  BAD (version 1):
  ┌─────────────────────────────────────────────────┐
  │ User: Where is my order #12345?                 │
  │                                                 │
  │ Bot: I'm having a bit of trouble right now,     │
  │ but I'm still here! Could you try asking again  │
  │ in a moment? In the meantime, you can browse    │
  │ our manga catalog directly.                     │
  │                                                 │
  │ User: I asked about my ORDER, not the catalog!  │
  │                                                 │
  │ Bot: I'm having a bit of trouble right now...   │
  │ [same message repeated]                         │
  └─────────────────────────────────────────────────┘

  GOOD (version 2):
  ┌─────────────────────────────────────────────────┐
  │ User: Where is my order #12345?                 │
  │                                                 │
  │ Bot: I've received your question about order    │
  │ #12345. Our systems are currently experiencing  │
  │ temporary difficulties and I'm unable to look   │
  │ up order details right now.                     │
  │                                                 │
  │ Here's what you can do:                         │
  │ • Track your order: mangaassist.jp/orders/12345 │
  │ • Contact support: support@mangaassist.jp       │
  │ • Call us: 0120-XXX-XXXX (9AM-9PM JST)         │
  │                                                 │
  │ I'll be fully back within a few minutes and     │
  │ will be happy to help then. Our team is         │
  │ actively working on this.                       │
  │                                                 │
  │ [Service Status: Temporary disruption]          │
  │ [Auto-retry in: 2 minutes]                      │
  └─────────────────────────────────────────────────┘

RUNBOOK: Graceful Degradation UX Issues
SEVERITY: P2 (High — customer experience, not technical failure)
ON-CALL TEAM: MangaAssist Platform Engineering + UX Team

STEP 1: DEPLOY IMPROVED DEGRADATION MESSAGE (0-5 minutes)
  ├── Switch to context-aware degradation message:
  │   └── Update ECS environment variable: DEGRADATION_MESSAGE_VERSION=v2
  │   └── V2 messages include:
  │       - Acknowledgment of the user's specific question type
  │       - Explicit service status ("Our AI assistant is temporarily unavailable")
  │       - Actionable alternatives with links
  │       - Estimated recovery time (if known)
  │       - NO "try again" language (prevents retry storms)
  ├── Add service status banner to chat UI:
  │   └── "Service Notice: AI assistant is in limited mode.
  │        You can still browse products and track orders."
  └── Disable auto-retry in client WebSocket (prevents automated retry loops)

STEP 2: REDUCE RETRY STORM (concurrent with Step 1)
  ├── Client-side: Update WebSocket handler to not auto-retry on degradation
  ├── Server-side: Add rate limit specifically for repeated identical queries
  │   └── Same user, same query hash within 60s → return cached degradation
  │   └── Do not re-process through the full fallback cascade
  └── Monitor: Watch retry rate metric for reduction

STEP 3: CUSTOMER COMMUNICATION (5-15 minutes)
  ├── Post service status on mangaassist.jp/status
  ├── Tweet from @MangaAssist: "We're aware of temporary AI assistant issues.
  │    You can still shop and track orders normally. We'll be fully back soon!"
  ├── Update in-app banner with current status
  └── Notify customer support team with pre-written response templates

STEP 4: EVALUATE MESSAGE EFFECTIVENESS (post-incident)
  ├── Survey users who received the degradation message
  ├── Analyze: Did users understand what was happening?
  ├── Analyze: Did users find the alternative actions helpful?
  ├── Compare retry rates between v1 and v2 messages
  └── A/B test improved message variants for future incidents

Anti-Pattern 1: Exponential Backoff Without Jitter
  Scenario: 1
  Effect: Thundering herd across fleet
  Fix: Decorrelated jitter + fleet retry budget

Anti-Pattern 2: Static Rate Limits Without Auto-Scale
  Scenario: 2
  Effect: Blocks legitimate organic traffic
  Fix: Dynamic limits with anomaly detection

Anti-Pattern 3: Uniform Cache TTL for All Content Types
  Scenario: 3
  Effect: Stale prices served from cache
  Fix: Content-aware TTLs + event-driven invalidation

Anti-Pattern 4: Uniform X-Ray Sampling for All Traffic
  Scenario: 4
  Effect: Errors invisible in traces
  Fix: 100% error sampling rule

Anti-Pattern 5: Engineer-Designed Degradation Messages
  Scenario: 5
  Effect: Retry storms + user confusion
  Fix: UX-designed, context-aware messages with alternatives

Topic: Exponential Backoff (Scenario 1)
  - AWS SDK retry modes: standard, adaptive, legacy
  - Jitter strategies: full, equal, decorrelated
  - Circuit breaker: closed → open → half-open → closed
  - Fleet-wide retry budget prevents thundering herd

Topic: Rate Limiting (Scenario 2)
  - API Gateway: account, stage, method throttling
  - Token bucket: rate (steady) + burst (peak) + quota (period)
  - Usage plans: per-tier access control
  - Dynamic scaling for organic traffic spikes

Topic: Fallback/Degradation (Scenarios 3, 5)
  - Fallback cascade: primary → secondary → cached → static → graceful
  - Content-aware caching: different TTLs for different data types
  - Cache invalidation: event-driven (DynamoDB Streams)
  - Graceful degradation: UX design, not just technical implementation

Topic: Observability (Scenario 4)
  - X-Ray: distributed tracing, service maps, annotations
  - Sampling: reservoir (guaranteed) + fixed rate (probabilistic)
  - Error sampling: separate rule, 100% capture
  - CloudWatch: metrics, logs, alarms, dashboards, anomaly detection