Resource Allocation Scenarios & Runbooks

flowchart TD
    A["ALERT: ECS CPU > 85%<br/>P95 Latency > 3,000ms"] --> B{"Is auto-scaling<br/>already responding?"}

    B -->|"Yes — tasks increasing"| C{"Current task count<br/>near max capacity?"}
    B -->|"No — no scaling activity"| D["CHECK: Is scaling policy<br/>attached and active?"]

    C -->|"Yes — at or near max"| E["IMMEDIATE: Increase<br/>max task count limit<br/>via ECS service update"]
    C -->|"No — room to grow"| F{"Are new tasks<br/>starting successfully?"}

    D --> D1["Fix: Verify auto-scaling<br/>target, re-attach policy"]
    D1 --> D2["Manual: Set desired count<br/>to immediate need"]

    F -->|"Yes — tasks pending"| G["WAIT: Fargate<br/>provisioning lag ~60-120s<br/>Monitor recovery"]
    F -->|"No — tasks failing"| H["CHECK: Task definition<br/>errors, ECR pull failures,<br/>health check failures"]

    E --> E1["Increase max to 200"]
    E1 --> E2{"CPU still > 90%<br/>after 5 min?"}
    E2 -->|"Yes"| E3["ESCALATE: Increase task<br/>size (4 vCPU / 8 GB)<br/>OR enable Bedrock throttle<br/>graceful degradation"]
    E2 -->|"No — recovering"| E4["Monitor for 30 min<br/>then review scaling policy"]

    G --> G1{"Latency recovering<br/>within 5 min?"}
    G1 -->|"Yes"| G2["Incident resolved.<br/>Post-mortem: tune scale-out<br/>aggressiveness"]
    G1 -->|"No"| E["Increase max capacity"]

    H --> H1["Fix task definition<br/>or container image issues"]
    H1 --> D2

1. Increase max task count: 50 → 200 (ECS service update)
2. Set desired count manually: 50 → 120 (skip auto-scaling wait)
3. Enable graceful degradation: route all traffic to Haiku (faster, cheaper)
4. Activate semantic cache aggressive mode: lower similarity threshold 0.92 → 0.85

5. Verify all 120 tasks are healthy and serving
6. Monitor CPU stabilization below 70%
7. Monitor latency recovery to < 2,000ms P95
8. Re-enable Sonnet routing once load stabilizes

flowchart TD
    A["ALERT: DynamoDB<br/>ReadThrottleEvents > 0"] --> B{"Table mode?"}

    B -->|"Provisioned"| C{"Auto-scaling<br/>active and responding?"}
    B -->|"On-Demand"| D["Check for<br/>hot partition keys"]

    C -->|"Yes — scaling up"| E{"Current RCU near<br/>account/table limit?"}
    C -->|"No — not scaling"| F["CHECK: Auto-scaling<br/>policy configuration"]

    E -->|"Yes — near limit"| G["Request limit increase<br/>via AWS Support"]
    E -->|"No — has headroom"| H{"Throttle events<br/>persisting > 5 min?"}

    H -->|"Yes"| I["Hot partition detected.<br/>Check partition key<br/>distribution."]
    H -->|"No"| J["Scaling caught up.<br/>Monitor recovery."]

    F --> F1["Fix: Re-attach auto-scaling<br/>Set target utilization 70%"]
    F1 --> F2["Manual: Increase provisioned<br/>RCU immediately"]

    D --> D1{"Throttle on specific<br/>partition keys?"}
    D1 -->|"Yes"| D2["Hot partition: session_id<br/>or user_id skew.<br/>Add write sharding suffix."]
    D1 -->|"No"| D3["On-demand table hit<br/>2x previous peak limit.<br/>Gradually increase traffic<br/>or pre-warm."]

    I --> I1["Redesign partition key:<br/>Add shard suffix<br/>or use composite key<br/>(user_region#user_id)"]

    G --> G1["SHORT-TERM: Switch<br/>table to On-Demand mode<br/>(takes effect immediately)"]

1. Switch table to On-Demand mode (AWS Console or CLI — takes effect in minutes)
2. If staying provisioned: manually set RCU to 10,000 (well above current need)
3. Enable application-level retry with exponential backoff (should already exist)
4. If session reads fail after 3 retries: serve with empty context (degraded but functional)

5. Verify throttle events dropped to zero
6. Analyze partition key distribution using CloudWatch Contributor Insights
7. If hot partitions confirmed: plan partition key redesign
8. Cost-compare on-demand vs provisioned for this traffic pattern

flowchart TD
    A["ALERT: OpenSearch<br/>P95 Latency > 100ms"] --> B{"Is search traffic<br/>elevated vs baseline?"}

    B -->|"Yes — traffic spike"| C{"Are OCUs<br/>auto-scaling up?"}
    B -->|"No — normal traffic"| D["Check index health:<br/>segment count, merge<br/>activity, shard imbalance"]

    C -->|"Yes — OCUs increasing<br/>(visible in CloudWatch)"| E["WAIT: OCU provisioning<br/>takes 5-15 min.<br/>Enable mitigations."]
    C -->|"No — OCUs static"| F["CHECK: Collection<br/>auto-scaling config.<br/>Is max OCU set too low?"]

    E --> E1["MITIGATE while waiting:"]
    E1 --> E2["1. Reduce k in k-NN<br/>from 10 to 5"]
    E1 --> E3["2. Increase cache<br/>aggressiveness for<br/>search results"]
    E1 --> E4["3. Route simple queries<br/>to keyword search<br/>instead of vector search"]

    E2 --> G{"Latency recovered<br/>after OCU scale?"}
    G -->|"Yes"| H["Resolved. Review base<br/>OCU allocation."]
    G -->|"No"| I["ESCALATE: Check<br/>HNSW index size vs OCU<br/>memory. May need<br/>index optimization."]

    F --> F1["Increase max search OCU<br/>from 4 to 20"]
    F1 --> F2["If still not scaling:<br/>Check if collection<br/>type supports auto-scaling"]

    D --> D1["Run _cat/segments<br/>Check for large<br/>segment merges"]
    D1 --> D2["If merge storms:<br/>Reduce refresh interval<br/>or schedule bulk indexing<br/>during off-peak"]

1. Reduce k-NN k parameter from 10 to 5 (fewer neighbors = faster search)
2. Enable search result caching in Redis (cache top queries for 5 min TTL)
3. Route FAQ-intent queries to keyword search (bypass vector search)
4. If available: increase max search OCU limit in collection settings

5. Increase base search OCU to 4 (prevents cold-start lag for moderate spikes)
6. Implement search result caching layer in orchestrator
7. Analyze query patterns: are users sending identical queries? Pre-compute and cache
8. Review HNSW index parameters (ef_search, m) for efficiency

flowchart TD
    A["ALERT: ElastiCache Memory > 80%"] --> B{"Eviction policy?"}

    B -->|"noeviction"| C["CRITICAL: Redis will reject<br/>writes at 100% memory.<br/>Change policy immediately."]
    B -->|"allkeys-lfu or allkeys-lru"| D{"Eviction rate<br/>abnormally high?"}

    C --> C1["IMMEDIATE: Change maxmemory-policy<br/>to allkeys-lfu<br/>(no restart needed)"]
    C1 --> C2{"Memory still > 95%<br/>after policy change?"}

    C2 -->|"Yes — eviction not<br/>fast enough"| C3["Manually flush low-value<br/>cache entries by pattern:<br/>SCAN + DEL old TTL keys"]
    C2 -->|"No — eviction working"| C4["Monitor. Evictions will<br/>keep memory in check."]

    D -->|"Yes — evicting too much"| E{"Cache growing due to<br/>new entry volume or<br/>entry size increase?"}
    D -->|"No — healthy eviction"| F["System healthy.<br/>Adjust alarm threshold."]

    E -->|"New entries (unique queries)"| G["Check: Are low-value<br/>queries being cached?<br/>Add minimum similarity<br/>threshold for caching."]
    E -->|"Entry size growth"| H["Check: Are cached responses<br/>getting larger? Review<br/>response truncation policy."]

    G --> G1["Implement selective caching:<br/>Only cache queries with<br/>>5 occurrences in 24hr"]

    C3 --> C5["PLAN: Upgrade node type<br/>to r6g.xlarge (26 GB)<br/>or enable cluster mode"]

1. Change maxmemory-policy to allkeys-lfu:
   CONFIG SET maxmemory-policy allkeys-lfu
   (takes effect immediately, no restart)

2. Set TTL on all existing keys without one:
   Use SCAN to iterate + EXPIRE on each key (24hr TTL)

3. If memory at 100% and write errors occurring:
   Flush low-frequency keys manually:
   Use SCAN + OBJECT FREQ + DEL pattern

4. Monitor: Eviction count should start rising, memory should stabilize

5. Implement selective caching logic in orchestrator:
   - Only cache queries with similarity < 0.95 to existing entries
   - Only cache queries seen > 3 times in rolling 24hr window
6. Set default TTL of 24 hours on all new cache entries
7. Add CloudWatch alarm: memory > 75% for 10 min → warning
8. Add CloudWatch alarm: memory > 85% for 5 min → critical

flowchart TD
    A["ALERT: Daily cost<br/>280% over budget"] --> B["Identify cost drivers<br/>via Cost Explorer<br/>by service and tag"]

    B --> C{"Which services<br/>are over-budget?"}

    C --> D["ECS: 100 tasks running<br/>at 12% CPU utilization"]
    C --> E["Bedrock: Provisioned throughput<br/>at event levels"]
    C --> F["OpenSearch: 12 OCU<br/>vs 4 needed"]

    D --> D1{"Auto-scaling policy<br/>scale-in working?"}
    D1 -->|"Yes but min capacity<br/>was raised for event"| D2["Reset min capacity<br/>from 100 → 10"]
    D1 -->|"No — scale-in disabled<br/>or cooldown too long"| D3["Fix scale-in policy:<br/>Enable, reduce cooldown"]

    E --> E1["Review provisioned throughput:<br/>Is it auto-managed or manual?"]
    E1 --> E2["Manual: Reduce to normal<br/>levels immediately"]
    E1 --> E3["If using on-demand: costs<br/>scale automatically (no action)"]

    F --> F1["OpenSearch auto-scaling<br/>should reduce OCU.<br/>Check if collection has<br/>min OCU set too high."]
    F1 --> F2["Reset min search OCU<br/>to 2 if manually raised"]

    D2 --> G["VERIFY: All services<br/>returning to baseline<br/>capacity within 1 hour"]
    D3 --> G
    E2 --> G
    F2 --> G

    G --> H["POST-MORTEM: Why wasn't<br/>this caught sooner?<br/>Add cost monitoring."]

    H --> I["Implement: Daily cost<br/>anomaly alert + weekly<br/>utilization efficiency report"]

1. Reset ECS min_capacity: 100 → 10
2. Reset ECS desired count: let auto-scaling determine (remove manual override)
3. Reduce Bedrock provisioned throughput to normal levels
4. Verify OpenSearch search OCU is auto-scaling down
5. Estimated savings: ~$3,800/day immediately

6. Implement daily cost anomaly detection (AWS Cost Anomaly Detection)
7. Create post-event scaling revert checklist in runbook
8. Add automated revert: EventBridge rule that resets min_capacity
   48 hours after a manual override event
9. Create weekly "efficiency report" dashboard:
   - Service utilization vs capacity
   - Cost per message trend
   - Over-provisioned resource flags

quadrantChart
    title Scenario Severity vs Detection Urgency
    x-axis Low Urgency --> High Urgency
    y-axis Low Severity --> High Severity
    quadrant-1 Critical — Immediate action
    quadrant-2 Important — Action within hours
    quadrant-3 Monitor — Track and prevent
    quadrant-4 Urgent — Fix fast, low blast radius
    ECS under-provisioned: [0.85, 0.90]
    DynamoDB throttling: [0.80, 0.75]
    OpenSearch OCU lag: [0.65, 0.60]
    ElastiCache memory: [0.45, 0.55]
    Cost over-provision: [0.25, 0.70]