Scenarios and Runbooks — Intelligent Model Routing

14:00 JST — Daily cost tracker shows $2,847 spend (expected: $1,900 by this hour)
14:05 JST — CloudWatch alarm: Sonnet invocations/minute = 312 (baseline: 70)
14:10 JST — Haiku invocations/minute dropped from 620 to 380
14:15 JST — Cost anomaly detector fires (150% of projected daily budget)
14:22 JST — PagerDuty alert to on-call engineer
14:25 JST — Investigation begins
14:35 JST — Root cause identified: complexity threshold = 4.0 (should be 6.5)
14:40 JST — Threshold corrected, cache invalidated
14:50 JST — Sonnet routing rate returns to normal (11%)
15:00 JST — Cost burn rate normalized

flowchart TB
    subgraph Problem["Problem Chain"]
        CONFIG[DynamoDB Config Update<br/>threshold: 6.5 → 4.0] -->|5-min cache TTL| CACHE[Redis Cache Refresh]
        CACHE --> SCORER[ComplexityScorer<br/>Now: score >= 4.0 → Sonnet]
        SCORER -->|Moderate queries routed up| SONNET[Sonnet Invocations<br/>70/min → 312/min]
        SONNET --> COST[Cost Spike<br/>$1,900 → $4,200 projected]
    end

    subgraph Cascade["Cascade Effects"]
        SONNET -->|Higher latency| LATENCY[P95 Latency<br/>1.2s → 2.8s]
        SONNET -->|Throttling risk| THROTTLE[Bedrock Throttle Rate<br/>0.1% → 3.2%]
        LATENCY -->|Timeout increase| UX[User Experience<br/>Slower responses]
    end

    style CONFIG fill:#f44336,color:#fff
    style SONNET fill:#FF9800,color:#fff
    style COST fill:#f44336,color:#fff
    style LATENCY fill:#FF9800,color:#fff

# Verify current production threshold value
aws dynamodb get-item \
  --table-name MangaAssist-RoutingConfig \
  --key '{"PK": {"S": "GLOBAL#config"}, "SK": {"S": "THRESHOLD#complexity"}}' \
  --region ap-northeast-1 \
  --query 'Item.sonnet_threshold.N'

# Restore correct threshold
aws dynamodb update-item \
  --table-name MangaAssist-RoutingConfig \
  --key '{"PK": {"S": "GLOBAL#config"}, "SK": {"S": "THRESHOLD#complexity"}}' \
  --update-expression "SET sonnet_threshold = :val, updated_at = :ts, updated_by = :who" \
  --expression-attribute-values '{
    ":val": {"N": "6.5"},
    ":ts": {"S": "2026-03-31T14:35:00Z"},
    ":who": {"S": "oncall-engineer@mangaassist.jp"}
  }' \
  --region ap-northeast-1

# Invalidate Redis cache for the threshold config
redis-cli -h mangaassist-redis.xxxxx.apne1.cache.amazonaws.com \
  DEL "config:complexity_threshold"

# Publish cache invalidation event to all ECS tasks
redis-cli -h mangaassist-redis.xxxxx.apne1.cache.amazonaws.com \
  PUBLISH "routing:cache_invalidation" '{"key": "complexity_threshold", "action": "refresh"}'

# Check Sonnet vs Haiku invocation rates (last 10 minutes)
aws cloudwatch get-metric-statistics \
  --namespace MangaAssist/Routing \
  --metric-name InvocationCount \
  --dimensions Name=ModelId,Value=anthropic.claude-3-sonnet-20240229-v1:0 \
  --start-time $(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 \
  --statistics Sum \
  --region ap-northeast-1

# Get today's Bedrock cost breakdown
aws ce get-cost-and-usage \
  --time-period Start=$(date -u +%Y-%m-%d),End=$(date -u -d 'tomorrow' +%Y-%m-%d) \
  --granularity DAILY \
  --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["Amazon Bedrock"]}}' \
  --metrics BlendedCost

Day 0, 10:00 JST — Claude 3.5 Sonnet enabled in Bedrock account
Day 0, 10:30 JST — Integration tests pass against new model (direct invocation)
Day 0, 11:00 JST — Deployment marked as "complete" in Jira ticket
Day 0-3          — All production traffic continues to old model (no alerts)
Day 3, 15:00 JST — A/B test report shows 0 queries to new model
Day 3, 15:30 JST — Investigation reveals routing table not updated
Day 3, 16:00 JST — 23 DynamoDB items identified as stale
Day 3, 17:00 JST — Bulk update applied, canary testing started

flowchart TB
    subgraph Deployment["Deployment Actions (Completed)"]
        ENABLE[Bedrock Model Access<br/>Claude 3.5 Sonnet Enabled]
        TEST[Integration Tests<br/>Passed]
        DASH[Dashboard Updated]
    end

    subgraph Missed["Deployment Actions (MISSED)"]
        DDB_UPDATE[DynamoDB Route Update<br/>NOT DONE]
        CACHE_INV[Redis Cache Invalidation<br/>NOT DONE]
        VERIFY[Traffic Verification<br/>NOT DONE]
    end

    subgraph Production["Production State"]
        DDB[DynamoDB Routing Table<br/>Still: claude-3-sonnet-v1:0]
        REDIS[Redis Cache<br/>Still: claude-3-sonnet-v1:0]
        ROUTER[StaticRouter<br/>Routes to OLD model]
        OLD[Claude 3 Sonnet<br/>RECEIVING ALL TRAFFIC]
        NEW[Claude 3.5 Sonnet<br/>ZERO TRAFFIC]
    end

    ENABLE -.->|Should trigger| DDB_UPDATE
    DDB_UPDATE -.->|Should trigger| CACHE_INV
    CACHE_INV -.->|Should trigger| VERIFY

    DDB --> REDIS
    REDIS --> ROUTER
    ROUTER --> OLD

    style DDB_UPDATE fill:#f44336,color:#fff
    style CACHE_INV fill:#f44336,color:#fff
    style VERIFY fill:#f44336,color:#fff
    style OLD fill:#FF9800,color:#fff
    style NEW fill:#9E9E9E,color:#fff

# Scan routing table for entries referencing old model
aws dynamodb scan \
  --table-name MangaAssist-RoutingConfig \
  --filter-expression "contains(model_id, :old_model)" \
  --expression-attribute-values '{":old_model": {"S": "claude-3-sonnet-20240229"}}' \
  --projection-expression "PK, SK, model_id, intent_category, sub_intent" \
  --region ap-northeast-1

"""
Bulk update routing table entries from old model to new model.
Run with: python update_model_routes.py --dry-run first, then without --dry-run.
"""

import sys
import boto3
from datetime import datetime

OLD_MODEL = "anthropic.claude-3-sonnet-20240229-v1:0"
NEW_MODEL = "anthropic.claude-3-5-sonnet-20241022-v2:0"
TABLE_NAME = "MangaAssist-RoutingConfig"
REGION = "ap-northeast-1"

dry_run = "--dry-run" in sys.argv

dynamodb = boto3.resource("dynamodb", region_name=REGION)
table = dynamodb.Table(TABLE_NAME)

# Scan for stale entries
response = table.scan(
    FilterExpression="model_id = :old",
    ExpressionAttributeValues={":old": OLD_MODEL},
)

items = response.get("Items", [])
print(f"Found {len(items)} entries to update")

for item in items:
    pk = item["PK"]
    sk = item["SK"]
    intent = item.get("intent_category", "unknown")
    sub = item.get("sub_intent", "default")

    print(f"  {'[DRY RUN] ' if dry_run else ''}Updating {intent}:{sub} → {NEW_MODEL}")

    if not dry_run:
        table.update_item(
            Key={"PK": pk, "SK": sk},
            UpdateExpression="SET model_id = :new, updated_at = :ts, updated_by = :who",
            ExpressionAttributeValues={
                ":new": NEW_MODEL,
                ":ts": datetime.utcnow().isoformat(),
                ":who": "model-migration-script",
            },
        )

print(f"{'[DRY RUN] ' if dry_run else ''}Updated {len(items)} entries")

# Flush all route cache keys
redis-cli -h mangaassist-redis.xxxxx.apne1.cache.amazonaws.com \
  EVAL "local keys = redis.call('keys', 'route:*'); for i,k in ipairs(keys) do redis.call('del', k) end; return #keys" 0

# Publish global cache invalidation
redis-cli -h mangaassist-redis.xxxxx.apne1.cache.amazonaws.com \
  PUBLISH "routing:cache_invalidation" '{"action": "flush_all", "reason": "model_migration"}'

# Use RoutingTableManager to stage the new routes with 5% canary traffic
# Monitor for 30 minutes before promoting
python -c "
from routing_table_manager import RoutingTableManager
mgr = RoutingTableManager()
for route in mgr.get_active_routes():
    if 'claude-3-5-sonnet' in route.model_id:
        mgr.start_test(route.route_id, traffic_pct=5.0, test_duration_minutes=30)
"

08:55 JST — Morning traffic ramp begins (Japan business hours)
09:05 JST — Kinesis shard iterator age rises above 30 seconds
09:10 JST — Metric aggregation Lambda falling behind by 2 minutes
09:15 JST — Redis metrics still showing overnight P95 latency (800ms for Sonnet)
09:20 JST — Actual Sonnet P95 latency at 3.1s (MetricBasedSelector unaware)
09:25 JST — User timeout rate crosses 2% (3-second WebSocket timeout)
09:30 JST — Customer complaints in support queue: "chatbot is slow"
09:35 JST — Kinesis iterator age at 8 minutes behind real-time
09:40 JST — On-call engineer paged via CloudWatch alarm on timeout rate
09:45 JST — Root cause identified: stale metrics driving bad routing
09:50 JST — Emergency: force all routing to Haiku via static override
09:55 JST — Timeout rate drops to 0.3%
10:15 JST — Kinesis resharding completes (2 → 4 shards)
10:30 JST — Metric pipeline catches up, dynamic routing re-enabled

flowchart TB
    subgraph Trigger["Traffic Spike"]
        SPIKE[Morning Ramp<br/>700 → 2,500 msg/min]
    end

    subgraph Pipeline["Metric Pipeline (Broken)"]
        KDS[Kinesis 2 Shards<br/>THROTTLED]
        LAMBDA[Lambda Aggregator<br/>8 min BEHIND]
        REDIS_STALE[Redis Metrics<br/>STALE: P95=800ms]
    end

    subgraph Routing["Routing (Bad Decisions)"]
        SELECTOR[MetricBasedSelector<br/>Trusts Stale Metrics]
        SONNET[Routes to Sonnet<br/>Actual P95=4.2s]
    end

    subgraph User["User Impact"]
        TIMEOUT[WebSocket Timeout<br/>3s limit exceeded]
        ERROR[User Error Rate<br/>0.1% → 2%+]
    end

    SPIKE --> KDS
    KDS -->|Throttled| LAMBDA
    LAMBDA -->|Can't update| REDIS_STALE
    REDIS_STALE --> SELECTOR
    SELECTOR --> SONNET
    SONNET --> TIMEOUT
    TIMEOUT --> ERROR

    style KDS fill:#f44336,color:#fff
    style REDIS_STALE fill:#FF9800,color:#fff
    style TIMEOUT fill:#f44336,color:#fff
    style ERROR fill:#f44336,color:#fff

# Set emergency routing override: all traffic to Haiku
aws dynamodb put-item \
  --table-name MangaAssist-RoutingConfig \
  --item '{
    "PK": {"S": "GLOBAL#override"},
    "SK": {"S": "EMERGENCY#static_only"},
    "override_type": {"S": "force_haiku"},
    "enabled": {"BOOL": true},
    "reason": {"S": "Metric pipeline lag causing timeout spike"},
    "activated_by": {"S": "oncall-engineer"},
    "activated_at": {"S": "2026-03-31T09:50:00+09:00"},
    "ttl": {"N": "1711872600"}
  }' \
  --region ap-northeast-1

# Invalidate metric-based selector cache
redis-cli -h mangaassist-redis.xxxxx.apne1.cache.amazonaws.com \
  SET "routing:override" '{"mode": "static_only", "model": "haiku"}' EX 3600

# Publish override event
redis-cli -h mangaassist-redis.xxxxx.apne1.cache.amazonaws.com \
  PUBLISH "routing:emergency" '{"action": "force_haiku", "reason": "metric_lag"}'

# Check current shard count and iterator age
aws kinesis describe-stream-summary \
  --stream-name MangaAssist-Metrics \
  --region ap-northeast-1

# Option A: Enable on-demand mode (recommended for variable traffic)
aws kinesis update-stream-mode \
  --stream-arn arn:aws:kinesis:ap-northeast-1:ACCOUNT:stream/MangaAssist-Metrics \
  --stream-mode-details StreamMode=ON_DEMAND \
  --region ap-northeast-1

# Option B: Manual reshard to 4 shards
aws kinesis update-shard-count \
  --stream-name MangaAssist-Metrics \
  --target-shard-count 4 \
  --scaling-type UNIFORM_SCALING \
  --region ap-northeast-1

# Add this check to MetricBasedSelector.get_model_metrics()
def _is_metric_stale(self, metrics: ModelMetrics, max_age_seconds: int = 120) -> bool:
    """Check if metrics are too old to trust."""
    if not metrics.last_updated:
        return True
    try:
        from datetime import datetime
        updated = datetime.fromisoformat(metrics.last_updated)
        age = (datetime.utcnow() - updated).total_seconds()
        return age > max_age_seconds
    except (ValueError, TypeError):
        return True

# In rank_models(), add staleness handling:
def rank_models(self, query_type="general", override_weights=None):
    # ... existing code ...
    for model_id, metrics in all_metrics.items():
        if self._is_metric_stale(metrics):
            logger.warning("Stale metrics for %s — using conservative scores", model_id)
            # Penalize latency score for stale metrics (assume worst case)
            latency_score = 0.3  # Conservative — don't trust stale latency
            # ... rest of scoring uses conservative defaults

# Verify Kinesis iterator age is back to near-real-time
aws cloudwatch get-metric-statistics \
  --namespace AWS/Kinesis \
  --metric-name GetRecords.IteratorAgeMilliseconds \
  --dimensions Name=StreamName,Value=MangaAssist-Metrics \
  --start-time $(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 \
  --statistics Maximum \
  --region ap-northeast-1

# Remove emergency override once metrics are fresh
redis-cli -h mangaassist-redis.xxxxx.apne1.cache.amazonaws.com \
  DEL "routing:override"

aws dynamodb delete-item \
  --table-name MangaAssist-RoutingConfig \
  --key '{"PK": {"S": "GLOBAL#override"}, "SK": {"S": "EMERGENCY#static_only"}}' \
  --region ap-northeast-1

11:00 JST — API Gateway deployment via CloudFormation stack update
11:05 JST — VTL template deployed with missing closing parenthesis
11:10 JST — First requests hit ECS without routing headers
11:15 JST — Dynamic router processing 100% of traffic (expected: 40%)
11:20 JST — Average response latency rises from 0.8s to 1.2s
11:30 JST — P95 latency rises from 1.5s to 2.4s
11:45 JST — CloudWatch alarm: "AverageLatency > 1.0s for 15 minutes"
11:50 JST — Investigation begins
12:00 JST — Root cause: VTL template dropping headers
12:10 JST — Rollback API Gateway deployment to previous stage
12:15 JST — Latency returns to normal

## Broken VTL (line 14)
#set($route = $util.escapeJavaScript($input.path('$.route'))    <-- missing closing paren

## Correct VTL (line 14)
#set($route = $util.escapeJavaScript($input.path('$.route')))

flowchart TB
    subgraph Deploy["API Gateway Deployment"]
        CFN[CloudFormation Update]
        VTL[VTL Template<br/>SYNTAX ERROR]
    end

    subgraph Gateway["API Gateway (Broken)"]
        WS[WebSocket $connect]
        TRANSFORM[Request Transform<br/>Headers: EMPTY]
    end

    subgraph ECS["ECS Fargate"]
        RECEIVE[Receive Request<br/>No routing headers]
        FALLBACK[Default: Full Dynamic Analysis<br/>100% of traffic]
    end

    subgraph Impact["Impact"]
        LATENCY[Latency Increase<br/>Greetings: 0.3s → 1.5s]
        COMPUTE[Unnecessary Compute<br/>ComplexityScorer on trivial queries]
        COST[Wasted Analysis Cost<br/>Step Functions invocations up 150%]
    end

    CFN --> VTL
    VTL --> WS
    WS --> TRANSFORM
    TRANSFORM -->|Empty headers| RECEIVE
    RECEIVE --> FALLBACK
    FALLBACK --> LATENCY
    FALLBACK --> COMPUTE
    FALLBACK --> COST

    style VTL fill:#f44336,color:#fff
    style TRANSFORM fill:#FF9800,color:#fff
    style FALLBACK fill:#FF9800,color:#fff

# Export current API Gateway stage configuration
aws apigatewayv2 get-integration \
  --api-id YOUR_API_ID \
  --integration-id YOUR_INTEGRATION_ID \
  --region ap-northeast-1

# Check recent deployments
aws apigatewayv2 get-deployments \
  --api-id YOUR_API_ID \
  --region ap-northeast-1 \
  --max-results 5

# List stages to find previous deployment
aws apigatewayv2 get-stages \
  --api-id YOUR_API_ID \
  --region ap-northeast-1

# Update stage to point to previous (working) deployment
aws apigatewayv2 update-stage \
  --api-id YOUR_API_ID \
  --stage-name production \
  --deployment-id PREVIOUS_DEPLOYMENT_ID \
  --region ap-northeast-1

# Corrected VTL template (fix the missing parenthesis)
# Then redeploy via CloudFormation with the fixed template
aws cloudformation update-stack \
  --stack-name MangaAssist-APIGateway \
  --template-body file://corrected-template.yaml \
  --region ap-northeast-1

# Add to the ECS route controller: validate routing headers
def validate_routing_headers(self, headers: Dict[str, str]) -> bool:
    """Check that API Gateway sent valid routing headers."""
    required = ["X-Route-Target", "X-User-Tier"]
    for header in required:
        value = headers.get(header, "")
        if not value or value.strip() == "":
            logger.warning(
                "Missing routing header: %s — API Gateway transform may be broken",
                header,
            )
            # Emit metric for monitoring
            self._emit_metric("missing_routing_header", 1, {"header": header})
            return False
    return True

Day 1, 09:00 JST — Shadow routing config deployed via feature flag
Day 1, 09:05 JST — ShadowRouter starts with shadow_traffic_pct=100 (intended: 10)
Day 1, 09:30 JST — Bedrock invocation count doubles (Haiku + Sonnet per query)
Day 1, 12:00 JST — Midday cost check shows $1,600 spend (expected: $800)
Day 1, 14:00 JST — AWS Budgets alert: 200% of daily Bedrock forecast
Day 1, 14:15 JST — Primary (Haiku) P95 latency increases 30% due to shared throughput
Day 1, 14:30 JST — Investigation begins
Day 1, 14:45 JST — Root cause: shadow_traffic_pct=100, shadow_model=Sonnet
Day 1, 14:50 JST — Shadow routing disabled via feature flag
Day 1, 15:00 JST — Costs and latency return to baseline
Day 1 end       — Total overspend: ~$2,400 (one full day of Sonnet shadow)

{
  "shadow_enabled": true,
  "shadow_model_id": "anthropic.claude-3-sonnet-20240229-v1:0",
  "shadow_traffic_pct": 100,
  "shadow_max_tokens": 1024
}

flowchart TB
    subgraph Config["Misconfiguration"]
        FLAG[Feature Flag<br/>shadow_traffic_pct: 100<br/>shadow_model: Sonnet]
    end

    subgraph Normal["Normal Path (Still Working)"]
        QUERY[Every Query<br/>1M/day]
        HAIKU[Haiku Primary<br/>$625/day]
        RESPONSE[User Response<br/>From Haiku]
    end

    subgraph Shadow["Shadow Path (Unintended 100%)"]
        SHADOW[ShadowRouter<br/>100% duplication]
        SONNET[Sonnet Shadow<br/>$7,500/day<br/>RESPONSES DISCARDED]
    end

    subgraph Impact["Cost + Performance Impact"]
        COST[Daily Cost<br/>$625 → $8,125<br/>13x increase]
        THROUGHPUT[Bedrock Throughput<br/>Shared contention]
        LATENCY[Haiku P95 Latency<br/>+30% from contention]
    end

    FLAG --> SHADOW
    QUERY --> HAIKU
    QUERY --> SHADOW
    HAIKU --> RESPONSE
    SHADOW --> SONNET
    SONNET -->|Discarded| WASTE[Wasted Responses]

    SONNET --> COST
    SONNET --> THROUGHPUT
    THROUGHPUT --> LATENCY

    style FLAG fill:#f44336,color:#fff
    style SONNET fill:#f44336,color:#fff
    style COST fill:#f44336,color:#fff
    style WASTE fill:#9E9E9E,color:#fff

# Disable shadow via feature flag
aws dynamodb update-item \
  --table-name MangaAssist-FeatureFlags \
  --key '{"PK": {"S": "FLAG#shadow_routing"}, "SK": {"S": "CONFIG"}}' \
  --update-expression "SET shadow_enabled = :disabled, disabled_by = :who, disabled_at = :ts, disable_reason = :reason" \
  --expression-attribute-values '{
    ":disabled": {"BOOL": false},
    ":who": {"S": "oncall-engineer"},
    ":ts": {"S": "2026-03-31T14:50:00+09:00"},
    ":reason": {"S": "Cost spike - shadow_traffic_pct was 100 instead of 10"}
  }' \
  --region ap-northeast-1

# Force immediate effect via Redis
redis-cli -h mangaassist-redis.xxxxx.apne1.cache.amazonaws.com \
  SET "feature:shadow_routing:enabled" "false" EX 3600

# Check Sonnet invocation rate (should drop to near-zero for shadow)
aws cloudwatch get-metric-statistics \
  --namespace MangaAssist/Routing \
  --metric-name ShadowInvocationCount \
  --start-time $(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 \
  --statistics Sum \
  --region ap-northeast-1

# Get today's Bedrock cost
aws ce get-cost-and-usage \
  --time-period Start=$(date -u +%Y-%m-%d),End=$(date -u -d 'tomorrow' +%Y-%m-%d) \
  --granularity DAILY \
  --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["Amazon Bedrock"]}}' \
  --metrics BlendedCost UnblendedCost

# Get shadow-specific cost from internal metrics
redis-cli -h mangaassist-redis.xxxxx.apne1.cache.amazonaws.com \
  GET "shadow:total_shadow_cost"

# Add to ShadowRouter.__init__():
class ShadowRouter:
    MAX_SHADOW_TRAFFIC_PCT = 25.0  # Hard cap
    MAX_SHADOW_DAILY_COST_USD = 500.0  # Daily cost circuit breaker

    def __init__(self, config: ShadowConfig, ...):
        # Validate shadow traffic percentage
        if config.shadow_traffic_pct > self.MAX_SHADOW_TRAFFIC_PCT:
            logger.error(
                "Shadow traffic %.1f%% exceeds maximum %.1f%% — capping",
                config.shadow_traffic_pct,
                self.MAX_SHADOW_TRAFFIC_PCT,
            )
            config.shadow_traffic_pct = self.MAX_SHADOW_TRAFFIC_PCT

        # Estimate daily cost and warn
        estimated_daily_cost = self._estimate_daily_shadow_cost(config)
        if estimated_daily_cost > self.MAX_SHADOW_DAILY_COST_USD:
            logger.warning(
                "Shadow routing estimated cost: $%.2f/day (limit: $%.2f) — DISABLING",
                estimated_daily_cost,
                self.MAX_SHADOW_DAILY_COST_USD,
            )
            config.enabled = False

    def _estimate_daily_shadow_cost(self, config: ShadowConfig) -> float:
        """Estimate daily shadow cost based on traffic and model pricing."""
        daily_queries = 1_000_000  # MangaAssist scale
        shadow_queries = daily_queries * (config.shadow_traffic_pct / 100)
        avg_input_tokens = 500
        avg_output_tokens = 400
        costs = self.COSTS.get(config.shadow_model_id, {"input": 0, "output": 0})
        return (
            shadow_queries
            * (avg_input_tokens * costs["input"] + avg_output_tokens * costs["output"])
            / 1_000_000
        )

# Re-enable with validated parameters
aws dynamodb update-item \
  --table-name MangaAssist-FeatureFlags \
  --key '{"PK": {"S": "FLAG#shadow_routing"}, "SK": {"S": "CONFIG"}}' \
  --update-expression "SET shadow_enabled = :enabled, shadow_traffic_pct = :pct, shadow_model_id = :model, updated_by = :who" \
  --expression-attribute-values '{
    ":enabled": {"BOOL": true},
    ":pct": {"N": "10"},
    ":model": {"S": "anthropic.claude-3-sonnet-20240229-v1:0"},
    ":who": {"S": "data-science-lead"}
  }' \
  --region ap-northeast-1

flowchart TB
    subgraph Preventive["Preventive Controls"]
        VAL[Input Validation<br/>Thresholds, percentages, model IDs]
        AUTO[Deployment Automation<br/>Routing table in same pipeline]
        CAP[Capacity Planning<br/>Kinesis on-demand mode]
        CI[CI/CD Validation<br/>VTL parsing, integration tests]
    end

    subgraph Detective["Detective Controls"]
        COST_MON[Cost Monitoring<br/>Budget alerts at 130%, 150%, 200%]
        METRIC_FRESH[Metric Freshness<br/>Staleness > 2min alarm]
        HEADER_MON[Header Presence<br/>Missing header rate alarm]
        TRAFFIC_MON[Traffic Distribution<br/>Model split deviation alarm]
    end

    subgraph Corrective["Corrective Controls"]
        CIRCUIT[Cost Circuit Breaker<br/>Auto-disable at threshold]
        FALLBACK[Stale Metric Fallback<br/>Conservative routing]
        ROLLBACK[Auto-Rollback<br/>Deploy fails → revert]
        OVERRIDE[Emergency Override<br/>Force static routing]
    end

    VAL -->|Prevents| S1[Scenario 1 + 5]
    AUTO -->|Prevents| S2[Scenario 2]
    CAP -->|Prevents| S3[Scenario 3]
    CI -->|Prevents| S4[Scenario 4]

    COST_MON -->|Detects| S15[Scenario 1 + 5]
    METRIC_FRESH -->|Detects| S3B[Scenario 3]
    HEADER_MON -->|Detects| S4B[Scenario 4]
    TRAFFIC_MON -->|Detects| S12[Scenario 1 + 2]

    CIRCUIT -->|Corrects| S15C[Scenario 1 + 5]
    FALLBACK -->|Corrects| S3C[Scenario 3]
    ROLLBACK -->|Corrects| S4C[Scenario 4]
    OVERRIDE -->|Corrects| SALL[All Scenarios]

    style VAL fill:#4CAF50,color:#fff
    style AUTO fill:#4CAF50,color:#fff
    style CAP fill:#4CAF50,color:#fff
    style CI fill:#4CAF50,color:#fff
    style COST_MON fill:#2196F3,color:#fff
    style METRIC_FRESH fill:#2196F3,color:#fff
    style HEADER_MON fill:#2196F3,color:#fff
    style TRAFFIC_MON fill:#2196F3,color:#fff
    style CIRCUIT fill:#FF9800,color:#fff
    style FALLBACK fill:#FF9800,color:#fff
    style ROLLBACK fill:#FF9800,color:#fff
    style OVERRIDE fill:#FF9800,color:#fff