08: Best Practices and Patterns

graph TD
    A[MangaAssist Metrics] --> B[Content Handling<br>5.2.1]
    A --> C[FM Integration<br>5.2.2]
    A --> D[Prompt Quality<br>5.2.3]
    A --> E[Retrieval Health<br>5.2.4]
    A --> F[Prompt Ops<br>5.2.5]

    B --> B1[BudgetUtilization %]
    B --> B2[TruncationCount]
    B --> B3[CompressionRatio]

    C --> C1[InvocationLatency P95]
    C --> C2[ErrorRate %]
    C --> C3[ThrottleCount]
    C --> C4[CircuitBreakerState]

    D --> D1[GoldenTestPassRate %]
    D --> D2[QualityScoreMean]
    D --> D3[HallucinationRate %]

    E --> E1[EmbeddingDriftP95]
    E --> E2[RetrievalMRR]
    E --> E3[DocumentStalenessHrs]

    F --> F1[TemplateHealthScore]
    F --> F2[SchemaViolationRate %]
    F --> F3[ConfusionPairCount]

┌──────────────────────────────────────────────────────────┐
│  Row 1: FM Health (Skill 5.2.2)                         │
│  [Error Rate %]  [Latency P50/P95/P99]  [Circuit State] │
├──────────────────────────────────────────────────────────┤
│  Row 2: Content & Retrieval (Skills 5.2.1 + 5.2.4)     │
│  [Budget Util %]  [Embedding Drift]  [Doc Staleness]    │
├──────────────────────────────────────────────────────────┤
│  Row 3: Prompt Quality (Skills 5.2.3 + 5.2.5)          │
│  [Quality Score]  [Health Score]  [Schema Violations]    │
├──────────────────────────────────────────────────────────┤
│  Row 4: Cost Tracking                                    │
│  [Token Usage]  [Bedrock Cost]  [Cache Hit Rate]         │
└──────────────────────────────────────────────────────────┘

flowchart TD
    A[User reports<br>bad output] --> B{Output<br>empty?}
    B -->|Yes| C{Bedrock<br>error in logs?}
    C -->|Yes| D[Skill 5.2.2:<br>FM Integration]
    C -->|No| E{Budget > 95%?}
    E -->|Yes| F[Skill 5.2.1:<br>Content Overflow]
    E -->|No| G[Check template<br>rendering logs]

    B -->|No| H{Output<br>incorrect?}
    H -->|Yes| I{RAG context<br>relevant?}
    I -->|No| J[Skill 5.2.4:<br>Retrieval Issue]
    I -->|Yes| K{Prompt version<br>changed recently?}
    K -->|Yes| L[Skill 5.2.3/5.2.5:<br>Prompt Regression]
    K -->|No| M[Skill 5.2.3:<br>Prompt Quality]

    H -->|No| N{Output<br>slow?}
    N -->|Yes| O[Check X-Ray spans<br>for bottleneck]
    O --> P{Bedrock span<br>slow?}
    P -->|Yes| D
    P -->|No| Q[Check retrieval<br>or preprocessing]

incident_type: prompt_quality_regression
severity: P2
detection:
  signal: TemplateHealthScore dropped below 0.7
  alert: CloudWatch Alarm "MangaAssist-PromptHealth-Critical"
  time_to_detect: ~15 minutes (SPC anomaly detection)

immediate_actions:
  - Check last deployment: "Was a prompt template deployed in the last 2 hours?"
  - Check PromptObservabilityPipeline logs: filter by template_name and version
  - Compare quality_score between current and previous version
  - If confirmed regression: rollback template version via config update

investigation:
  - Pull golden test results for the new version
  - Run PromptTestRunner.compare_versions(old, new)
  - Identify which test cases regressed
  - Check if model version also changed (compound root cause)

resolution:
  - Rollback prompt template
  - Add failing cases to golden test suite
  - Update CI gate to prevent similar regression
  - Post-incident: update prompt change process checklist

prevention:
  - Mandatory golden test pass before prompt deploy
  - Canary deployment: 5% traffic for 1 hour before full rollout
  - PromptHealthChecker SPC window reduced from 24h to 6h for tighter detection

┌─────────────────────────────────────────────┐
│ Layer 5: Business Rules                     │
│ • Compliance checks, content policy         │
│ • Guardrails (Bedrock Guardrails)           │
├─────────────────────────────────────────────┤
│ Layer 4: Observability                      │
│ • X-Ray traces, CloudWatch metrics          │
│ • Structured logging, anomaly detection     │
├─────────────────────────────────────────────┤
│ Layer 3: Validation                         │
│ • Schema validation on responses            │
│ • Token budget pre-flight checks            │
│ • Hallucination detection                   │
├─────────────────────────────────────────────┤
│ Layer 2: Resilience                         │
│ • Circuit breakers, retries with backoff    │
│ • Model tiering (fallback Sonnet → Haiku)   │
│ • Response caching (ElastiCache)            │
├─────────────────────────────────────────────┤
│ Layer 1: Infrastructure                     │
│ • ECS auto-scaling, Bedrock provisioned     │
│ • OpenSearch replica shards                 │
│ • DynamoDB on-demand capacity               │
└─────────────────────────────────────────────┘

flowchart LR
    A[Issue Reported] --> B[Cheap: Check logs<br>$0 — CloudWatch]
    B --> C[Cheap: Check metrics<br>$0 — CloudWatch]
    C --> D[Medium: Replay<br>with Haiku<br>~$0.001/test]
    D --> E[Expensive: Replay<br>with Sonnet<br>~$0.01/test]
    E --> F[Most Expensive:<br>Full regression<br>suite ~$2-5]

TRIGGER: InvocationLatency P95 > 5s for 5 minutes
SEVERITY: P2

STEP 1: Check Bedrock service health
  → AWS Health Dashboard → Bedrock → ap-northeast-1
  → If service degradation: nothing to do, wait for AWS resolution

STEP 2: Check throttling
  → CloudWatch: MangaAssist/BedrockHealth → ThrottleCount
  → If high: check Bedrock quota usage (Service Quotas console)
  → If near quota: request increase or enable model tiering fallback

STEP 3: Check circuit breaker state
  → CloudWatch: MangaAssist/BedrockHealth → CircuitBreakerState
  → If OPEN: system is self-protecting, check underlying cause
  → If CLOSED but slow: issue is latency, not errors

STEP 4: Check input size
  → Logs Insights:
    fields @timestamp, input_tokens, latency_ms
    | filter log_type = "bedrock_call"
    | stats avg(input_tokens), avg(latency_ms) by bin(5m)
  → If input_tokens increased: check TokenBudgetManager logs

STEP 5: Mitigate
  → If traffic spike: no action if auto-scaling handles it
  → If input bloat: enable aggressive history compression
  → If Bedrock issue: switch to Haiku fallback via feature flag

TRIGGER: TemplateHealthScore < 0.7 for 1 hour
SEVERITY: P2

STEP 1: Identify affected template
  → CloudWatch: MangaAssist/PromptObservability → filter by Template dimension
  → Note template_name and template_version

STEP 2: Check for recent deployments
  → git log --since="2h ago" -- prompts/
  → If template was changed: this is the likely root cause

STEP 3: Compare versions
  → Run PromptTestRunner.compare_versions(prev_version, current_version)
  → Identify which quality dimensions regressed

STEP 4: Rollback if confirmed
  → Update prompt template config to previous version
  → Monitor TemplateHealthScore recovery (expect 15-30 min)

STEP 5: Root cause analysis
  → Review the prompt change: what was the intent?
  → Add failing cases to golden test suite
  → Update CI pipeline if test gap identified