Optimization Tradeoffs User Stories - MangaAssist Chatbot

The Core Problem: You Cannot Optimize Everything Simultaneously

MangaAssist operates under three competing optimization pressures. Improving any two necessarily degrades the third. Every engineering decision in this system is a negotiation between these forces.

graph TD
    subgraph "The Optimization Trilemma"
        COST["💰 Cost Optimization<br/>Reduce spend on compute,<br/>LLM tokens, storage, bandwidth"]
        PERF["⚡ Performance Optimization<br/>Minimize latency, maximize<br/>throughput, improve availability"]
        INF["🧠 Inference Optimization<br/>Maximize response quality,<br/>accuracy, grounding, safety"]
    end

    COST <-->|"Tension"| PERF
    PERF <-->|"Tension"| INF
    INF <-->|"Tension"| COST

    style COST fill:#f9d71c,stroke:#333,color:#000
    style PERF fill:#4ecdc4,stroke:#333,color:#000
    style INF fill:#ff6b6b,stroke:#333,color:#000

Why These Three Cannot Be Optimized Together

The Three Teams (Perspectives)

Each user story is written from the perspective of three competing stakeholders:

User Stories

How to Read These Stories

graph LR
    A["US-01<br/>Framework"] --> B["US-02 to US-09<br/>Dimension-Specific<br/>Tradeoffs"]
    B --> C["US-10<br/>Unified Dashboard"]

    style A fill:#ff9f43,stroke:#333,color:#000
    style B fill:#54a0ff,stroke:#333,color:#000
    style C fill:#5f27cd,stroke:#333,color:#fff

1. Start with US-01 to understand the decision framework and the trilemma model.
2. Read US-02 through US-09 for specific tradeoff dimensions — each one is self-contained.
3. End with US-10 to see how all tradeoffs roll up into a unified monitoring and decision system.

How to Read the Evolution

These documents now preserve both the earlier architecture guidance and the current-state refresh:

• The original sections in each file describe the baseline or previous architecture and the reasoning that led to those earlier tradeoffs.
• The 2026 Update section in each file shows the current architecture direction and explains what should change now based on newer platform capabilities and techniques.
• Read each document top to bottom if you want the transition path from previous architecture to current architecture.
• Treat the newer sections as an evolution layer, not a replacement of the original material. This keeps the migration path and historical decision context visible.

2026 Refresh Highlights

These stories were refreshed on March 25, 2026 against current AWS Bedrock, Anthropic, vLLM, KServe, and recent research on routing and adaptive RAG. The core tradeoff model still holds, and the original content has been intentionally preserved as the baseline architecture, but the frontier has moved in a few important ways:

• Routing has matured from static tiering to utility-aware routing and cascades. Teams should benchmark current model classes regularly and route by expected quality lift, latency impact, and uncertainty rather than hard-coding a single "cheap" and "smart" model forever.
• RAG depth is no longer just "pick a fixed top-K." Hybrid retrieval, contextual retrieval, adaptive retrieval, and corrective retrieval now outperform static chunk-count rules in many production systems.
• Caching is now multi-layer. The important distinction is not only Redis TTLs, but also prompt caching, prefix/KV caching, deterministic tool-result caches, and tightly scoped semantic caches.
• Latency work should separate TTFT from the rest of completion time. Managed stacks can use Bedrock prompt caching, latency-optimized inference, and cross-Region inference; self-hosted stacks can tune chunked prefill, disaggregated prefill, and request-queue-aware autoscaling.
• Guardrails are becoming layered control planes, not just blocklists. Deterministic validation, structured outputs, detect-and-repair flows, and policy validation such as automated reasoning checks are now more practical than "block everything synchronously."
• Dashboards need trace-level observability. Per-intent cost, TTFT, queue time, prefill/decode split, cache hit rates, intervention types, and experiment annotations matter more than a single blended average.

Representative references: AWS Bedrock prompt caching, AWS Bedrock cross-Region inference, AWS Bedrock latency-optimized inference, Anthropic contextual retrieval, vLLM optimization and tuning, KServe autoscaling with LLM metrics.

Relationship to Other Documents