Detailed Technology Stack - MangaAssist

Complete inventory of every technology choice across the stack, with rationale, alternatives considered, and why each was selected.

How to Use This Document

• Read this file top-to-bottom if you want the full system stack in one place.
• Jump to 02-open-source-libraries.md for deeper discussion of the OSS runtime choices behind inference and serving.
• Jump to 03-mlflow-llm-observability.md or 04-innovation-and-tradeoffs.md if you want tooling and decision-making depth rather than layer-by-layer inventory.

Stack Overview

graph TB
    subgraph Client["Client Layer"]
        A1[React Chat Widget]
        A2[WebSocket / HTTPS]
        A3[CloudFront CDN]
    end

    subgraph Gateway["Edge & Gateway"]
        B1[Amazon API Gateway]
        B2[AWS WAF]
        B3[Amazon Cognito]
    end

    subgraph Compute["Compute & Orchestration"]
        C1[ECS Fargate - baseline]
        C2[AWS Lambda - burst]
        C3[Step Functions - workflows]
    end

    subgraph Intelligence["AI & ML Layer"]
        D1[Amazon Bedrock - LLM]
        D2[SageMaker - custom models]
        D3[vLLM - self-hosted inference]
        D4[OpenSearch - vector store]
    end

    subgraph Data["Data Layer"]
        E1[DynamoDB - conversations]
        E2[ElastiCache Redis - caching]
        E3[Redshift - analytics]
        E4[Kinesis - streaming]
    end

    subgraph Observability["Observability"]
        F1[MLflow Tracing]
        F2[CloudWatch / X-Ray]
        F3[OpenTelemetry]
        F4[Prometheus / Grafana]
    end

    Client --> Gateway --> Compute --> Intelligence
    Intelligence --> Data
    Compute --> Observability
    Intelligence --> Observability

Layer-by-Layer Breakdown

1. Frontend & Client

Why not alternatives? - Vue/Angular: Amazon's frontend ecosystem is React-native; switching adds integration burden with zero benefit - Server-Sent Events (SSE): WebSocket chosen because we need bidirectional communication (user typing indicators, real-time context updates)

2. API Gateway & Edge Security

Why not alternatives? - Kong/Nginx: API Gateway is fully managed; operational burden of self-hosting not justified for our scale - Auth0/Okta: Cognito integrates natively with all AWS services; external auth adds network hop + vendor dependency

3. Compute & Orchestration

Scaling Model:

Normal:  ECS Fargate (10-50 tasks, predictable cost)
Spike:   Lambda (0 to 10,000 concurrent in seconds)
Peak:    ECS + Lambda hybrid (cost-optimized: Fargate for base, Lambda for overflow)

Why not alternatives? - EKS (Kubernetes): Overkill for our service count; Fargate gives us container benefits without cluster management - EC2: Manual scaling and patching; Fargate eliminates this entirely - Temporal/Airflow: Step Functions is native; simpler for AWS-only workflows

4. AI & LLM Layer

This is the most critical layer and where the majority of technology innovation happened.

Why not alternatives? - GPT-4: Higher latency, no native AWS integration, data residency concerns for Amazon data - Open-source LLMs (Llama, Mistral): Evaluated but Claude 3.5 Sonnet beat them on Japanese language quality; we do use vLLM for self-hosted fine-tuned models - Pinecone/Weaviate: OpenSearch is already in Amazon's ecosystem with zero egress costs - FAISS: No serverless option; requires managing infrastructure

5. Data Layer

Caching Strategy (3 layers):

L1: In-memory (application-level, per-container)     -> <1ms, small capacity
L2: ElastiCache Redis (shared across containers)      -> 1-5ms, medium capacity
L3: DynamoDB DAX (conversation-specific acceleration)  -> 1-3ms, large capacity

Why not alternatives? - PostgreSQL/Aurora: DynamoDB's single-digit ms latency at any scale beats RDS for key-value access patterns - Memcached: Redis supports richer data structures (sorted sets for ranking, pub/sub for real-time updates) - Snowflake: Redshift Serverless is cheaper within AWS ecosystem; no data egress

6. Observability & Monitoring

Why MLflow over alternatives? - Langfuse/LangSmith: MLflow is fully open-source, self-hosted (no data leaves AWS), and integrates with our existing MLflow experiment tracking - Datadog LLM Observability: Costly at our scale; vendor lock-in; MLflow gives us the same capabilities at zero license cost - Detailed comparison in 03-mlflow-llm-observability.md

7. Security & Compliance

8. Infrastructure & DevOps

Cost Summary (100K conversations/day)

Cost optimizations that I drove (see 04-innovation-and-tradeoffs.md): - Inferentia migration for classifiers: -$8,400/month - Semantic caching of LLM responses: -$12,000/month - vLLM for self-hosted models: -$15,000/month (50% GPU reduction) - Intelligent routing (Haiku vs Sonnet): -$18,000/month - Prompt compression and optimization: -$6,000/month - Total monthly savings: ~$59,400/month (~$713K/year)

Technology Decision Framework

For every component, I applied this evaluation matrix:

This framework is detailed further in 04-innovation-and-tradeoffs.md.