Troubleshoot GenAI Applications — Task 5.2
AWS AIP-C01 Exam Domain Coverage
This folder covers Task Statement 5.2: Troubleshoot GenAI applications from the AWS Certified AI Practitioner (AIP-C01) exam guide. Every file is grounded in the MangaAssist chatbot system — a production-grade, microservices-based, event-driven conversational assistant for Amazon's JP Manga store.
Skill-to-File Mapping
| AIP-C01 Skill | Description | Primary File | Supporting Files |
|---|---|---|---|
| 5.2.1 | Resolve content handling issues | 01-content-handling-troubleshooting.md | 06, 07 (POC 1), 09 (Scenarios 1, 9) |
| 5.2.2 | Diagnose and resolve FM integration issues | 02-fm-integration-troubleshooting.md | 06, 07 (POC 2), 09 (Scenarios 2, 7) |
| 5.2.3 | Troubleshoot prompt engineering problems | 03-prompt-engineering-troubleshooting.md | 06, 07 (POC 3), 09 (Scenarios 3, 8) |
| 5.2.4 | Troubleshoot retrieval system issues | 04-retrieval-system-troubleshooting.md | 06, 07 (POC 4), 09 (Scenarios 4, 6, 10) |
| 5.2.5 | Troubleshoot prompt maintenance issues | 05-prompt-maintenance-troubleshooting.md | 06, 07 (POC 5), 09 (Scenario 5) |
File Index
Core Skill Files (01–05)
Each file follows a consistent structure: User Story → Acceptance Criteria → High-Level Design (with Mermaid diagrams) → Low-Level Design (with production code) → MangaAssist Scenarios → Intuition Gained.
| # | File | What You Learn |
|---|---|---|
| 01 | Content Handling Troubleshooting | Context window overflow diagnostics, dynamic chunking, prompt design optimization, truncation error analysis |
| 02 | FM Integration Troubleshooting | Error logging, request validation, response analysis, Bedrock error resolution, circuit breakers |
| 03 | Prompt Engineering Troubleshooting | Prompt testing frameworks, version comparison, systematic refinement, regression detection |
| 04 | Retrieval System Troubleshooting | Relevance analysis, embedding diagnostics, drift monitoring, vector search optimization |
| 05 | Prompt Maintenance Troubleshooting | CloudWatch/X-Ray observability, schema validation, prompt confusion diagnosis, refinement workflows |
Cross-Cutting Files (06–11)
| # | File | What You Learn |
|---|---|---|
| 06 | Technical Decisions and Tradeoffs | Decision matrices, risk/mitigation tables, cost-complexity analysis across all 5 skills |
| 07 | POC Implementations | 5 working proof-of-concept systems with architecture diagrams and deployable code |
| 08 | Best Practices and Patterns | Anti-patterns, operational runbooks, monitoring templates, incident response patterns |
| 09 | Debugging Scenarios and Runbooks | 10 production-grade debugging scenarios with root cause analysis and prevention |
| 10 | Interview Q&A | 30 interview questions mapped to skills 5.2.1–5.2.5, behavioral + technical mix |
| 11 | Intuition and Strategic Direction | Meta-learning synthesis, decision frameworks, career growth signals, future direction |
How to Use This Material
For Exam Prep (AIP-C01)
- Start with files 01–05 to understand each skill area
- Use file 10 (Interview Q&A) to test your knowledge
- Review file 06 (Tradeoffs) to understand decision reasoning
For Production Work
- Start with file 08 (Best Practices) for quick operational patterns
- Use file 09 (Debugging Scenarios) when troubleshooting real incidents
- Reference files 01–05 for deep dives into specific problem areas
- Deploy POCs from file 07 to build monitoring and testing infrastructure
For Career Growth
- Read file 11 (Intuition and Strategic Direction) to understand the meta-skills
- Use the career growth signals to assess your current level
- Apply the decision frameworks when designing new GenAI systems
For Interview Preparation
- Study files 01–05 for technical depth
- Practice with file 10 (Interview Q&A) — answers reference MangaAssist scenarios
- Use file 09 (Debugging Scenarios) for behavioral "tell me about a time" questions
- Review file 11 (Intuition) for strategic thinking questions
Relationship to Other Folders
This folder focuses on troubleshooting workflows and does not duplicate foundational design content in other folders:
| Folder | Relationship |
|---|---|
Debugging/ |
Covers general application and Bedrock logging. This folder builds on that with GenAI-specific troubleshooting |
Prompt-Engineering/ |
Covers prompt design, patterns, and hardening. This folder covers what to do when prompts break |
Security-Privacy-Guardrails/ |
Covers security design. This folder covers diagnosing guardrail-related failures |
Model-Inference/ |
Covers inference pipeline design. This folder covers troubleshooting inference failures |
LLMOps/ |
Covers MLflow and deployment pipelines. This folder covers operational troubleshooting |
Fine-Tuning-Foundational-Models/ |
Covers model training. This folder covers post-deployment troubleshooting |
MangaAssist System Context
All scenarios reference the MangaAssist chatbot architecture:
- Orchestrator (ECS Fargate) → Intent Classifier (SageMaker) → RAG Pipeline (OpenSearch Serverless + Titan Embeddings) → LLM (Bedrock Claude 3.5 Sonnet) → Guardrails → Response
- Conversation Memory: DynamoDB with TTL and summarization
- Caching: ElastiCache Redis for product data, recommendations, promotions
- Observability: CloudWatch, X-Ray, Kinesis → Redshift
- Human Handoff: Amazon Connect
See 04-architecture-hld.md and 04b-architecture-lld.md for full architecture details.