Scenarios and Runbooks — Intelligent Tool Integrations

SCENARIO N: <Title>
├── Severity: P1/P2/P3/P4
├── Blast Radius: <what is affected>
├── Symptom: <what operators/users observe>
├── Root Cause: <why it happened>
├── Timeline: <detection to resolution>
├── Runbook: <step-by-step fix>
├── Prevention: <how to stop it recurring>
├── Code Fix: <before/after code>
└── Exam Relevance: <AIP-C01 mapping>

[2026-03-28 14:23:17 UTC] WARN  mangaassist.tools — product_search returned
0 results for query="進撃の巨人" with genre="action" (invalid enum value)

[2026-03-28 14:23:17 UTC] INFO  mangaassist.orchestrator — FM selected
product_search with params: {"query": "進撃の巨人", "genre": "action",
"max_results": "10", "in_stock_only": "yes"}

# Check zero-result rate in CloudWatch
aws cloudwatch get-metric-statistics \
  --namespace "MangaAssist/Tools" \
  --metric-name "ZeroResultCount" \
  --dimensions Name=ToolName,Value=product_search \
  --start-time "2026-03-28T14:00:00Z" \
  --end-time "2026-03-28T15:00:00Z" \
  --period 300 \
  --statistics Sum

# Search CloudWatch Logs for recent product_search invocations
aws logs filter-log-events \
  --log-group-name "/aws/lambda/mangaassist-tool-gateway" \
  --filter-pattern '{ $.tool_name = "product_search" && $.result_count = 0 }' \
  --start-time 1711630800000 \
  --limit 50

# List current layer version
aws lambda get-function-configuration \
  --function-name mangaassist-tool-gateway \
  --query 'Layers'

# Download and inspect layer contents
aws lambda get-layer-version \
  --layer-name mangaassist-tools-layer \
  --version-number 14 \
  --query 'Content.Location' --output text | xargs curl -o layer.zip

unzip -l layer.zip | grep parameter_validator

# Rollback to previous layer version
aws lambda update-function-configuration \
  --function-name mangaassist-tool-gateway \
  --layers "arn:aws:lambda:ap-northeast-1:123456789012:layer:mangaassist-tools-layer:13"

# Monitor zero-result rate after rollback
watch -n 30 "aws cloudwatch get-metric-statistics \
  --namespace 'MangaAssist/Tools' \
  --metric-name 'ZeroResultCount' \
  --dimensions Name=ToolName,Value=product_search \
  --start-time '$(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%SZ)' \
  --end-time '$(date -u +%Y-%m-%dT%H:%M:%SZ)' \
  --period 60 --statistics Sum"

# BEFORE — vague description, FM hallucinates enum values
"genre": {
    "type": "string",
    "description": "Filter results by manga genre.",
    "enum": ["shonen", "shojo", "seinen", "josei", "kodomomuke",
             "isekai", "mecha", "slice_of_life", "horror", "sports"]
}

# AFTER — explicit description reinforces valid values
"genre": {
    "type": "string",
    "description": (
        "Filter results by manga genre. MUST be one of these exact values: "
        "shonen (action/adventure like Naruto, One Piece), "
        "shojo (romance like Fruits Basket), "
        "seinen (mature like Berserk, Vagabond), "
        "josei (women's like Nana), "
        "kodomomuke (children's like Doraemon), "
        "isekai (transported to another world), "
        "mecha (giant robots like Gundam), "
        "slice_of_life (daily life stories), "
        "horror (scary/dark themes like Junji Ito), "
        "sports (athletic competition like Haikyuu). "
        "If the customer's request does not match any genre exactly, "
        "omit this parameter and search by query text only."
    ),
    "enum": ["shonen", "shojo", "seinen", "josei", "kodomomuke",
             "isekai", "mecha", "slice_of_life", "horror", "sports"]
}

[2026-03-29 09:14:22 UTC] ERROR mangaassist.chains — Chain
'search_and_check_stock' step 1 (inventory_check) timed out after 1000ms

[2026-03-29 09:14:22 UTC] WARN  mangaassist.chains — Returning partial
result: product_search succeeded, inventory_check failed

[2026-03-29 09:14:23 UTC] INFO  mangaassist.orchestrator — FM received
partial chain result, generating response without inventory data

# Check throttled read requests
aws cloudwatch get-metric-statistics \
  --namespace "AWS/DynamoDB" \
  --metric-name "ReadThrottleEvents" \
  --dimensions Name=TableName,Value=MangaAssist-Inventory \
  --start-time "2026-03-29T09:00:00Z" \
  --end-time "2026-03-29T10:00:00Z" \
  --period 60 \
  --statistics Sum

# Check consumed vs provisioned capacity
aws cloudwatch get-metric-statistics \
  --namespace "AWS/DynamoDB" \
  --metric-name "ConsumedReadCapacityUnits" \
  --dimensions Name=TableName,Value=MangaAssist-Inventory \
  --start-time "2026-03-29T09:00:00Z" \
  --end-time "2026-03-29T10:00:00Z" \
  --period 60 \
  --statistics Sum

# Check if the batch import job is running
aws ecs list-tasks \
  --cluster mangaassist-cluster \
  --family mangaassist-inventory-import \
  --desired-status RUNNING

# Check batch job CloudWatch logs for throughput
aws logs filter-log-events \
  --log-group-name "/ecs/mangaassist-inventory-import" \
  --filter-pattern "items_processed" \
  --start-time 1711699200000 \
  --limit 10

# Update the batch job's rate limiter via SSM Parameter Store
aws ssm put-parameter \
  --name "/mangaassist/inventory-import/max-writes-per-sec" \
  --value "50" \
  --type String \
  --overwrite

# The batch job reads this parameter and throttles itself

# BEFORE — inventory_check is required, chain fails completely
search_and_check_stock = ToolChain(
    name="search_and_check_stock",
    steps=[
        ChainStep(
            tool_name="product_search",
            param_mapper=lambda ctx: {"query": ctx["query"], "max_results": 1},
            timeout_ms=1500,
        ),
        ChainStep(
            tool_name="inventory_check",
            param_mapper=lambda ctx: {
                "product_id": ctx.get("results", [{}])[0].get("product_id", ""),
            },
            required=True,  # <-- BUG: chain fails if inventory_check fails
            timeout_ms=1000,
        ),
    ],
)

# AFTER — inventory_check is optional with a fallback value
search_and_check_stock = ToolChain(
    name="search_and_check_stock",
    steps=[
        ChainStep(
            tool_name="product_search",
            param_mapper=lambda ctx: {"query": ctx["query"], "max_results": 1},
            timeout_ms=1500,
        ),
        ChainStep(
            tool_name="inventory_check",
            param_mapper=lambda ctx: {
                "product_id": ctx.get("results", [{}])[0].get("product_id", ""),
            },
            required=False,  # <-- FIX: chain continues with partial results
            fallback_value={
                "status": "unknown",
                "message": "Inventory status is being updated. "
                "Please check the product page for current availability.",
            },
            timeout_ms=1000,
        ),
    ],
)

aws dynamodb update-table \
  --table-name MangaAssist-Inventory \
  --billing-mode PAY_PER_REQUEST

# Monitor timeout rate
aws cloudwatch get-metric-statistics \
  --namespace "MangaAssist/Tools" \
  --metric-name "Error_timeout" \
  --dimensions Name=ToolName,Value=inventory_check \
  --start-time "$(date -u -d '15 minutes ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --period 60 --statistics Sum

[2026-03-30 11:42:05 UTC] INFO  mangaassist.orchestrator — FM response
contains tool_use: recommendation_engine (attempt 1)
[2026-03-30 11:42:06 UTC] INFO  mangaassist.orchestrator — Tool result
returned to FM
[2026-03-30 11:42:07 UTC] INFO  mangaassist.orchestrator — FM response
contains tool_use: recommendation_engine (attempt 2, same params)
[2026-03-30 11:42:08 UTC] INFO  mangaassist.orchestrator — Tool result
returned to FM
[2026-03-30 11:42:09 UTC] INFO  mangaassist.orchestrator — FM response
contains tool_use: recommendation_engine (attempt 3, same params)
[2026-03-30 11:42:09 UTC] WARN  mangaassist.orchestrator — Tool loop
detected: recommendation_engine called 3 times with identical params

# v1 response (what the FM expects based on training/system prompt)
{
    "recommendations": [
        {"title": "Naruto", "author": "Kishimoto", "score": 0.95},
        {"title": "Bleach", "author": "Kubo", "score": 0.88},
    ],
    "count": 2,
    "strategy": "collaborative"
}

# v2 response (what the tool now returns after the service update)
{
    "data": {
        "items": [
            {"product_name": "Naruto", "creator": "Kishimoto", "relevance": 95},
            {"product_name": "Bleach", "creator": "Kubo", "relevance": 88},
        ]
    },
    "metadata": {
        "total": 2,
        "algorithm": "collaborative_v2"
    }
}

# Find sessions with excessive tool calls
aws logs filter-log-events \
  --log-group-name "/ecs/mangaassist-orchestrator" \
  --filter-pattern '{ $.event = "tool_loop_detected" }' \
  --start-time 1711792800000 \
  --limit 20

# Get a sample tool result from the logs
aws logs filter-log-events \
  --log-group-name "/aws/lambda/mangaassist-tool-gateway" \
  --filter-pattern '{ $.tool_name = "recommendation_engine" && $.status = "success" }' \
  --start-time 1711792800000 \
  --limit 5

# Compare the schema of the returned result against the expected schema
# documented in the tool's system prompt or tool description

"""
response_adapter.py — Transforms tool responses to match expected schemas.

This adapter sits between the tool result and the FM, ensuring the
response schema matches what the FM was trained/prompted to expect.
"""

from typing import Any, Dict


def adapt_recommendation_response(raw: Dict[str, Any]) -> Dict[str, Any]:
    """Transform recommendation_engine v2 response to v1 schema."""

    # Detect v2 format
    if "data" in raw and "items" in raw.get("data", {}):
        items = raw["data"]["items"]
        metadata = raw.get("metadata", {})

        adapted = {
            "recommendations": [
                {
                    "title": item.get("product_name", item.get("title", "")),
                    "author": item.get("creator", item.get("author", "")),
                    "score": item.get("relevance", 0) / 100.0
                    if item.get("relevance", 0) > 1
                    else item.get("relevance", 0),
                }
                for item in items
            ],
            "count": metadata.get("total", len(items)),
            "strategy": metadata.get("algorithm", "unknown").replace("_v2", ""),
        }
        return adapted

    # Already v1 format or unknown — return as-is
    return raw


# Register adapters for all tools that might change schemas
RESPONSE_ADAPTERS = {
    "recommendation_engine": adapt_recommendation_response,
    # Add others as needed
}


def adapt_tool_response(
    tool_name: str, raw_response: Dict[str, Any]
) -> Dict[str, Any]:
    """Apply the appropriate adapter for a tool's response."""
    adapter = RESPONSE_ADAPTERS.get(tool_name)
    if adapter:
        return adapter(raw_response)
    return raw_response

"""
loop_detector.py — Detects and breaks FM tool-call loops.
"""

from collections import defaultdict
from typing import Any, Dict, List, Optional
import hashlib
import json


class ToolLoopDetector:
    """
    Detects when the FM repeatedly calls the same tool with identical params.

    Detection rules:
    - Same tool + same params called 3+ times -> loop detected
    - Any tool called 5+ times in one turn -> excessive tool use
    - Total tool calls exceed 10 in one turn -> force stop
    """

    def __init__(
        self,
        same_call_limit: int = 3,
        per_tool_limit: int = 5,
        total_limit: int = 10,
    ) -> None:
        self.same_call_limit = same_call_limit
        self.per_tool_limit = per_tool_limit
        self.total_limit = total_limit
        self._call_hashes: List[str] = []
        self._tool_counts: Dict[str, int] = defaultdict(int)

    def check(self, tool_name: str, params: Dict[str, Any]) -> Optional[str]:
        """
        Check if a tool call should be blocked.

        Returns None if OK, or a reason string if the call should be blocked.
        """
        # Track total calls
        self._tool_counts[tool_name] += 1
        total = sum(self._tool_counts.values())

        # Hash the call for duplicate detection
        call_hash = hashlib.md5(
            json.dumps({"tool": tool_name, "params": params}, sort_keys=True).encode()
        ).hexdigest()
        self._call_hashes.append(call_hash)

        # Check: same exact call repeated
        identical_count = self._call_hashes.count(call_hash)
        if identical_count >= self.same_call_limit:
            return (
                f"Loop detected: {tool_name} called {identical_count} times "
                f"with identical parameters"
            )

        # Check: per-tool limit
        if self._tool_counts[tool_name] >= self.per_tool_limit:
            return (
                f"Excessive use: {tool_name} called "
                f"{self._tool_counts[tool_name]} times in this turn"
            )

        # Check: total limit
        if total >= self.total_limit:
            return f"Total tool call limit reached: {total} calls in this turn"

        return None

    def reset(self) -> None:
        """Reset counters for a new conversation turn."""
        self._call_hashes.clear()
        self._tool_counts.clear()

# In the orchestrator's convoLoop, when loop_detector.check() returns a reason:
loop_break_message = {
    "role": "user",
    "content": [
        {
            "type": "text",
            "text": (
                "SYSTEM NOTE: You have called the same tool multiple times "
                "with the same parameters. The tool is working correctly. "
                "Please use the most recent tool result to formulate your "
                "response to the customer. Do not call the tool again."
            ),
        }
    ],
}

[2026-03-30 03:12:44 UTC] INFO  AWS Lambda — INIT_START
Runtime.Version = python:3.12
[2026-03-30 03:12:48 UTC] INFO  AWS Lambda — INIT_REPORT
Duration: 4200.55 ms   Init Duration: 4200.55 ms

[2026-03-30 03:12:48 UTC] ERROR mangaassist.chains — Tool
'product_search' timed out after 2000ms (Lambda init took 4200ms)

# Check Lambda init durations
aws logs filter-log-events \
  --log-group-name "/aws/lambda/mangaassist-tool-gateway" \
  --filter-pattern "INIT_REPORT" \
  --start-time 1711760400000 \
  --limit 20

# Check for correlation with tool timeouts
aws logs filter-log-events \
  --log-group-name "/aws/lambda/mangaassist-tool-gateway" \
  --filter-pattern '{ $.error_code = "GATEWAY_TIMEOUT" }' \
  --start-time 1711760400000 \
  --limit 20

# Add timing instrumentation to the Lambda module-level code
import time
_init_start = time.time()

import boto3  # ~600ms
_after_boto3 = time.time()

import jsonschema  # ~200ms
_after_jsonschema = time.time()

from mangaassist.tools.tool_registry import ToolRegistry  # ~800ms
_after_registry = time.time()

# ... OpenSearch client init: ~1200ms
# ... Redis connection: ~400ms

print(f"Init breakdown: boto3={_after_boto3 - _init_start:.0f}ms, "
      f"jsonschema={_after_jsonschema - _after_boto3:.0f}ms, "
      f"registry={_after_registry - _after_jsonschema:.0f}ms")

# Set provisioned concurrency to keep 5 instances warm at all times
aws lambda put-provisioned-concurrency-config \
  --function-name mangaassist-tool-gateway \
  --qualifier prod \
  --provisioned-concurrent-executions 5

# Verify configuration
aws lambda get-provisioned-concurrency-config \
  --function-name mangaassist-tool-gateway \
  --qualifier prod

"""
Optimization strategies for reducing Lambda cold start time.
"""

# Strategy 1: Lazy initialization — don't connect at import time
class LazyOpenSearchClient:
    """Only establish connection on first use, not at module load."""

    def __init__(self):
        self._client = None

    @property
    def client(self):
        if self._client is None:
            from opensearchpy import OpenSearch
            self._client = OpenSearch(
                hosts=[{"host": os.environ["OPENSEARCH_ENDPOINT"], "port": 443}],
                use_ssl=True,
                connection_class=RequestsHttpConnection,
            )
        return self._client


# Strategy 2: Reduce package size — use Lambda layers efficiently
# Move rarely-used dependencies into separate layers loaded on-demand

# Strategy 3: Use SnapStart equivalent for Python
# (As of 2026, AWS supports Lambda SnapStart for Python 3.12+)
# Configure in SAM template:
# Properties:
#   SnapStart:
#     ApplyOn: PublishedVersions

# In the tool chain executor, detect likely cold start and extend timeout
async def invoke_with_cold_start_awareness(
    self, tool_name: str, params: dict, base_timeout_ms: int = 2000
) -> dict:
    """
    Invoke a tool with extended timeout for likely cold starts.

    If this is the first call to this tool in the session, allow
    extra time for potential Lambda cold start.
    """
    is_first_call = tool_name not in self._warm_tools

    timeout_ms = base_timeout_ms
    if is_first_call:
        timeout_ms = base_timeout_ms + 5000  # Extra 5s for cold start
        logger.info(
            f"First call to '{tool_name}' — extending timeout to "
            f"{timeout_ms}ms for potential cold start"
        )

    result = await asyncio.wait_for(
        self._invoke(tool_name, params),
        timeout=timeout_ms / 1000,
    )

    self._warm_tools.add(tool_name)
    return result

[2026-03-31 08:45:12 UTC] INFO  FM -> tool_use: product_search(query="One Piece Vol 108")
[2026-03-31 08:45:13 UTC] INFO  FM -> tool_use: inventory_check(product_id="MNG-A1B2C3D4")
[2026-03-31 08:45:13 UTC] INFO  inventory_check returned: {"in_stock": false,
"restock_date": "2026-04-15"}
[2026-03-31 08:45:14 UTC] INFO  FM -> tool_use: recommendation_engine(
seed_title="One Piece Vol 108", exclude_owned=true)
[2026-03-31 08:45:15 UTC] INFO  recommendation_engine returned:
{"recommendations": [{"title": "One Piece Vol 107", "product_id": "MNG-E5F6G7H8"}]}
[2026-03-31 08:45:16 UTC] INFO  FM -> tool_use: inventory_check(product_id="MNG-E5F6G7H8")
[2026-03-31 08:45:16 UTC] INFO  inventory_check returned: {"in_stock": false}
[2026-03-31 08:45:17 UTC] INFO  FM -> tool_use: recommendation_engine(
seed_title="One Piece Vol 107", exclude_owned=true)
[2026-03-31 08:45:18 UTC] INFO  recommendation_engine returned:
{"recommendations": [{"title": "One Piece Vol 108", "product_id": "MNG-A1B2C3D4"}]}
[2026-03-31 08:45:19 UTC] INFO  FM -> tool_use: inventory_check(product_id="MNG-A1B2C3D4")
  ... [LOOP: Vol 108 -> Vol 107 -> Vol 108 -> Vol 107 -> ...]

# Find sessions with high tool call counts
aws logs filter-log-events \
  --log-group-name "/ecs/mangaassist-orchestrator" \
  --filter-pattern '{ $.tool_call_count > 8 }' \
  --start-time 1711871100000 \
  --limit 20

# Extract the tool call sequence for a specific session
aws logs filter-log-events \
  --log-group-name "/ecs/mangaassist-orchestrator" \
  --filter-pattern '{ $.session_id = "sess_abc123" && $.event = "tool_use" }' \
  --start-time 1711871100000

# In the orchestrator's convoLoop
MAX_TOOL_CALLS_PER_TURN = 10

tool_call_count = 0
while True:
    response = bedrock.converse(...)
    if response["stopReason"] == "tool_use":
        tool_call_count += 1
        if tool_call_count > MAX_TOOL_CALLS_PER_TURN:
            # Force the FM to respond without more tools
            messages.append({
                "role": "user",
                "content": [{
                    "type": "text",
                    "text": (
                        "SYSTEM: Maximum tool calls reached. Please provide "
                        "the best answer you can with the information gathered "
                        "so far. If a product is out of stock, tell the customer "
                        "the expected restock date and suggest they check back later."
                    ),
                }],
            })
            # Remove tools from the next call to force text response
            response = bedrock.converse(
                messages=messages,
                modelId=model_id,
                # No toolConfig -> FM must respond with text
            )
            break
    else:
        break

"""
cycle_detector.py — Detects circular patterns in tool call workflows.

Unlike the ToolLoopDetector which catches identical calls, this detects
cycles in the workflow graph where different tools with different params
form a repeating pattern.
"""

from collections import deque
from typing import Any, Dict, List, Optional, Tuple


class WorkflowCycleDetector:
    """
    Detects cyclic patterns in sequences of tool calls.

    A cycle is detected when a subsequence of tool calls repeats.
    For example: [A, B, C, A, B, C] has cycle [A, B, C] of length 3.

    Algorithm:
    - Maintain a sliding window of recent tool calls (name + key params)
    - After each call, check if the last N calls match the N calls before them
    - If so, a cycle of length N has been detected
    """

    def __init__(self, max_cycle_length: int = 5) -> None:
        self.max_cycle_length = max_cycle_length
        self._history: List[str] = []

    def record_and_check(
        self, tool_name: str, key_params: Dict[str, Any]
    ) -> Optional[Tuple[int, List[str]]]:
        """
        Record a tool call and check for cycles.

        Args:
            tool_name: Name of the tool being called.
            key_params: Key parameters that identify this specific call
                       (e.g., product_id, not timeout settings).

        Returns:
            None if no cycle, or (cycle_length, cycle_pattern) if detected.
        """
        # Create a fingerprint for this call
        fingerprint = f"{tool_name}:{sorted(key_params.items())}"
        self._history.append(fingerprint)

        # Check for cycles of various lengths
        for cycle_len in range(2, self.max_cycle_length + 1):
            if len(self._history) >= cycle_len * 2:
                recent = self._history[-cycle_len:]
                previous = self._history[-cycle_len * 2 : -cycle_len]
                if recent == previous:
                    pattern = [h.split(":")[0] for h in recent]
                    return (cycle_len, pattern)

        return None

    def reset(self) -> None:
        """Reset for a new conversation turn."""
        self._history.clear()

# Add to the MangaAssist system prompt:
ANTI_CYCLE_PROMPT_ADDITION = """
IMPORTANT RULES FOR TOOL USAGE:
- If you check inventory and a product is OUT OF STOCK, do NOT immediately
  search for alternatives and check their stock in a loop.
- Instead, tell the customer the restock date (if available) and ask if they
  would like recommendations for DIFFERENT types of manga (not the same series).
- Never call more than 6 tools total for a single customer question.
- If after 2 recommendation attempts you cannot find an in-stock alternative,
  inform the customer that the titles are temporarily out of stock and suggest
  they enable restock notifications.
"""

# Track which product IDs have been checked in the session
# and pass this context to the recommendation engine
@tool
def recommendation_engine_v2(
    seed_title: Optional[str] = None,
    exclude_product_ids: Optional[List[str]] = None,
    # ... other params
) -> Dict[str, Any]:
    """
    Generate manga recommendations.

    Args:
        exclude_product_ids: List of product IDs already checked and found
            out of stock. Recommendations will not include these products.
    """
    # ... implementation that filters out already-checked products