Tool Performance Framework Architecture

mindmap
  root((Tool Performance<br/>Frameworks))
    Call Pattern Tracking
      Invocation Frequency
      Call Chains
      Dependency Mapping
      Temporal Patterns
      Intent-to-Tool Correlation
    Performance Metrics
      Latency p50/p95/p99
      Success Rate
      Timeout Rate
      Parameter Accuracy
      Retry Rate
      Throughput per Tool
    Multi-Agent Coordination
      Handoff Tracing
      Context Transfer Completeness
      Coordination Overhead
      Agent Routing Accuracy
      Cross-Agent Tool Sharing
    Usage Baselines
      Rolling Averages 7d/30d
      Anomaly Types
        Missing Calls
        Excess Calls
        Wrong Tool Selection
      Health State Machine
      Seasonal Adjustment
      Trend Detection
    Tool Observability
      Instrumentation Decorator
      Structured Logging
      Trace Correlation via X-Ray
      Error Categorization
      Parameter Sanitization

product_search      ████████████████████████████  42%
order_lookup        ████████████████              24%
faq_retrieval       ████████████                  18%
inventory_check     ██████                         9%
price_lookup        ███                            4%
recommendation      █                              2%
return_initiate     ▏                              0.7%
shipping_estimate   ▏                              0.3%

graph LR
    subgraph UserLayer["User Layer"]
        U[Customer via WebSocket]
    end

    subgraph FMLayer["FM Decision Layer"]
        B[Bedrock Claude 3 Sonnet]
        TR[Tool Router<br/>Action Group Dispatcher]
    end

    subgraph ToolExecution["Tool Execution Layer"]
        direction TB
        PS[product_search<br/>OpenSearch]:::healthy
        OL[order_lookup<br/>DynamoDB]:::healthy
        IC[inventory_check<br/>DynamoDB]:::healthy
        PL[price_lookup<br/>DynamoDB]:::healthy
        RE[recommendation_engine<br/>OpenSearch k-NN]:::healthy
        FAQ[faq_retrieval<br/>OpenSearch]:::healthy
    end

    subgraph Instrumentation["Instrumentation Layer"]
        DEC[Decorator Wrapper<br/>@monitor.instrument]
        SL[Structured Logger<br/>JSON to CloudWatch Logs]
        XR[X-Ray Trace<br/>Segment per Tool Call]
    end

    subgraph Monitoring["Monitoring & Aggregation"]
        CW[CloudWatch Metrics<br/>Namespace: MangaAssist/Tools]
        CWL[CloudWatch Logs<br/>Insights Queries]
        XRD[X-Ray Service Map<br/>Tool Dependencies]
    end

    subgraph Analysis["Analysis Layer"]
        BL[Baseline Calculator<br/>7-day Rolling Window]
        AD[Anomaly Detector<br/>Statistical + ML]
        PT[Pattern Tracker<br/>Call Chain Analysis]
        HS[Health State Machine<br/>per-Tool Health]
    end

    subgraph Actions["Action Layer"]
        AL[CloudWatch Alarms<br/>per-Tool Thresholds]:::alert
        FB[Auto-Fallback<br/>Degrade Gracefully]:::alert
        SN[SNS Notifications<br/>PagerDuty/Slack]:::alert
        DB[Dashboard<br/>Tool Performance View]
    end

    U -->|query| B
    B -->|select tools| TR
    TR --> PS & OL & IC & PL & RE & FAQ

    PS & OL & IC & PL & RE & FAQ --> DEC
    DEC --> SL & XR
    SL --> CWL
    XR --> XRD
    DEC --> CW

    CW --> BL & AD & PT & HS
    CWL --> PT

    BL --> AD
    AD --> AL
    HS --> AL & FB
    AL --> SN
    PT --> DB
    HS --> DB

    classDef healthy fill:#d4edda,stroke:#28a745,color:#155724
    classDef alert fill:#f8d7da,stroke:#dc3545,color:#721c24

sequenceDiagram
    participant User
    participant WS as API Gateway<br/>WebSocket
    participant ECS as ECS Fargate<br/>Orchestrator
    participant FM as Bedrock Claude 3<br/>Sonnet
    participant MON as ToolPerformance<br/>Monitor
    participant PS as product_search<br/>OpenSearch
    participant IC as inventory_check<br/>DynamoDB
    participant PL as price_lookup<br/>DynamoDB
    participant RE as recommendation<br/>OpenSearch k-NN
    participant CW as CloudWatch

    User->>WS: "Is One Piece Vol 107 in stock?<br/>What's the price and similar options?"
    WS->>ECS: Route message
    ECS->>FM: InvokeModel (with tool definitions)

    Note over FM: FM plans: product_search →<br/>inventory_check + price_lookup (parallel) →<br/>recommendation_engine

    rect rgb(220, 240, 220)
        Note over FM,PS: Step 1: product_search
        FM->>ECS: tool_use: product_search(query="One Piece Volume 107")
        ECS->>MON: start_call("product_search", params)
        MON->>MON: Record start timestamp
        ECS->>PS: Search query + embedding
        PS-->>ECS: [{product_id: "OP-107", title: "One Piece Vol 107", ...}]
        ECS->>MON: end_call(duration=115ms, status=success)
        MON->>CW: PutMetricData(Latency=115, ToolName=product_search)
        MON->>MON: Check baseline: 115ms < p95(180ms) ✓
    end

    ECS->>FM: tool_result: product found (product_id=OP-107)

    rect rgb(255, 230, 230)
        Note over FM,IC: Step 2a: inventory_check (TIMEOUT)
        FM->>ECS: tool_use: inventory_check(product_id="OP-107")
        ECS->>MON: start_call("inventory_check", params)
        ECS->>IC: GetItem(PK=OP-107)
        Note over IC: DynamoDB throttled!<br/>3000ms timeout
        IC--xECS: Timeout after 3000ms
        ECS->>MON: end_call(duration=3000ms, status=timeout)
        MON->>CW: PutMetricData(Timeout=1, ToolName=inventory_check)
        MON->>MON: Check baseline: TIMEOUT exceeds p95(90ms) ⚠️
        MON->>CW: PutMetricData(AnomalyDetected=1, AnomalyType=latency_spike)
        MON->>MON: Update health: Healthy → Degraded
        ECS->>ECS: Fallback: return cached inventory<br/>(stale_ok=true, cache_age=5min)
    end

    rect rgb(220, 240, 220)
        Note over FM,PL: Step 2b: price_lookup (parallel with 2a)
        FM->>ECS: tool_use: price_lookup(product_id="OP-107")
        ECS->>MON: start_call("price_lookup", params)
        ECS->>PL: GetItem(PK=OP-107#pricing)
        PL-->>ECS: {base: $9.99, discount: 10%, final: $8.99}
        ECS->>MON: end_call(duration=48ms, status=success)
        MON->>CW: PutMetricData(Latency=48, ToolName=price_lookup)
    end

    ECS->>FM: tool_results: inventory(fallback), pricing(success)

    rect rgb(220, 240, 220)
        Note over FM,RE: Step 3: recommendation_engine
        FM->>ECS: tool_use: recommendation_engine(customer_id="C-123", context="One Piece")
        ECS->>MON: start_call("recommendation_engine", params)
        ECS->>RE: k-NN search + collaborative filtering
        RE-->>ECS: [{Jujutsu Kaisen Vol 25, ...}, {Chainsaw Man Vol 17, ...}]
        ECS->>MON: end_call(duration=195ms, status=success)
        MON->>CW: PutMetricData(Latency=195, ToolName=recommendation_engine)
    end

    ECS->>FM: tool_result: recommendations
    FM->>ECS: Final response with all data
    ECS->>MON: record_chain(pattern=conditional, tools=4, total=3358ms, bottleneck=inventory_check)
    MON->>CW: PutMetricData(ChainDuration=3358, ChainPattern=conditional)
    ECS->>WS: Stream response
    WS->>User: "One Piece Vol 107 is likely in stock<br/>(last checked 5min ago) at $8.99 (10% off!).<br/>You might also like Jujutsu Kaisen Vol 25..."

stateDiagram-v2
    [*] --> IntentClassification: User query arrives

    IntentClassification --> SingleTool: Simple factual query
    IntentClassification --> SequentialChain: Multi-step lookup
    IntentClassification --> ParallelCalls: Independent data needs
    IntentClassification --> ConditionalChain: Depends on first result
    IntentClassification --> Iterative: Refinement needed

    state SingleTool {
        [*] --> CallToolA
        CallToolA --> ReturnResult
        ReturnResult --> [*]
        note right of CallToolA
            Example: "What's my order status?"
            → order_lookup(order_id)
            Expected: 1 tool, ~50ms
        end note
    }

    state SequentialChain {
        [*] --> ToolA_Seq
        ToolA_Seq --> ToolB_Seq: A output → B input
        ToolB_Seq --> ToolC_Seq: B output → C input
        ToolC_Seq --> MergeResults_Seq
        MergeResults_Seq --> [*]
        note right of ToolA_Seq
            Example: "Can I return my last order?"
            → order_lookup → return_initiate
            Expected: 2-3 tools, sequential
        end note
    }

    state ParallelCalls {
        [*] --> ForkPoint
        ForkPoint --> ToolA_Par
        ForkPoint --> ToolB_Par
        ToolA_Par --> JoinPoint
        ToolB_Par --> JoinPoint
        JoinPoint --> [*]
        note right of ForkPoint
            Example: "Compare price and stock of OP vol 107"
            → inventory_check ‖ price_lookup
            Expected: 2 tools, parallel, ~max(60,55)ms
        end note
    }

    state ConditionalChain {
        [*] --> ToolA_Cond
        ToolA_Cond --> CheckCondition
        CheckCondition --> ToolB_Cond: Condition TRUE
        CheckCondition --> ToolC_Cond: Condition FALSE
        ToolB_Cond --> [*]
        ToolC_Cond --> [*]
        note right of CheckCondition
            Example: "Is OP 107 in stock? If yes, price?"
            → product_search → if found → price_lookup
                              → if not → faq_retrieval
        end note
    }

    state Iterative {
        [*] --> FirstCall
        FirstCall --> EvaluateResult
        EvaluateResult --> RefinedCall: Not sufficient
        EvaluateResult --> [*]: Sufficient
        RefinedCall --> EvaluateResult
        note right of RefinedCall
            Example: "Find manga like Naruto under $10"
            → product_search("Naruto similar under 10")
            → product_search("shonen action manga budget")
            Expected: 2-4 calls, same tool
        end note
    }

from dataclasses import dataclass, field
from typing import Dict, List, Optional
from datetime import datetime
from enum import Enum


class ToolCallStatus(Enum):
    """Outcome of a single tool invocation."""
    SUCCESS = "success"
    TIMEOUT = "timeout"
    ERROR = "error"
    FALLBACK = "fallback"
    RETRIED = "retried"


class CallPattern(Enum):
    """How tools were orchestrated within a single user request."""
    SINGLE = "single"
    SEQUENTIAL_CHAIN = "sequential_chain"
    PARALLEL = "parallel"
    CONDITIONAL = "conditional"
    ITERATIVE = "iterative"


class AnomalyType(Enum):
    """Types of tool usage anomalies the FM can exhibit."""
    MISSING_CALL = "missing_call"          # FM should have called a tool but didn't
    EXCESS_CALL = "excess_call"            # FM called a tool unnecessarily
    WRONG_TOOL = "wrong_tool"              # FM picked the wrong tool for the intent
    WRONG_PARAMETERS = "wrong_parameters"  # FM sent incorrect/incomplete params
    LATENCY_SPIKE = "latency_spike"        # Tool took much longer than baseline
    REPEATED_FAILURE = "repeated_failure"  # Same tool failing across requests


class HealthState(Enum):
    """Health state of a tool (state machine)."""
    HEALTHY = "healthy"
    DEGRADED = "degraded"
    UNHEALTHY = "unhealthy"
    FAILED = "failed"
    RECOVERING = "recovering"


@dataclass
class ToolCallRecord:
    """Record of a single tool invocation."""
    call_id: str
    request_id: str                        # Ties to user request
    session_id: str                        # Ties to WebSocket session
    tool_name: str
    timestamp: datetime
    duration_ms: float
    status: ToolCallStatus
    input_parameters: Dict[str, str]       # Sanitized — no PII or secrets
    output_summary: str                    # Truncated output for debugging
    parameter_accuracy: float              # 0.0–1.0: did FM provide correct params?
    retry_count: int = 0
    parent_call_id: Optional[str] = None   # For chained calls
    error_message: Optional[str] = None
    fallback_used: bool = False
    trace_id: Optional[str] = None         # X-Ray trace ID


@dataclass
class ToolCallChain:
    """A sequence of related tool calls for one user request."""
    chain_id: str
    request_id: str
    pattern: CallPattern
    calls: List[ToolCallRecord] = field(default_factory=list)
    total_duration_ms: float = 0.0
    chain_success: bool = True
    bottleneck_tool: Optional[str] = None  # The slowest tool in the chain
    intent_category: Optional[str] = None  # "order_status", "product_inquiry", etc.


@dataclass
class ToolBaseline:
    """Rolling baseline for a specific tool — used for anomaly detection."""
    tool_name: str
    window_days: int = 7
    avg_latency_ms: float = 0.0
    p50_latency_ms: float = 0.0
    p95_latency_ms: float = 0.0
    p99_latency_ms: float = 0.0
    avg_calls_per_hour: float = 0.0
    success_rate: float = 1.0
    timeout_rate: float = 0.0
    avg_parameter_accuracy: float = 1.0
    avg_retry_rate: float = 0.0
    last_updated: datetime = field(default_factory=datetime.utcnow)


@dataclass
class ToolHealthSnapshot:
    """Current health state of a tool."""
    tool_name: str
    state: HealthState
    since: datetime
    consecutive_failures: int = 0
    last_success: Optional[datetime] = None
    degradation_reason: Optional[str] = None
    current_latency_ms: float = 0.0
    current_success_rate: float = 1.0

import boto3
import time
import functools
import json
import logging
import uuid
from datetime import datetime, timedelta
from typing import Dict, List, Callable, Any, Optional
from dataclasses import asdict
from enum import Enum

logger = logging.getLogger("manga_assist.tool_monitor")
logger.setLevel(logging.INFO)


class ToolPerformanceMonitor:
    """
    Monitors and tracks performance of all FM tool calls in MangaAssist.

    Provides:
    - Decorator-based instrumentation for zero-code-change monitoring
    - CloudWatch metric emission per tool call
    - Baseline tracking with rolling 7-day windows
    - Anomaly detection against baselines
    - Health state machine per tool
    - Call chain tracking for multi-tool requests
    """

    # Sensitive parameter keys to redact
    SENSITIVE_KEYS = frozenset({
        "password", "token", "key", "secret", "auth",
        "credit_card", "ssn", "card_number", "cvv"
    })

    def __init__(self, config: dict):
        self.cloudwatch = boto3.client("cloudwatch")
        self.namespace = config.get("namespace", "MangaAssist/Tools")
        self.environment = config.get("environment", "production")
        self.anomaly_threshold = config.get("anomaly_threshold", 1.5)  # multiplier on p95
        self.health_check_window = config.get("health_check_window_sec", 300)

        self._baselines: Dict[str, dict] = {}
        self._health_states: Dict[str, dict] = {}
        self._active_chains: Dict[str, list] = {}  # request_id → list of call records

    # ------------------------------------------------------------------ #
    #  Decorator — primary instrumentation entry point                    #
    # ------------------------------------------------------------------ #

    def instrument(self, tool_name: str, timeout_ms: float = 5000):
        """Decorator to instrument a tool function with performance monitoring."""
        def decorator(func: Callable) -> Callable:
            @functools.wraps(func)
            def wrapper(*args, **kwargs) -> Any:
                call_id = f"{tool_name}-{uuid.uuid4().hex[:12]}"
                request_id = kwargs.pop("_request_id", None)
                start = time.monotonic()
                status = "success"
                error_msg = None
                result = None

                try:
                    result = func(*args, **kwargs)
                except TimeoutError:
                    status = "timeout"
                    error_msg = "Tool execution timed out"
                    raise
                except Exception as e:
                    status = "error"
                    error_msg = str(e)[:200]  # Truncate to avoid log bloat
                    raise
                finally:
                    duration_ms = (time.monotonic() - start) * 1000
                    self._record_call(
                        call_id=call_id,
                        request_id=request_id,
                        tool_name=tool_name,
                        duration_ms=duration_ms,
                        status=status,
                        error_message=error_msg,
                        parameters=self._sanitize_params(kwargs),
                    )
                return result
            return wrapper
        return decorator

    # ------------------------------------------------------------------ #
    #  Core metric recording                                              #
    # ------------------------------------------------------------------ #

    def _record_call(
        self,
        call_id: str,
        request_id: Optional[str],
        tool_name: str,
        duration_ms: float,
        status: str,
        error_message: Optional[str],
        parameters: dict,
    ):
        """Record a tool call to CloudWatch metrics and structured logs."""
        dimensions = [
            {"Name": "ToolName", "Value": tool_name},
            {"Name": "Environment", "Value": self.environment},
        ]

        metric_data = [
            {
                "MetricName": "CallCount",
                "Value": 1,
                "Unit": "Count",
                "Dimensions": dimensions,
            },
            {
                "MetricName": "Latency",
                "Value": duration_ms,
                "Unit": "Milliseconds",
                "Dimensions": dimensions,
                "StorageResolution": 1,  # High-resolution (1-second)
            },
            {
                "MetricName": "Success",
                "Value": 1 if status == "success" else 0,
                "Unit": "Count",
                "Dimensions": dimensions,
            },
            {
                "MetricName": "Error",
                "Value": 1 if status == "error" else 0,
                "Unit": "Count",
                "Dimensions": dimensions,
            },
            {
                "MetricName": "Timeout",
                "Value": 1 if status == "timeout" else 0,
                "Unit": "Count",
                "Dimensions": dimensions,
            },
        ]

        self.cloudwatch.put_metric_data(
            Namespace=self.namespace,
            MetricData=metric_data,
        )

        # Structured log for CloudWatch Logs Insights queries
        log_record = {
            "event": "tool_call",
            "call_id": call_id,
            "request_id": request_id,
            "tool_name": tool_name,
            "duration_ms": round(duration_ms, 2),
            "status": status,
            "error_message": error_message,
            "parameters": parameters,
            "timestamp": datetime.utcnow().isoformat(),
        }
        logger.info(json.dumps(log_record))

        # Track chain if request_id present
        if request_id:
            self._active_chains.setdefault(request_id, []).append(log_record)

        # Baseline comparison
        self._check_baseline(tool_name, duration_ms, status)

        # Health state update
        self._update_health(tool_name, duration_ms, status)

    # ------------------------------------------------------------------ #
    #  Baseline management                                                #
    # ------------------------------------------------------------------ #

    def update_baseline(self, tool_name: str, baseline: dict):
        """Set or update the rolling baseline for a tool."""
        self._baselines[tool_name] = baseline
        logger.info(json.dumps({
            "event": "baseline_updated",
            "tool_name": tool_name,
            "baseline": baseline,
        }))

    def _check_baseline(self, tool_name: str, duration_ms: float, status: str):
        """Compare current call against baseline and emit anomaly if needed."""
        baseline = self._baselines.get(tool_name)
        if not baseline:
            return

        p95 = baseline.get("p95_latency_ms", float("inf"))
        threshold = p95 * self.anomaly_threshold

        if duration_ms > threshold:
            self._emit_anomaly(
                tool_name,
                "latency_spike",
                f"Latency {duration_ms:.0f}ms exceeds {self.anomaly_threshold}x "
                f"p95 baseline {p95:.0f}ms (threshold: {threshold:.0f}ms)",
            )

        expected_success_rate = baseline.get("success_rate", 1.0)
        if status != "success" and expected_success_rate > 0.95:
            self._emit_anomaly(
                tool_name,
                "unexpected_failure",
                f"Tool failed with status={status} but baseline success rate "
                f"is {expected_success_rate:.1%}",
            )

    # ------------------------------------------------------------------ #
    #  Anomaly emission                                                   #
    # ------------------------------------------------------------------ #

    def _emit_anomaly(self, tool_name: str, anomaly_type: str, description: str):
        """Emit an anomaly metric and log."""
        self.cloudwatch.put_metric_data(
            Namespace=self.namespace,
            MetricData=[
                {
                    "MetricName": "AnomalyDetected",
                    "Value": 1,
                    "Unit": "Count",
                    "Dimensions": [
                        {"Name": "ToolName", "Value": tool_name},
                        {"Name": "AnomalyType", "Value": anomaly_type},
                    ],
                }
            ],
        )
        logger.warning(json.dumps({
            "event": "tool_anomaly",
            "tool_name": tool_name,
            "anomaly_type": anomaly_type,
            "description": description,
            "timestamp": datetime.utcnow().isoformat(),
        }))

    # ------------------------------------------------------------------ #
    #  Health state machine                                               #
    # ------------------------------------------------------------------ #

    def _update_health(self, tool_name: str, duration_ms: float, status: str):
        """Update the health state machine for a tool."""
        state = self._health_states.setdefault(tool_name, {
            "state": "healthy",
            "consecutive_failures": 0,
            "recent_latencies": [],
            "recent_statuses": [],
        })

        state["recent_latencies"].append(duration_ms)
        state["recent_statuses"].append(status)

        # Keep only last 20 calls for rolling calculation
        state["recent_latencies"] = state["recent_latencies"][-20:]
        state["recent_statuses"] = state["recent_statuses"][-20:]

        recent_success_rate = (
            state["recent_statuses"].count("success")
            / len(state["recent_statuses"])
        )
        avg_recent_latency = (
            sum(state["recent_latencies"]) / len(state["recent_latencies"])
        )

        baseline = self._baselines.get(tool_name, {})
        p95 = baseline.get("p95_latency_ms", 1000)
        prev_state = state["state"]

        if status != "success":
            state["consecutive_failures"] += 1
        else:
            state["consecutive_failures"] = 0

        # State transitions
        if recent_success_rate < 0.50 or state["consecutive_failures"] >= 5:
            state["state"] = "failed"
        elif recent_success_rate < 0.90 or avg_recent_latency > p95 * 2:
            state["state"] = "unhealthy"
        elif recent_success_rate < 0.97 or avg_recent_latency > p95 * 1.3:
            if prev_state in ("unhealthy", "failed"):
                state["state"] = "recovering"
            else:
                state["state"] = "degraded"
        else:
            if prev_state == "recovering":
                state["state"] = "healthy"  # Recovery complete
            else:
                state["state"] = "healthy"

        # Emit health state metric
        if state["state"] != prev_state:
            self.cloudwatch.put_metric_data(
                Namespace=self.namespace,
                MetricData=[{
                    "MetricName": "HealthStateChange",
                    "Value": 1,
                    "Unit": "Count",
                    "Dimensions": [
                        {"Name": "ToolName", "Value": tool_name},
                        {"Name": "FromState", "Value": prev_state},
                        {"Name": "ToState", "Value": state["state"]},
                    ],
                }],
            )
            logger.warning(json.dumps({
                "event": "health_state_change",
                "tool_name": tool_name,
                "from": prev_state,
                "to": state["state"],
                "success_rate": round(recent_success_rate, 3),
                "avg_latency_ms": round(avg_recent_latency, 1),
            }))

    def get_health(self, tool_name: str) -> str:
        """Get current health state of a tool."""
        return self._health_states.get(tool_name, {}).get("state", "unknown")

    # ------------------------------------------------------------------ #
    #  Call chain tracking                                                #
    # ------------------------------------------------------------------ #

    def finalize_chain(self, request_id: str, intent_category: str = "unknown") -> dict:
        """Finalize and record a call chain after all tools complete."""
        calls = self._active_chains.pop(request_id, [])
        if not calls:
            return {}

        total_duration = sum(c["duration_ms"] for c in calls)
        all_success = all(c["status"] == "success" for c in calls)
        bottleneck = max(calls, key=lambda c: c["duration_ms"])
        pattern = self._classify_pattern(calls)

        chain_record = {
            "chain_id": f"chain-{uuid.uuid4().hex[:12]}",
            "request_id": request_id,
            "pattern": pattern,
            "tool_count": len(calls),
            "total_duration_ms": round(total_duration, 2),
            "chain_success": all_success,
            "bottleneck_tool": bottleneck["tool_name"],
            "bottleneck_duration_ms": round(bottleneck["duration_ms"], 2),
            "intent_category": intent_category,
            "tools": [c["tool_name"] for c in calls],
        }

        # Emit chain-level metrics
        self.cloudwatch.put_metric_data(
            Namespace=self.namespace,
            MetricData=[
                {
                    "MetricName": "ChainDuration",
                    "Value": total_duration,
                    "Unit": "Milliseconds",
                    "Dimensions": [
                        {"Name": "ChainPattern", "Value": pattern},
                        {"Name": "IntentCategory", "Value": intent_category},
                    ],
                },
                {
                    "MetricName": "ChainToolCount",
                    "Value": len(calls),
                    "Unit": "Count",
                    "Dimensions": [
                        {"Name": "ChainPattern", "Value": pattern},
                    ],
                },
            ],
        )

        logger.info(json.dumps({"event": "chain_complete", **chain_record}))
        return chain_record

    def _classify_pattern(self, calls: list) -> str:
        """Classify the call pattern based on observed calls."""
        if len(calls) == 1:
            return "single"
        tool_names = [c["tool_name"] for c in calls]
        if len(set(tool_names)) < len(tool_names):
            return "iterative"   # Same tool called multiple times
        return "sequential_chain"  # Default for multi-tool

    # ------------------------------------------------------------------ #
    #  Parameter sanitization                                             #
    # ------------------------------------------------------------------ #

    def _sanitize_params(self, params: dict) -> dict:
        """Remove sensitive data from parameters before logging."""
        sanitized = {}
        for key, value in params.items():
            if any(sensitive in key.lower() for sensitive in self.SENSITIVE_KEYS):
                sanitized[key] = "***REDACTED***"
            else:
                sanitized[key] = str(value)[:100]  # Truncate long values
        return sanitized


# ====================================================================== #
#  Usage example — how tools are instrumented in MangaAssist              #
# ====================================================================== #

monitor = ToolPerformanceMonitor({
    "namespace": "MangaAssist/Tools",
    "environment": "production",
    "anomaly_threshold": 1.5,
})

# Set baselines (normally loaded from DynamoDB or CloudWatch statistics)
monitor.update_baseline("product_search", {
    "p50_latency_ms": 120,
    "p95_latency_ms": 180,
    "p99_latency_ms": 350,
    "success_rate": 0.995,
})
monitor.update_baseline("inventory_check", {
    "p50_latency_ms": 60,
    "p95_latency_ms": 90,
    "p99_latency_ms": 150,
    "success_rate": 0.998,
})


@monitor.instrument("product_search")
def product_search(query: str, category: str = None, top_k: int = 5) -> list:
    """Search OpenSearch for manga products."""
    # ... actual OpenSearch query implementation ...
    pass


@monitor.instrument("order_lookup")
def order_lookup(order_id: str = None, customer_id: str = None) -> dict:
    """Look up order from DynamoDB."""
    # ... actual DynamoDB GetItem implementation ...
    pass


@monitor.instrument("inventory_check", timeout_ms=3000)
def inventory_check(product_id: str, warehouse: str = None) -> dict:
    """Check product inventory levels."""
    # ... actual DynamoDB query implementation ...
    pass

sequenceDiagram
    participant User
    participant Sup as Supervisor Agent<br/>(Bedrock Claude 3 Sonnet)
    participant MON as Tool Performance<br/>Monitor
    participant PA as Product Agent<br/>(Bedrock Claude 3 Haiku)
    participant OA as Order Agent<br/>(Bedrock Claude 3 Haiku)
    participant SA as Support Agent<br/>(Bedrock Claude 3 Haiku)
    participant PS as product_search
    participant RE as recommendation
    participant OL as order_lookup
    participant FAQ as faq_retrieval

    User->>Sup: "I ordered One Piece Vol 107 last week,<br/>hasn't arrived. Also any similar manga deals?"

    Note over Sup: Intent: order_issue + product_recommendation<br/>Route to Order Agent AND Product Agent

    rect rgb(230, 240, 255)
        Note over Sup,OA: Handoff 1: Order Agent
        Sup->>MON: record_handoff(from=supervisor, to=order_agent, context_size=245 tokens)
        Sup->>OA: Delegated: "Order issue — One Piece Vol 107, ordered last week"
        OA->>OL: order_lookup(customer_id="C-123")
        MON-->>MON: instrument(order_lookup, 42ms, success)
        OL-->>OA: {order_id: "ORD-789", status: "in_transit", ETA: "2 days"}
        OA-->>Sup: "Order ORD-789 is in transit, arriving in 2 days"
        MON-->>MON: record_handoff_return(duration=180ms, context_preserved=true)
    end

    rect rgb(255, 245, 230)
        Note over Sup,PA: Handoff 2: Product Agent (parallel)
        Sup->>MON: record_handoff(from=supervisor, to=product_agent, context_size=180 tokens)
        Sup->>PA: Delegated: "Find similar manga deals to One Piece"
        PA->>PS: product_search(query="manga similar to One Piece deals")
        MON-->>MON: instrument(product_search, 125ms, success)
        PS-->>PA: [Jujutsu Kaisen, Chainsaw Man, ...]
        PA->>RE: recommendation_engine(customer_id="C-123", context="One Piece similar")
        MON-->>MON: instrument(recommendation, 190ms, success)
        RE-->>PA: [Dragon Ball Super Vol 22 (20% off), My Hero Academia Vol 40]
        PA-->>Sup: "Similar: JJK ($9.99), Dragon Ball Super ($7.99, 20% off)"
        MON-->>MON: record_handoff_return(duration=420ms, context_preserved=true)
    end

    Sup->>User: "Your One Piece Vol 107 (ORD-789) is in transit — arriving in ~2 days!<br/>Meanwhile, Dragon Ball Super Vol 22 is 20% off at $7.99..."

    MON->>MON: finalize_multi_agent_chain<br/>agents=3, tools=3, total=480ms

stateDiagram-v2
    [*] --> Healthy

    Healthy --> Degraded: Latency > 1.3x p95 baseline<br/>OR success rate 90-97%
    Healthy --> Unhealthy: Latency > 2x p95 baseline<br/>OR success rate < 90%

    Degraded --> Healthy: Metrics return to baseline<br/>for 5+ consecutive calls
    Degraded --> Unhealthy: Latency > 2x p95 baseline<br/>OR success rate drops < 90%

    Unhealthy --> Failed: Success rate < 50%<br/>OR 5 consecutive failures
    Unhealthy --> Recovering: Success rate improving<br/>AND latency trending down

    Failed --> Recovering: At least 1 successful call<br/>AND latency < 2x p95

    Recovering --> Healthy: All metrics within baseline<br/>for 10+ consecutive calls
    Recovering --> Unhealthy: Metrics deteriorate again
    Recovering --> Failed: 3 consecutive failures during recovery

    state Healthy {
        [*] --> Normal
        Normal: ✅ All metrics within baseline
        Normal: Latency ≤ p95
        Normal: Success rate ≥ 97%
        Normal: No alerts active
    }

    state Degraded {
        [*] --> Warning
        Warning: ⚠️ Elevated but functional
        Warning: Alert: P3 (informational)
        Warning: Action: Log + dashboard update
    }

    state Unhealthy {
        [*] --> Critical
        Critical: 🔶 Significant degradation
        Critical: Alert: P2 (requires attention)
        Critical: Action: Increase fallback rate
        Critical: Action: Notify on-call team
    }

    state Failed {
        [*] --> Down
        Down: 🔴 Tool effectively unavailable
        Down: Alert: P1 (immediate action)
        Down: Action: Full fallback mode
        Down: Action: Page on-call engineer
    }

    state Recovering {
        [*] --> Improving
        Improving: 🔄 Trending toward healthy
        Improving: Alert: P3 (monitoring)
        Improving: Action: Gradually reduce fallback
    }