02. API Testing Strategy — How I Tested Every Layer

"Testing a chatbot isn't like testing a CRUD API. You have non-deterministic LLM outputs, a 4-model inference chain, 9 downstream service integrations, WebSocket streaming, and a guardrails pipeline that needs to catch hallucinated prices. I built 6 types of tests to cover all of it — from unit tests on regex rules to LLM quality evaluation with golden datasets."

Testing Pyramid Overview

1. Unit Testing

Unit tests covered all deterministic logic — the parts that don't involve ML models or external services.

What I Tested

Tools and Approach

• Framework: pytest with parametrized test cases
• Mocking: unittest.mock for SageMaker and Bedrock clients, moto for DynamoDB and S3
• Key pattern: Every guardrail rule was tested with both positive cases (should catch) and negative cases (should pass through). I maintained a test fixture of 50+ adversarial inputs per guardrail

# Example: Testing the PII guardrail never leaks credit card numbers
@pytest.mark.parametrize("input_text,expected_redacted", [
    ("My card is 4111-1111-1111-1111", "My card is [REDACTED]"),
    ("Call me at 555-123-4567", "Call me at [PHONE]"),
    ("Email: user@example.com", "Email: [EMAIL]"),
    ("No PII in this message", "No PII in this message"),  # passthrough
])
def test_pii_redaction(input_text, expected_redacted):
    result = pii_filter.redact(input_text)
    assert result == expected_redacted

Key Insight

I invested heavily in unit testing the guardrails because they're deterministic — unlike LLM outputs, I could assert exact expected behavior. This gave me high confidence that safety controls would hold regardless of what the LLM generated.

2. Integration Testing

Integration tests verified that the orchestrator correctly communicated with each downstream service under real conditions.

What I Tested

Tools

• LocalStack: Local emulation of DynamoDB, S3, SQS, Kinesis — ran in Docker during CI
• Test Containers: Spun up Redis and OpenSearch containers per test run
• WireMock: Simulated downstream REST APIs with configurable delays and error responses
• Staging SageMaker endpoints: For ML model integrations, I used real SageMaker staging endpoints (not mocks) because latency characteristics matter

Key Test: Retry and Circuit Breaker Behavior

def test_catalog_service_retry_on_timeout():
    """Verify orchestrator retries twice with backoff then opens circuit breaker."""
    # Configure WireMock to return 504 for first 3 calls, then 200
    wiremock.stub_for(
        get(url_path="/catalog/B08X1YRSTR")
        .in_scenario("retry-test")
        .when_scenario_state_is("Started")
        .will_return(status=504, delay=3000)
        .will_set_state_to("failed-once")
    )

    response = orchestrator.fetch_product("B08X1YRSTR")

    # After 2 retries + circuit breaker open: should get fallback response
    assert response.source == "fallback"
    assert response.message == "I couldn't load product details right now."
    assert metrics.get("circuit_breaker_open_total") == 1

Key Insight

I never mocked the SageMaker endpoint for integration tests. Mocks hide real latency issues — a test that completes in 1ms doesn't tell you the endpoint takes 15ms in staging and 55ms at P99 under load. Using real staging endpoints caught a cold-start issue in the intent classifier that mocks would have missed.

3. Contract Testing (Consumer-Driven)

MangaAssist consumed APIs owned by 6 different teams (Catalog, Orders, Returns, Promotions, Shipping, Reviews). Any of them could deploy a breaking change without telling us.

How I Set It Up

• Framework: Pact (consumer-driven contract testing)
• MangaAssist = Consumer: I wrote Pact contracts specifying exactly what fields I needed from each provider
• Provider teams ran verification: Each provider's CI pipeline verified their service satisfied our contracts

Contract Examples

// Contract: MangaAssist expects from Product Catalog
{
  "consumer": "manga-assist",
  "provider": "product-catalog",
  "interactions": [
    {
      "description": "Get product by ASIN",
      "request": {
        "method": "GET",
        "path": "/catalog/B08X1YRSTR"
      },
      "response": {
        "status": 200,
        "body": {
          "asin": "B08X1YRSTR",
          "title": "string",
          "author": "string",
          "format": "string",
          "language": "string",
          "price": {
            "current": "number",
            "list": "number",
            "currency": "string"
          },
          "in_stock": "boolean"
        }
      }
    }
  ]
}

What Contracts Caught

Key Insight

Contract tests aren't about testing our code — they're about protecting us from other teams' changes. In a microservices environment with 9 upstream dependencies, this was essential. Without contracts, we would have discovered breaking changes in production.

4. End-to-End Testing

E2E tests exercised the complete chat flow — from WebSocket connection through orchestration, ML inference, and response streaming.

Test Scenarios (All 8 Intent Paths)

Multi-Turn Context Test

async def test_multi_turn_context():
    """Verify the chatbot remembers context across turns."""
    ws = await connect_websocket(session_id="test-multi-turn")

    # Turn 1: Ask for recommendations
    await ws.send("Recommend action manga")
    response_1 = await ws.receive_full_response()
    assert len(response_1["products"]) >= 3
    first_product = response_1["products"][0]["title"]

    # Turn 2: Reference previous response
    await ws.send("Tell me more about the first one")
    response_2 = await ws.receive_full_response()
    assert first_product in response_2["response_text"]

    # Turn 3: Add to cart
    await ws.send("Add it to my cart")
    response_3 = await ws.receive_full_response()
    assert response_3["actions"][0]["type"] == "add_to_cart"
    assert response_3["actions"][0]["asin"] == response_1["products"][0]["asin"]

Tools

• WebSocket client: Custom pytest fixture wrapping websockets library for streaming assertions
• Newman (Postman CLI): For REST fallback endpoint E2E tests — ran Postman collections in CI
• Seeded test data: Pre-loaded DynamoDB test sessions and a test product catalog with 100 known ASINs

Key Insight

The hardest E2E test was multi-turn context ("Tell me more about the second one you mentioned"). This tested conversation memory (DynamoDB), prompt construction (injecting history), and the LLM's ability to resolve anaphoric references. It caught a bug where the memory summarizer was stripping product ASINs from older turns, breaking context resolution.

5. LLM Quality / Evaluation Testing

LLM outputs are non-deterministic — you can't assert exact strings. I built a quality evaluation framework that gated model promotions.

Golden Dataset

500 curated test prompts with expected properties (not exact expected outputs):

Evaluation Metrics

LLM-as-a-Judge Framework

For subjective quality (tone, helpfulness, conciseness), I used a separate Claude instance to evaluate responses:

judge_prompt = """
You are evaluating a chatbot response for a manga store.

USER QUERY: {query}
CHATBOT RESPONSE: {response}
EXPECTED BEHAVIOR: {expected}

Rate the response on these dimensions (1-5):
1. Relevance: Does it answer the user's question?
2. Accuracy: Are product details correct?
3. Helpfulness: Would this help the user make a purchase decision?
4. Tone: Is it friendly and professional?

Return a JSON object with scores and a one-line justification for each.
"""

Gate Criteria

A model promotion (staging → production) required:

• All golden dataset metrics above thresholds
• No regression > 5% on any metric vs. the current production model
• Guardrail adversarial test pass rate > 98%
• P99 latency within budget

This ran as a CodePipeline step. If any gate failed, the promotion was blocked automatically.

Key Insight

The most valuable eval wasn't accuracy — it was the hallucination check. Early in development, the LLM generated plausible-sounding prices about 6% of the time. I couldn't fix this with prompting alone. The combination of the price validation guardrail (runtime) + hallucination rate metric (eval gate) brought it to < 0.5% in production.

6. Security Testing

Security testing targeted the unique attack surfaces of a chatbot system — prompt injection, PII leakage, auth bypass, and abuse.

Test Categories

Prompt Injection Defense (Tested Scenarios)

Key Insight

Security testing for chatbots goes beyond OWASP Web Top 10. The LLM introduces a novel attack surface — prompt injection — that doesn't exist in traditional APIs. I treated the input validator and guardrails pipeline as security controls with the same rigor as WAF rules and IAM policies. Every new prompt pattern discovered in the wild was added to the adversarial test suite within 24 hours.

Interview Q&A — Testing Strategy

Q: How do you test a system where the main output (LLM responses) is non-deterministic?

• Easy: I separated deterministic logic (guardrails, intent regex, prompt builder, formatters) from non-deterministic logic (LLM responses). Deterministic parts got traditional unit tests with exact assertions. LLM outputs were evaluated with property-based checks — factual accuracy, relevance scores, hallucination rate — instead of string matching
• Medium: The key was the golden dataset approach. 500 curated prompts with expected properties rather than expected outputs. For each prompt, I defined what a correct response must contain (correct ASIN, accurate price, relevant genre) and what it must not contain (PII, fabricated prices, competitor mentions). An LLM-as-a-judge framework scored responses on relevance, accuracy, helpfulness, and tone. All metrics had to pass thresholds before a model promotion proceeded
• Hard: The hardest part was testing for regressions when we changed the prompt template. A small wording change in the system prompt could improve recommendations but degrade FAQ accuracy. I ran the full golden dataset against every prompt change and built a regression dashboard that showed per-intent score deltas. Any regression > 5% on any metric blocked the deploy. This prevented "whack-a-mole" prompt tuning where fixing one thing breaks another

Grill Follow-Up 1: "Property-based assertions are good for factual checks. But how do you test that a manga recommendation is relevant rather than just factually correct? A response could cite real ASINs and accurate prices but still recommend completely wrong genres."

The ASIN/price validation layer only checks structural correctness — it doesn't measure semantic fitness. For relevance, I used three additional layers: (1) Genre constraint checks: for a recommendation query tagged with a genre (e.g., "dark psychological"), required_elements included a genre classifier run on each recommended title. If the classifier returned < 0.75 confidence for the target genre, the recommendation was flagged. (2) LLM-as-judge rubric with a domain-specific dimension: "Would a fan of [seed title] find this recommendation genuinely relevant?" The judge was given the seed title, the user's stated preferences, and the recommended title. (3) A/B comparison against a vector-similarity baseline: if the LLM's recommendation had lower semantic similarity to the seed title than the top result from pure vector search, it was considered a quality regression.

Grill Follow-Up 2: "Isn't 'LLM judging LLM' circular? Both might share the same blind spots around niche manga titles."

Yes — the judge has the same gaps in niche manga domain knowledge. My mitigation: I split judgment into two strict tracks. Track 1 — factual claims — went through a deterministic catalog validator, never the judge. Track 2 — subjective quality (relevance, tone, helpfulness) — went to the judge only after passing catalog validation. For Track 1, "Is Eiichiro Oda the author of One Piece?" is verified by a catalog lookup, not the judge. The judge only scores "Is this recommendation helpful to someone who liked One Piece?" — which has no deterministic ground truth and is where LLM judgment is appropriate.

Grill Follow-Up 3: "What's your false negative rate? Hallucinations the judge missed?"

I measured this via human-vs-judge disagreement on a weekly sample of 100 responses. Judge-human agreement on "factual correctness" was κ=0.55 — moderate, meaning meaningful judge blind spots existed. Specifically: the judge missed ~8% of manga-specific factual errors (wrong volume counts, misattributed characters). This is why factual correctness was moved out of judge scope entirely. For subjective quality dimensions, judge-human agreement was κ=0.71 — better but still imperfect. The 29% disagreement cases were mostly borderline responses where humans and judge had different quality standards. I tracked this divergence as a "judge drift" metric and recalibrated the judge prompt quarterly when disagreement exceeded 30%.

Q: What was the most valuable test that caught a real bug?

• Easy: The contract test that caught the Catalog team renaming format to product_format. Without it, every product card in chat responses would have been broken
• Medium: The multi-turn E2E test that found a memory summarizer bug. When conversation memory exceeded 20 turns, the summarizer was stripping ASIN references from older turns. This meant "tell me more about the first one" failed for long conversations because the ASIN was lost. The fix was to preserve entities (ASINs, series names) as metadata alongside the summarized text
• Hard: The LLM evaluation pipeline that tracked hallucination rate. During early development, Claude was fabricating prices in ~6% of responses. Prompt engineering reduced it to ~3%, but the remaining cases were subtle (e.g., applying a real discount percentage to the wrong base price). Only the combination of runtime guardrail (price validator) + offline evaluation (hallucination metric) brought it under 0.5%. Neither approach alone was sufficient

Grill Follow-Up 1: "The memory summarizer bug — how did you design the test that caught it specifically?"

I parameterized the multi-turn test over n_turns values: [5, 10, 15, 20, 25]. At n_turns=19 (below the 20-turn summarization threshold), context was preserved correctly. At n_turns=21 (post-summarization), ASIN references were gone. By running both values in a single parameterized test, the boundary where the bug appeared was immediately visible. The key pattern: never test multi-turn context at only one conversation length. The summarizer boundary is a state change, and bugs live at state transitions.

Grill Follow-Up 2: "For the price hallucination — you say prompt engineering got it from 6% to 3% but couldn't go lower. What specifically was the remaining 3% doing?"

The remaining cases were compound hallucinations: the LLM would correctly reference the price from the PRICE_DATA section, then apply a real-but-inapplicable discount percentage. For example: it would correctly state $12.99 (from PRICE_DATA) and then say "with your Prime discount that's $10.39" — applying an unadvertised discount that didn't exist for that product. The prompt instruction "use only PRICE_DATA" was satisfied (it did use PRICE_DATA), but the LLM added arithmetic on top. Fix: I added a second instruction — "Do not perform price arithmetic. Do not apply discounts unless they appear explicitly in the PRICE_DATA section." Also added a runtime price arithmetic detector in the guardrails layer.

Grill Follow-Up 3 (Architect level): "Both the guardrail AND the offline eval metric were needed to bring hallucination below 0.5%. Why couldn't the offline eval alone drive it to 0%?"

Offline eval metrics measure the rate but don't fix the problem — they gate deploys when rates exceed thresholds, but they don't execute at inference time. If a user sends a query at runtime that matches a hallucination-prone pattern the prompt doesn't suppress, the offline eval can't help them — it only would have caught the problem in the next evaluation cycle. The runtime guardrail (price validator) is the only mechanism that catches hallucinations at the moment of user impact. The offline eval ensures you don't regress to higher rates when you change the system; the runtime guardrail ensures individual users aren't harmed even if a rare hallucination escapes the offline gate. These two operate on different timescales: offline eval is a batch quality gate; the runtime guardrail is a per-request safety net. You need both.

Q: How did you handle testing with 9 downstream service dependencies?

• Easy: I used WireMock to simulate downstream APIs for integration tests, and Pact for contract tests to protect against breaking changes from provider teams
• Medium: For each dependency, I built three test modes: (1) mocked (unit-level, WireMock), (2) local emulation (LocalStack for DynamoDB, test containers for Redis/OpenSearch), (3) real staging endpoints (SageMaker). The higher the test level, the more real the dependencies. Unit tests ran with mocks in < 30s. Integration tests ran with local emulation and staging endpoints in ~5 min. E2E tests hit the full staging environment in ~10 min
• Hard: The real challenge was testing failure modes — not happy paths. I needed to verify that when the Catalog API returned 504, the circuit breaker opened after 2 retries, the orchestrator returned a graceful fallback, and the correct metric was emitted. WireMock's scenario support let me program multi-step failure sequences (timeout, timeout, circuit open, recover) in a single test. This caught a bug where the circuit breaker was opening on non-idempotent requests, which could have caused duplicate order operations

Grill Follow-Up 1: "You said 'I never mocked SageMaker for integration tests because mocks hide real latency.' But using real staging endpoints makes your integration tests environment-dependent. How did you handle staging endpoint flakiness causing test failures?"

This was a real operational pain. My approach: (1) Retry policy in CI — integration tests retried once on timeout before failing. This handled transient staging flakiness without masking real issues (a retry-and-pass means transient; two consecutive failures means real regression). (2) Staging endpoint health check before integration test run — if the SageMaker staging endpoint was unhealthy, the integration test suite was skipped (not failed) with a notification. A skipped run was retried in 30 minutes. (3) Flakiness dashboard — I tracked which tests had > 5% historical flakiness rate. High-flakiness tests were investigated and either fixed or moved to a "known-flaky" tier that didn't block merges but was reviewed daily.

Grill Follow-Up 2: "Pact contract testing — you said it blocked the Catalog team from renaming a field. But Pact only works if the provider team actually runs the verification. What do you do when a provider team doesn't run Pact in their CI?"

Three enforcement mechanisms: (1) The Pact Broker had a webhook that posted failures to a shared Slack channel. Provider teams were on notice that a failing Pact verification was visible to their leadership. Social accountability. (2) I escalated to the dependency owner's tech lead with a concrete impact description: "If your deploy proceeds, every product card in MangaAssist will be broken for 500K daily active users." This usually got fast remediation. (3) For the 2 teams that repeatedly didn't run Pact verification, I implemented a defensive layer: at startup, MangaAssist hit each provider's staging API with a health check that validated the fields we depended on. If the field structure changed, the startup check failed and MangaAssist refused to start. This gave us a hard failsafe independent of whether the provider ran Pact.

Grill Follow-Up 3 (Architect level): "Contract testing protects you from provider schema changes. But what about behavioral contracts — a provider returns the right schema but wrong data? For example, the Catalog API returns in_stock: true for items that are actually out of stock. Does Pact catch this?"

No — Pact is a structural contract, not a behavioral contract. It validates that the schema is correct, not that the data is accurate. This is a known limitation of consumer-driven contract testing. For behavioral accuracy, I relied on: (1) E2E tests with real data — our E2E test catalog contained 100 known ASINs with known attributes. If in_stock was wrong for a known-in-stock ASIN in staging, the E2E test would fail. (2) Production monitoring — I tracked the ratio of "chatbot said in-stock" vs. "user found out-of-stock" via the add-to-cart failure rate. A spike in add-to-cart failures correlated with in-stock data accuracy issues. (3) For the worst case: real-time product availability was fetched at response time (not from RAG cache), specifically because cached availability data was a known risk. The cost was ~10ms per product lookup; the benefit was accurate availability data. Behavioral accuracy for availability was a design requirement, not just a test requirement.

See Also: 04-offline-testing-quality-strategies.md — Deep-dive on golden dataset design, hallucination testing, manga-specific adversarial cases, and multi-round interview grilling chains for offline testing.