LOCAL PREVIEW View on GitHub

Interview Q&A — Standardized Technical Components

Skill 1.1.3 | Task 1.1 — Analyze Requirements and Design GenAI Solutions | Domain 1

Scenario 1: Bedrock Client Config Drift

Opening Question

Q: During a Bedrock regional latency spike in us-east-1, Service A recovered in 45 seconds, Service B timed out immediately after 1 second, and Service C hung for 4 minutes burning Lambda concurrency. All three services call the same Bedrock model. What is the root cause and what is the architectural remediation?

Model Answer

This is Bedrock client configuration drift. Each service built its own boto3 Bedrock client independently, inheriting different timeout, retry, and connect settings — either from library defaults, copy-paste from different documentation versions, or engineer preference. Service A had a 30-second socket timeout with 3 retries and exponential backoff — it recovered. Service B had a 1-second timeout with no retry — it timed out instantly. Service C had a 300-second socket timeout with no circuit breaker — it held Lambda execution environments open through the entire latency event, exhausting concurrency. The remediation is a shared Bedrock client factory: a centralized package (internal Lambda layer or shared Python library) that encapsulates client creation with standardized timeout (connect=2s, read=60s), retry config (max_attempts=3, mode=adaptive), region behavior (primary + cross-region fallback), and observability hooks (emit bedrock.client.version dimension on every call). All services import the factory, none create a raw boto3 client directly. Client policy changes deploy once and propagate uniformly.

Follow-up 1: What the factory looks like in code

Q: Give me the key parameters of the shared Bedrock client factory. What is the minimum contract it must enforce? A: The factory must enforce: (1) Connect timeout: 2 seconds — how long to wait for TCP connection establishment. Anything longer holds the socket through infrastructure-level issues. (2) Read timeout: 60 seconds — maximum wait for Bedrock to return a response after connection. This is generous for streaming but prevents indefinite hangs. (3) Max retries: 3 with adaptive retry mode — the AWS SDK's adaptive mode backs off automatically when the service is under pressure. (4) Region: primary is us-east-1; factory accepts an allow_cross_region flag that activates a fallback to us-west-2 after 2 failed attempts. (5) Emitted metadata: on each client initialization, push a CloudWatch dimension bedrock_client_version=X.Y.Z and the effective timeout values. Services that bypass the factory can be detected via absence of this dimension. Block direct boto3.client('bedrock-runtime') calls using a Snyk/Checkov custom rule in CI.

Follow-up 2: Rolling out a client policy change

Q: You need to reduce the default read timeout from 60s to 30s to improve resource efficiency. Two services have legitimate P99 latency of 45 seconds for large-context calls. How do you roll this change out without breaking them? A: The factory must support per-task-type timeout overrides, not just a global default. Design: BedrockClientFactory.create(task_type='synthesis', region='us-east-1') — the factory checks a configuration map: synthesis task type gets read_timeout=60s (long-context synthesis), classification gets read_timeout=15s, embedding gets read_timeout=10s. This map lives in AppConfig so it can be tuned without code deployment. Roll out: (1) deploy the factory update with new per-task defaults; (2) Services A and B with synthesis workloads retain 60s via task_type configuration; (3) classification and embedding services get the improved 15s/10s defaults automatically; (4) emit timeout value as a CloudWatch dimension per call so the change's effect on timeout rates is immediately measurable.

Follow-up 3: Detecting services that bypass the factory

Q: How do you enforce that every service uses the shared factory and detect deviations? A: Two enforcement layers: (1) Static analysis: add a custom Checkov rule or Semgrep pattern to CI that fails any build containing boto3.client('bedrock-runtime') or boto3.Session().client(...) for Bedrock outside the factory package. This prevents new violations. (2) Runtime detection: every call through the factory emits a CloudWatch metric bedrock_client_version=X.Y.Z. Any Bedrock invocation in CloudWatch Logs that does not have a corresponding bedrock_client_version tag is either from a service bypassing the factory or from an old undeployed version. A daily Lambda runs CloudWatch Logs Insights query to find invocations without the factory dimension — those services are flagged for engineering review and required to migrate within one sprint.

Follow-up 4: Cross-region fallback behavior

Q: Service A uses the factory with cross-region fallback enabled. During the latency event, the factory falls back to us-west-2. What observability do you need? A: Emit a custom metric BedrockRegionFallback/Count with dimensions from_region, to_region, reason (timeout vs. error vs. throttle). Alert when fallback count exceeds 5 within a 5-minute window — that indicates a regional issue, not a transient spike, and triggers the runbook for cross-region mode. Log the fallback decision per request: {"event":"bedrock_cross_region_fallback","primary_region":"us-east-1","fallback_region":"us-west-2","latency_to_failure_ms":2100}. Additionally: when in fallback mode, route only new requests to us-west-2; do not abandon in-flight streaming connections already established to us-east-1 (those will complete or timeout on their own terms). Separate the fallback routing decision (new requests) from the in-flight request management (let them run to completion).

Grill 1: "Each service has different needs — a shared factory is too rigid"

Q: Service team lead says: "Our service has unique retry requirements — we can't use a shared factory with one-size-fits-all settings." How do you respond? A: The factory doesn't need to be one-size-fits-all — it needs to be the one place where all configuration decisions are made and exposed via a typed API. Valid differences (task type, timeout tier, cross-region flag) are parameterized in the factory interface — service teams pass these in, the factory applies the approved configuration for that combination. What isn't acceptable is teams creating raw clients and setting arbitrary values. The distinction: "I need a 90-second timeout for my batch embedding job" → valid, submit a PR to add an embedding_batch task type to the factory config. "I need a 90-second timeout and I'll just set it in my Lambda env var" → invalid, that's the configuration drift we're preventing. The factory gives teams control over legitimate variation while maintaining a single observable record of every configuration choice in production.

Grill 2: The factory itself has a bug that causes a memory leak

Q: The shared factory has a bug introduced in v2.1 that causes a 10MB memory leak per container startup. Now every Lambda using the factory is OOM-crashing. How does centralization make this worse and how do you mitigate it? A: Centralization gives you a single point of failure — acknowledged. The mitigation is the same discipline applied to the factory as to any shared library: (1) Semantic versioning — services pin to a specific version (factory==2.0.3), not >=2.0. A v2.1 release doesn't automatically update consumers. (2) Staged rollout — v2.1 deploys to dev → staging → canary (5% of production Lambda versions) → full production over 48 hours with OOM metric gate at each stage. (3) Rollback capability — the shared Lambda layer version is immutable; roll back means publishing a new layer pointing to v2.0.3. Any service can revert independently by updating its Lambda layer ARN. The centralization trade-off: one bug can be widespread, but one fix also deploys everywhere. The versioning and staged rollout discipline controls blast radius in both directions.

Grill 3: The factory is 200ms slower than a direct boto3 client

Q: Engineers complain the factory adds 200ms latency compared to creating a boto3 client directly. Is this acceptable? A: First, diagnose the 200ms. A properly implemented factory should add < 5ms: client creation is one line, initialization of retry config and headers is microseconds. A 200ms overhead suggests the factory is creating a new client per request instead of reusing a module-level singleton. Fix: initialize the client once at module level (Lambda init phase, not per-request). bedrock_client = BedrockClientFactory.create(...) at the top of the Lambda handler module, not inside the handler function. Client creation per request would add the cost of establishing TLS + DNS + connection pool initialization = 150–200ms each time. A singleton client reuses the connection pool across requests within the same Lambda execution environment — latency drops to < 5ms. This is a Python module initialization pattern issue, not an inherent factory cost.

Red Flags — Weak Answer Indicators

  • Proposing service-by-service timeout remediation without addressing the factory pattern
  • No mention of detecting services that bypass the factory via static analysis or runtime telemetry
  • Missing the connection pool singleton pattern (creating client per-request = 200ms overhead)
  • No staged rollout plan for factory version changes
  • Treating cross-region fallback as only a routing switch without observability for the fallback events

Strong Answer Indicators

  • Names the five mandatory parameters the factory must enforce with specific values
  • Proposes parameterized task-type overrides to handle legitimate per-service variation
  • Designs both static (Checkov rule) and runtime (CloudWatch dimension) factory compliance enforcement
  • Addresses the centralization failure mode (shared bug) with versioning + staged rollout
  • Identifies singleton creation at module level as the fix for the 200ms latency concern

Scenario 2: Prompt Template Drift Across Environments

Opening Question

Q: A MangaAssist user in production reported the chatbot gave inappropriate genre recommendations. QA reproduced the bug in staging, but dev works fine. Identical code is deployed to all three environments. What is the most likely root cause and how do you systematically find and fix it?

Model Answer

The most likely root cause is prompt template drift: the system prompt text in dev, staging, and production diverged — someone manually edited the prompt directly in staging or production config without going through the review process, and dev was never updated. Without prompt versioning, a deployed "code" change does not capture prompt content changes. The three environments have identical application code but different behavioral models because the system prompt is treated as runtime configuration without a deployment pipeline. Diagnosis: pull the active system prompt hash from each environment (log prompt hash on every Bedrock invocation as a structured log field), compare the three hashes — a mismatch confirms drift. Investigation: use Git blame or the config store's change history to find when production/staging diverged. Remediation: (1) establish a canonical prompt registry in S3 or DynamoDB with prompt_id, version, hash, content, deployed_envs; (2) require all prompt changes to go through a Git PR with safety team review; (3) CI/CD publishes only reviewed prompt versions to the runtime config store; (4) never allow direct manual edits to runtime prompt config outside the pipeline.

Follow-up 1: What to log for every Bedrock call for prompt traceability

Q: What logging metadata enables prompt-version forensics after an incident? A: Every Bedrock invocation log entry must include: prompt_id (the registry identifier), prompt_version (semantic version string), prompt_hash (SHA-256 of the compiled prompt text including all substitutions), environment, service_name, request_id, and timestamp. With these fields, a CloudWatch Logs Insights query can answer "what prompt version handled this request?" for any request ID from the past 90 days. Post-incident: query filter prompt_id="manga_system_v1" | stats count() by prompt_version, environment — this shows the exact distribution of prompt versions handling requests in each environment. A mismatch between environments is immediately visible without any additional tooling.

Follow-up 2: Drift detection without waiting for a user complaint

Q: How do you detect prompt drift proactively — before a user reports a problem? A: Three automated checks: (1) Scheduled drift audit: a daily Lambda pulls the active prompt hash from each environment's config store and compares them against the registry's latest_deployed_versions record. Any environment divergence fires a P3 alert with the affected prompt_id and the hash delta. (2) CI/CD deployment gate: on every code deploy, the pipeline reads the active prompt version from the target environment and verifies it matches what the registry says should be there (the last CI-deployed version). If the environment has a hash that isn't in the registry, the deploy is blocked and the incident noted. (3) Prompt hash dimension on answer-quality metric: if the LLMAsJudgeScore metric drops exclusively in one environment (not others), and those environments have different prompt hashes, the correlation is automatic without manual investigation.

Follow-up 3: Rollback process for a bad prompt

Q: A prompt change caused a safety regression — the chatbot started recommending adult content. Walk me through the rollback procedure. A: Rollback is a promotion of a previous version, not a deletion. Steps: (1) identify the last known-good prompt version from the registry changelog (takes < 60 seconds if the registry is properly kept); (2) trigger a pipeline run with that version as the target — it pushes the known-good prompt to all affected environments within 2 minutes; (3) emit a PromptRollback event to the incident management system with fields: from_version, to_version, environment, rollback_reason, triggered_by; (4) simultaneously increase Bedrock Guardrails sensitivity for the affected content category as a belt-and-suspenders measure while the root cause is investigated; (5) post-mortem: add the bad prompt pattern to the golden-set safety test suite so the same regression is caught in CI before it reaches production again.

Follow-up 4: Who can approve a prompt change for production?

Q: Define an approval workflow for prompt changes that is both rigorous and fast enough for a product team iterating weekly. A: Design a two-track approval workflow: (1) Standard track (most changes): PR opened by any engineer with these automated gates — golden-set CI pass (all 500 questions within threshold), token-count delta < 10%, safety category test suite 100% pass. Requires 1 engineering reviewer familiar with the prompt domain. Deploys to dev → staging → 10% prod canary → full prod over 72 hours. (2) Expedited track (urgent safety fix or P1 regression): direct approval from tech lead + safety officer, same CI gates but simultaneous staging+prod deploy, post-deploy monitoring for 2 hours. Track timing: standard PR merges in 1–3 business days; expedited in < 4 hours. Prompt changes that affect content policy phrases (what the chatbot will/won't discuss) always require the safety officer as co-approver regardless of track. Log all approvals in the registry with reviewer identity for compliance audit.

Grill 1: "Prompt changes are too frequent — putting them in Git slows us down"

Q: The product team iterates on the prompt 5–10 times per day during optimization sprints. They say the PR process kills their velocity. How do you balance governance with speed? A: Separate the experimentation environment from the deployment pipeline. During a prompt optimization sprint: (1) grant the product team read-write access to a sandbox environment with no governance gate — full freedom to iterate on the prompt, run live tests, evaluate quality. Sandbox has real Bedrock access but synthetic/anonymized data and no customer traffic. (2) When a candidate prompt passes quality evaluation in sandbox, it enters the standard PR track for dev → staging → production promotion. The governance gate catches the 1-in-10-or-20 change that introduces a regression or safety issue; the sandbox removes friction from the 9-in-10 experiments that don't. (3) Offer tooling: a CLI command that automates the PR creation with the golden-set test results pre-attached, reducing manual PR overhead to 2–3 minutes. Speed and governance are not in conflict if the pipeline tooling is good.

Grill 2: The prompt registry is compromised — an attacker modifies the production prompt

Q: Security audit finds the S3 prompt registry bucket has a misconfigured bucket policy allowing public write access. An attacker has injected a prompt that exfiltrates user session data. What are the detection and response steps? A: This is prompt injection at the infrastructure level — one of the most severe security incidents in a GenAI system. Immediate response: (1) Revoke S3 bucket public write access — aws s3api put-bucket-acl --bucket prompt-registry --acl private within 2 minutes; (2) Pull the current production prompt and compare against the last Git-committed version — detect unauthorized modifications; (3) Roll back to the known-good version via the pipeline (< 5 minutes); (4) Audit session data exposure: query CloudWatch Logs for responses in the window the injected prompt was active — identify whether PII was included in responses; (5) Rotate Bedrock IAM credentials used by the prompt registry service as a precaution. Prevention: (a) bucket public access block policy must be applied at account level; (b) S3 Object Lock for prompt versions (COMPLIANCE mode) — no writes without MFA authorization; © CloudTrail alert on any S3 put-object to the prompt registry bucket outside of the CI/CD service account.

Grill 3: The same prompt is 800 tokens in dev and 1,200 tokens in production after variable interpolation

Q: Your prompt uses template variables ({{catalog_context}}, {{user_preferences}}) that expand at runtime. The drift alert fires because the raw template hashes match but the effective prompts differ. How do you handle this? A: Template-level hash comparisons are insufficient — you need compiled prompt hashes. At deployment, the CI pipeline compiles the template with a standardized set of representative variable values (a fixed synthetic test context for hashing purposes) and stores the compiled hash alongside the template hash. The drift alert then compares compiled hashes, not template hashes. This detects cases where the template changed in a way that affects output (a variable reference renamed, a new template section added) even if the structural template text is similar. Additionally: for variable-length injected content (like {{catalog_context}}), the token budget enforcement must use the maximum possible expansion size in the budget calculation, not the average. Log both template hash and a sampled compiled hash per request to detect runtime variable expansion anomalies (e.g., a code change that passes catalog_context containing 10× more content than expected).

Red Flags — Weak Answer Indicators

  • Not logging prompt hash/version on every Bedrock invocation
  • No automated drift detection between environments — relying only on user complaints
  • Missing the S3 bucket security control as part of prompt registry design
  • Confusing template hash with compiled-prompt hash for drift detection
  • No formal rollback procedure — treating rollback as "just copy-paste the old text back"

Strong Answer Indicators

  • Immediately prescribes prompt hash + version as mandatory structured log fields
  • Designs a daily automated drift audit across all environments
  • Creates a two-track PR approval workflow (standard vs. expedited) balancing speed vs. governance
  • Addresses the sandbox-vs-governance problem with environment separation
  • Distinguishes template hash from compiled prompt hash, handles variable interpolation edge case

Scenario 3: CloudWatch Metric Naming Drift

Opening Question

Q: Three MangaAssist services each track Bedrock error rate. During a live incident, the on-call engineer opens the monitoring dashboard and sees two panels showing zero and one showing non-zero. Another alert says "namespace error" — the metric doesn't exist. The system is clearly broken but the dashboard says everything is fine in 2 out of 3 panels. What happened and how do you prevent this permanently?

Model Answer

This is metric naming drift. Each service independently named its error rate metric: Service A emits BedrockErrorCount, Service B emits bedrock_errors_total, Service C emits Bedrock/ErrorRate. Each went to a separate CloudWatch namespace or used different units (count vs. rate). The dashboard was built against one of these names and the others never appeared on it — they look like zero because the dashboard query returns no data points for the wrong metric name. The worst outcome: the panel showing "zero errors" is the most-watched panel, creating false confidence during an incident. The fix has two layers: (1) Immediate: define a canonical metrics catalog — a versioned document listing every required metric name, namespace, unit, dimensions, and description. Add a CI lint step that rejects any CloudWatch metric emission using a name not in the catalog. (2) Structural: create a shared metrics library (MangaAssistMetrics) that every service imports and calls for standard emissions — the library owns the canonical names, services pass values. No raw cloudwatch.put_metric_data() calls with string metric names in service code.

Follow-up 1: Defining the canonical metrics catalog

Q: What does the mandatory metrics catalog for a GenAI chatbot service look like? List the five most important entries. A: Five mandatory entries with full specification: (1) Namespace=MangaAssist/Bedrock, MetricName=InvocationCount, Unit=Count, Dimensions=[service_name, model_id, task_type, environment]; (2) Namespace=MangaAssist/Bedrock, MetricName=ThrottlingCount, Unit=Count, Dimensions=[service_name, model_id, environment]; (3) Namespace=MangaAssist/Retrieval, MetricName=ChunkCount, Unit=Count, Dimensions=[service_name, index_name, environment] — non-zero mean of this metric confirms RAG grounding; (4) Namespace=MangaAssist/Latency, MetricName=EndToEndMs, Unit=Milliseconds, Dimensions=[service_name, request_type, environment]; (5) Namespace=MangaAssist/Session, MetricName=SessionCount, Unit=Count, Dimensions=[state=active|expired, environment]. Each entry has a description, an owner team, and a freshness SLA (expected emission frequency).

Follow-up 2: Shared metrics library implementation

Q: The shared metrics library approach has the same centralization risk as the shared Bedrock client factory. How do you ensure the library doesn't become a bottleneck or a single point of failure? A: The metrics library is a thin wrapper, not a service. It has no network dependencies — it calls cloudwatch.put_metric_data() (which itself uses the AWS SDK with local buffering). Key design choices: (1) Async emission: all put_metric_data calls are fire-and-forget using a background thread or asyncio task — metrics emission never blocks the application hot path. A failed metric call is logged and discarded, never raising an exception to the caller. (2) Buffering: the library buffers up to 20 metrics per put_metric_data call (CloudWatch API maximum) and flushes every 1 second — reduces CloudWatch API calls by 20× vs. individual calls. (3) Versioned as a Lambda layer: patch releases (new metric names) are backward compatible. Minor releases (new required dimensions) follow the same staged rollout as the Bedrock client factory. A bug in the metrics library causes metric loss, not service failure — acceptable degradation.

Follow-up 3: Dashboard and alarm alignment with the catalog

Q: The catalog is defined but the dashboards were built before the catalog. How do you migrate dashboards and alarms to use the canonical names? A: Structured migration in three phases: (1) Audit phase (1 week): run a CloudWatch Logs Insights query across all custom namespaces to inventory every unique metric name currently being emitted. Produce a mapping table: old_name → canonical_name (or old_name → deprecated, no equivalent). (2) Dual-emit phase (2 weeks): update the shared metrics library to emit both the old name and the new canonical name simultaneously. This ensures dashboards and alarms on the old name continue to work during migration. (3) Cutover phase (1 week): rebuild all dashboards and alarms to use canonical names, verify they show data, verify alarms fire correctly in a test scenario, then disable dual-emission for the old names. Monitor for any zero-data panels for 48 hours post-cutover. The dual-emit phase is critical — never cut over dashboards and alarms without a parallel observation period.

Follow-up 4: What does an alarm on a missing metric look like?

Q: How do you create a CloudWatch alarm that fires when a service stops emitting metrics entirely — i.e., the metric itself disappears? A: CloudWatch supports this with the TreatMissingData=breaching alarm configuration. Set up: AlarmName=MangaAssist-BedrockInvocationCount-Missing, Namespace=MangaAssist/Bedrock, MetricName=InvocationCount, Statistic=Sum, Period=300, EvaluationPeriods=2, Threshold=0, ComparisonOperator=LessThanOrEqualToThreshold, TreatMissingData=breaching. This alarm fires when either (a) the metric reports 0 invocations for 10 consecutive minutes during business hours (possible service outage) or (b) the metric stops being reported entirely (service crashed or metrics library failing). Pair it with a time-of-day condition: suppress the alarm outside business hours (02:00–08:00 JST) to avoid false alerts during low-traffic windows. This "dead man's switch" alarm pattern is essential for catching silent failures where the system appears healthy simply because it stopped reporting.

Grill 1: "Dashboards are maintained by each team — we shouldn't centralize"

Q: Team leads push back: "Each service has unique metrics — a central catalog creates too much coordination overhead." How do you balance team autonomy with platform-level consistency? A: The catalog has two tiers: platform-mandatory and team-local. Platform-mandatory metrics (the 5 listed above) must use canonical names and are enforced by CI lint. These cover the cross-service, on-call, and SLA-relevant metrics that every operator needs to understand regardless of which service is on fire. Team-local metrics can use any naming, emit any dimensions, and are the team's own engineering workspace — no central approval required. The key insight: the platform-mandatory tier is small (5–10 metrics per service category) and changes infrequently. The team-local tier is large and iteration-friendly. The catalog doesn't eliminate team autonomy — it defines the small shared vocabulary that enables reliable cross-service monitoring and incident response.

Grill 2: A new service deploys with no metrics at all — it passes CI because there's no lint for metric presence

Q: A service ships to production without emitting any metrics. No alert fires. How do you detect and prevent this? A: The current lint only catches wrong names — it doesn't check for metric presence. Add an observability presence gate to the deployment pipeline: a post-deploy canary Lambda sends 50 synthetic requests to the new service and verifies at least 5 of the mandatory metrics appear in CloudWatch within 5 minutes. If mandatory metrics are absent at the end of the 5-minute window, the deployment is rolled back automatically. Additionally, add an IaC check: every deployed Lambda function or ECS task definition must include a CloudFormation/Terraform resource that declares a CloudWatch alarm for at least 2 platform-mandatory metrics. If the IaC review doesn't include alarm definitions, the PR is blocked. This infrastructure-as-code enforcement makes metric presence a deployment requirement, not a post-hoc audit.

Grill 3: A metric name change breaks an on-call team's personal dashboards

Q: You deprecate BedrockErrorCount in favor of ThrottlingCount. An on-call engineer's personal dashboard (not in the catalog) breaks. She files an incident saying "your metric change caused an incident." How do you respond? A: This is a governance gap, not an incident: personal dashboards outside the catalog are un-governed assets — deprecation warnings are not required for them. However, the response should be pragmatic: (1) Dual-emit the old name for 30 days after deprecation (which should already be policy — dual-emit phase above); (2) publish the deprecation schedule in team channels and the engineering wiki 2 weeks in advance, listing every affected metric and its replacement; (3) provide a migration script that updates any CloudWatch dashboard JSON replacing old metric name with canonical name; (4) offer one 30-minute office hour with each affected team. The broader lesson: any on-call tooling that isn't in the catalog is a personal asset subject to breakage. Teams should submit critical personal dashboards to the catalog for protection — or accept that un-catalogued tooling breaks without warning.

Red Flags — Weak Answer Indicators

  • Treating metric naming as a cosmetic/documentation issue rather than a reliability issue
  • No shared metrics library — just documentation telling teams to "use the right names"
  • Missing the TreatMissingData=breaching pattern for detecting metric silence
  • Dual-emit transition phase absent — cutting over dashboards and alarms simultaneously
  • No distinction between platform-mandatory and team-local metric tiers

Strong Answer Indicators

  • Designs a canonical metrics catalog with all 5 required fields per entry (namespace, name, unit, dimensions, description)
  • Implements a shared metrics library as a thin async Lambda layer
  • Proposes a three-phase migration (audit → dual-emit → cutover) with explicit validation between phases
  • Creates a "dead man's switch" alarm using TreatMissingData=breaching
  • Separates platform-mandatory (enforced) from team-local (autonomous) metric tiers

Scenario 4: Missing Standard Error Envelope

Opening Question

Q: A MangaAssist cart update Lambda returns {"error": "Bedrock timeout"} on failure while the session Lambda returns {"message": "Session expired", "code": 440} and the recommendation Lambda raises an unhandled exception that becomes a raw Python traceback in the HTTP 500 body. The frontend team says "we handle errors differently depending on which service fails." Walk me through why this is a serious architectural problem and how you standardize it.

Model Answer

Every response shape variation the frontend must handle is a latent bug. When the frontend engineer codes if response.get('error'): for one service and if response.status_code == 440: for another, they have introduced a service-coupling dependency in the client side that breaks whenever the backend changes. More critically, a raw Python traceback in a 500 body is a security vulnerability: it may expose stack traces, AWS ARN structure, internal variable names, or library versions that assist an attacker. The standard error envelope solves both problems. Every response — success, partial success, and error — from every MangaAssist backend service must conform to a single JSON schema: {"status": "success|error|partial", "request_id": "...", "data": {...} | null, "error": {"code": "THROTTLE|SESSION_EXPIRED|RETRIEVAL_FAILED|...", "message": "...", "retryable": true|false, "support_ref": "..."} | null}. This contract is shared between backend and frontend via a schema registry (JSON Schema file in the shared repo). Any backend that emits a non-conforming response shape fails its CI contract test.

Follow-up 1: Defining the error code taxonomy

Q: What error codes does a GenAI chatbot service need, and what is the retryable classification for each? A: Taxonomy (with retryable flag): THROTTLE (retryable=true, backoff before retry), MODEL_UNAVAILABLE (retryable=true after 30s, try alternate model), SESSION_EXPIRED (retryable=false, user must start new session), CONTEXT_TOO_LARGE (retryable=false, user must shorten input), RETRIEVAL_FAILED (retryable=true once, fallback to model-only if second failure), CONTENT_POLICY_VIOLATION (retryable=false, show user-appropriate message), INTERNAL_ERROR (retryable=false, show support reference), AUTH_FAILED (retryable=false, redirect to login). The frontend consumes retryable directly: if true, auto-retry after the specified backoff. If false, show the error message and stop. The support_ref field contains an opaque token that maps to the CloudWatch log entry — users can report support_ref=XY7Q2 and on-call can query the exact error chain within 30 seconds.

Follow-up 2: Preventing raw exceptions from reaching the envelope

Q: How do you ensure unhandled exceptions in a Lambda function never reach the HTTP response body as a raw traceback? A: Two layers: (1) Global exception handler in every Lambda: wrap the handler entry point in a try/except that catches Exception and calls a to_error_envelope(exception) function — this maps known exception types to error codes and wraps unknown exceptions as INTERNAL_ERROR with the request_id and a log reference, never the traceback text. The traceback goes to CloudWatch Logs (structured, within the service boundary). (2) API Gateway response mapping: configure a GatewayResponse for DEFAULT_5XX that overrides any raw Internal Server Error with a static JSON body matching the error envelope schema. This is a defense-in-depth layer — if the Lambda somehow returns a non-conforming body (e.g., Lambda runtime crash before handler runs), API Gateway still returns a standard envelope shape. The two layers ensure that nothing outside the standard shape can ever reach the frontend.

Follow-up 3: Contract testing the envelope

Q: How do you enforce the error envelope schema in CI? A: Add a contract test suite that runs in CI against every service's HTTP responses. Using a tool like Pact or a simple pytest fixture: (1) instantiate the service Lambda handler locally with a mocked event; (2) trigger each error condition (mock Bedrock to throw ThrottlingException, mock DynamoDB to throw ProvisionedThroughputExceededException, etc.); (3) validate the response body against the JSON Schema file using jsonschema.validate(). The test fails if: the error.code field is missing, the request_id field is missing, the retryable field is missing, or the body contains any key not defined in the schema (strict additionalProperties=false). This test suite runs in < 30 seconds and gates every PR. The JSON Schema file is the ground truth — backend and frontend tests both import it. Schema changes require a PR with both teams as required reviewers.

Follow-up 4: Error rate alerting by error code

Q: Now that every error has a typed code, how do you use this for monitoring? A: Publish a CloudWatch metric ErrorCount with dimension error_code for every error response emitted by any service. Alarm thresholds by code: THROTTLE → alarm at > 1% of requests over 5 minutes (indicates Bedrock quota issue); RETRIEVAL_FAILED → alarm at > 2% over 5 minutes (indicates OpenSearch health); AUTH_FAILED → alarm at > 0.5% sustained (indicates possible credential rotation issue or attack pattern); CONTENT_POLICY_VIOLATION → alarm at > 0.1% surge in 1 minute (indicates potential adversarial prompt injection campaign). The error_code dimension lets on-call know which component and which response path is degraded within seconds of opening the alarm, without digging through raw logs. Also add a monthly report of unique error_code + service_name combinations to the engineering retro — new error codes appearing in production without being in the taxonomy are a signal of ungoverned error paths.

Grill 1: "The envelope adds 200 bytes to every response — it's overhead"

Q: An engineer argues the standard envelope adds unnecessary payload size for a high-traffic chatbot. How do you address this? A: On a Bedrock response containing 500–2,000 output tokens, 200 bytes of envelope overhead is < 0.5% of the response size — negligible at any scale. The more important point: the engineer is optimizing the wrong variable. The cost of a non-standard error shape is an untested frontend code path, a production incident requiring cross-team debugging, a possible security disclosure via raw traceback, and a user-facing crash. The cost of the envelope is 200 bytes per response and 2 hours of one-time contract design work. This is a strongly asymmetric tradeoff. If response size is genuinely a concern for streaming responses, the envelope applies only to the terminal stream_end and error frames (200 bytes once per session), not to each 5-byte token chunk.

Grill 2: A new error type appears in production not in the taxonomy

Q: Bedrock releases a new error CapacityExceededException that none of your services were designed to handle. It propagates as an unstructured 500. How does your system handle this gracefully? A: The global exception handler's catch-all INTERNAL_ERROR wraps the unknown exception — the user gets a standard envelope (not a raw traceback), the request_id and log reference are included. The CloudWatch error_code=INTERNAL_ERROR counter spikes, which triggers an alarm. On-call opens the log: {"error":"CapacityExceededException","service":"bedrock-runtime"} in the CloudWatch entry linked by the support_ref. Action: (1) add CapacityExceededException to the exception-to-error-code mapping in the global handler package (maps to MODEL_UNAVAILABLE, retryable=true); (2) deploy the update; (3) add this new exception to the contract test suite so it's covered going forward. The architecture is resilient to unknown errors (graceful degradation via catch-all); the process makes it iteratively more specific over time.

Grill 3: Different clients (mobile app, web, third-party API consumer) need different error message text

Q: The web frontend needs verbose error messages for developers. The mobile app needs short user-facing copy. A third-party API consumer needs machine-readable codes only. All get the same envelope. How? A: The envelope is the canonical representation — all three variants are derived from it. Structure: the error.code is always the machine-readable canonical string (THROTTLE, SESSION_EXPIRED, etc.) — used by all clients for logic. The error.message is a developer-level description — sufficient for the web app's developer mode and the API consumer. The mobile app uses a message catalogue: a static lookup table (ship in the app bundle) mapping error.code → localized_user_string. The mobile app never displays error.message to end users — it looks up the user-appropriate string locally: THROTTLE → "The chatbot is busy. Tap to try again.". This means the envelope is client-agnostic and each client presents it appropriately. Adding a new language or changing user-facing copy is a mobile app update, not a backend change — the two concerns are correctly separated.

Red Flags — Weak Answer Indicators

  • Treating this as purely a frontend problem ("the frontend should handle different response shapes")
  • No mention of the security risk of raw Python tracebacks in HTTP 500 responses
  • Missing the API Gateway GatewayResponse defense-in-depth layer
  • No contract test suite enforcing the schema in CI
  • Error taxonomy has no retryable flag — clients must implement their own retry logic per error

Strong Answer Indicators

  • Immediately identifies raw tracebacks as a security vulnerability (OWASP A05: Security Misconfiguration)
  • Designs a complete error envelope schema with 5 required fields
  • Builds a typed error code taxonomy with explicit retryable boolean per code
  • Implements two-layer protection (Lambda global handler + API Gateway GatewayResponse)
  • Creates per-error-code CloudWatch alarm thresholds with incident-specific rationale

Scenario 5: IAM Role Pattern Drift

Opening Question

Q: Service A deploys to staging and gets AccessDenied on Bedrock. An engineer adds AmazonBedrockFullAccess to fix it. Six months later, a security audit finds 4 services with bedrock:* wildcard and 2 services with manually crafted policies missing bedrock:InvokeModel for the specific model ARN. New engineers copy whatever IAM exists in the closest service. Describe the architectural remedy.

Model Answer

This is IAM role pattern drift caused by the absence of a standardized, reusable IAM module. The "fix it with a broad managed policy" pattern is a classic urgency-driven security anti-pattern: functional, but it grants far more permission than needed (bedrock:CreateModelCustomizationJob, bedrock:DeleteFoundationModel, etc. alongside bedrock:InvokeModel). When engineers copy-paste IAM from nearby services, each copy inherits whatever over-permissions that service had. The remediation requires making the secure path the easiest path: create a reusable Terraform/CDK module bedrock_task_role that accepts parameters (allowed_models: list[str], allow_inline_agents: bool, allow_embeddings: bool) and generates a least-privilege policy with explicit ARN-level resource scoping. Every service uses this module. No service creates a raw IAM role or directly attaches AmazonBedrockFullAccess. CI enforces this via a Checkov custom rule that fails any Terraform plan containing wildcard Bedrock actions or the AmazonBedrockFullAccess managed policy ARN.

Follow-up 1: Writing the least-privilege Bedrock policy

Q: What does a least-privilege IAM policy for a service that invokes Claude 3 Sonnet and Haiku look like? A: Exact policy JSON: {"Version":"2012-10-17","Statement":[{"Sid":"BedrockInvoke","Effect":"Allow","Action":["bedrock:InvokeModel","bedrock:InvokeModelWithResponseStream"],"Resource":["arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0","arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0"]},{"Sid":"BedrockListForHealthCheck","Effect":"Allow","Action":["bedrock:ListFoundationModels"],"Resource":"*"}]}. Notably absent: any write actions (create/delete/update-model), any model customization actions, any knowledge base access unless required. The ListFoundationModels allows health check calls without invoking inference (cheaply validates permissions on startup). New model ARNs are added via a PR to the module's allowed_models default list — never ad hoc in the service's Terraform.

Follow-up 2: Automated IAM compliance scanning

Q: How do you scan existing IAM roles in production for wildcard and over-permissive Bedrock policies without waiting for the next deploy? A: Automated continuous compliance with AWS Config + custom rule: (1) deploy an AWS Config custom rule (Lambda) that fires on AWS::IAM::Role resource configuration changes; (2) the rule uses iam.simulate_principal_policy to check if the role can execute any action outside the approved list against any resource (*); (3) roles with wildcard Bedrock permissions are marked NON_COMPLIANT; (4) an AWS Config remediation action sends a Slack notification + creates a Jira ticket with the violating role ARN, current policy, and a suggested least-privilege replacement. Run the Config rule evaluation daily for all existing roles, not just newly created ones. Additionally, run aws iam get-account-authorization-details in a weekly Lambda, filter for roles with bedrock:* or AmazonBedrockFullAccess, and publish a violation count to CloudWatch — this tracks the remediation progress over time from the initial 6 violations toward 0.

Follow-up 3: IAM drift between environments

Q: The prod role is correctly scoped, but staging uses AmazonBedrockFullAccess. An engineer tests a new model in staging with the broad role, finds it works, then realizes the prod role doesn't have the new model ARN. How do you prevent this class of environment divergence? A: The IaC module is the single source of truth — staging and production use the same module, same parameters. The only Terraform variable difference between environments is allowed values: prod allows anthropic.claude-3-haiku-* and anthropic.claude-3-sonnet-*; a new_model=true flag in staging allows an additional model ARN for evaluation, but it's explicitly parameterized, not achieved by swapping to a broad policy. The process for adding a new model: (1) open a PR to the IaC module adding the new model ARN to allowed_models for staging only; (2) test in staging; (3) open a second PR adding the ARN to allowed_models for production; (4) deploy via the standard pipeline. Each step is a code review and tracked in Git. The engineer can never say "it worked in staging but not in prod" without a clear explanation — either the module parameters differ intentionally (documented in the PR) or there's an unintended divergence (flagged by the CI parity check).

Follow-up 4: Startup validation of active IAM permissions

Q: How do you detect at runtime that a deployed Lambda has incorrect IAM permissions before the first user request fails? A: Add a startup validation check to the Lambda initialization phase (not the handler — the module-level code that runs during Lambda init): call sts.get_caller_identity() to confirm the role is loaded, then call bedrock.list_foundation_models() (read-only, no inference cost) to confirm Bedrock connectivity and basic permissions. If this call raises AccessDenied, the Lambda logs {"event":"iam_validation_failed","action":"bedrock:ListFoundationModels","role_arn":"..."} and raises a LambdaInitException which causes the Lambda to fail fast before entering the warm pool. CloudWatch picks up the init failure as a Lambda.InitError metric — alarm on any Lambda.InitError for the service name. This means IAM errors surface within 60 seconds of a bad deploy, during the deployment pipeline's smoke-test phase — not 10 minutes into production traffic when users start failing.

Grill 1: The IaC module itself has a bug that grants excessive permissions

Q: The shared bedrock_task_role module had a bug in version 1.3 that added an extra bedrock:* deny-override stanza that was inadvertently overridden by an allow. 40 services used it. How do you detect and remediate? A: AWS Config NON_COMPLIANT flag catches the wildcard within 24 hours of next config evaluation (or immediately if the rule runs on the deploy event). The remediation path: (1) release bedrock_task_role v1.4 with the bug fixed; (2) because all 40 services pin to a module version in Terraform, send a security advisory requiring all services to update to v1.4 within 48 hours; (3) services that don't update within the window get their roles temporarily restricted via an SCP (Service Control Policy) at the OU level that denies bedrock:* calls using the affected module version identifier tag. The SCP is a forceful lever — it's not typical, but a wildcard policy bug across 40 services warrants it. Post-remediation: add a module integration test that uses iam.simulate_principal_policy to verify the generated policy cannot execute actions outside the approved list before the module version is published.

Grill 2: Least privilege breaks a new feature development cycle

Q: A developer needs bedrock:CreateModelCustomizationJob to test fine-tuning in staging. The IaC module doesn't include it. The feature is due in 3 days. What happens? A: The IaC module is parametric — allow_fine_tuning=true is a module parameter that can be passed for the staging environment. The developer opens a 30-minute PR: (1) adds allow_fine_tuning parameter to the module with a boolean guard on the fine-tuning actions; (2) updates the staging Terraform manifest to pass allow_fine_tuning=true. The PR is reviewed by the security-aware engineer assigned to the module — review turnaround is same-day (not blocking because there's an expedited track for feature work). The production manifest passes allow_fine_tuning=false by default. This design achieves least privilege in production while enabling development velocity in staging. The critical property: the parameter and its production default are visible in code, not hidden in a one-off IAM console change.

Grill 3: An engineer bypasses the module via the AWS console to unblock production

Q: A P1 incident at 2 AM: the IAM role is missing a permission and causes production failures. The on-call engineer adds the permission directly in the IAM console to restore service. Now the Terraform state is out of sync with production. What is the process? A: The console change is acceptable as an emergency mitigation — restoring service takes priority over infrastructure discipline. The discipline comes immediately after: (1) the on-call engineer records the exact change made (action added, resource, condition) in the incident ticket before doing anything else; (2) within 4 hours of the incident, a PR is raised to the IaC module or service Terraform applying the same change as code — reviewed and merged before the on-call shift ends; (3) on the next business day, terraform plan is run against production to verify the state matches; (4) if there's remaining drift, a terraform apply -target=aws_iam_role_policy.X aligns the state. The console change is never the final state — it's a bookmark for the IaC change. Add a post-incident rule: any console-applied IAM change that is not codified in IaC within 24 hours triggers a P2 follow-up ticket.

Red Flags — Weak Answer Indicators

  • Accepting AmazonBedrockFullAccess as a pragmatic fix even temporarily without an IaC follow-up plan
  • No resource-level scoping in IAM (wildcard resource * for Bedrock actions)
  • Missing the startup validation pattern — IAM errors surface only under user traffic
  • No automated compliance scanning (Config rule, Checkov) — relying only on periodic security reviews
  • No IaC module strategy — telling teams to "write least-privilege policies" without a reusable template

Strong Answer Indicators

  • Writes exact least-privilege IAM policy JSON with specific ARN-level model resource constraints
  • Designs a parametric bedrock_task_role IaC module with typed boolean flags for capability expansion
  • Implements both pre-deploy (Checkov CI) and post-deploy (Config rule) detection layers
  • Adds startup Lambda permission validation with bedrock:ListFoundationModels probe
  • Addresses emergency console bypass with a formalized IaC codification SLA within 24 hours