Intent Classification And Entity Resolution

Covers Q6, Q7, Q21, Q31, Q32.

What The Interviewer Is Testing

• Whether you understand the cost and latency reasons for a two-stage classifier.
• Whether you can explain confidence thresholds instead of treating model output as truth.
• Whether you can handle ambiguity, model drift, and bad deployments operationally.

Deep Dive

Two-Stage Classifier Logic

• Stage 1 is a cheap rule-based pre-filter for obvious cases.
• Stage 2 is a heavier BERT-style model used only when Stage 1 confidence is low.
• The confidence threshold is a product choice, not a magic constant.

The strongest answers explain what the threshold protects:

• Cost, because not every request should hit SageMaker.
• Latency, because obvious intents should resolve in a few milliseconds.
• Stability, because fallback models should only handle the ambiguous tail.

Entity Extraction

Intent classification decides what workflow to execute. Entity extraction tells that workflow what the user actually referred to.

Useful entity dimensions in this system:

• series_name
• volume_number
• attribute
• asin
• optional language or edition signals

Ambiguity Resolution Ladder

1. Use page context first.
2. Use conversation history next.
3. Use ranking heuristics such as exact match and popularity.
4. If confidence is still low, ask a clarifying question instead of forcing a wrong resolution.

Model Drift And Bad Deployments

You should talk about the full operating loop:

• Confidence distribution monitoring.
• Human labeling of low-confidence or high-disagreement samples.
• Shadow evaluation before full rollout.
• Canary deployment with rollback thresholds.
• Periodic retraining or active-learning loop.

Strong Answer Pattern

• "The classifier is a routing aid, not the sole source of truth."
• "Confidence should drive escalation, fallback, or clarification."
• "Ambiguous entities are better handled with a clarification turn than a silent wrong guess."
• "Operational controls matter as much as model architecture."

Scenario 1: Ambiguous Entity Resolution

Primary Prompt

The user says, "Do you have Attack on Titan volume 1?" The catalog contains manga, anime artbooks, and collector editions. How do you resolve the entity?

Follow-Up 1

What if the user is already browsing a manga detail page? Does that change your ranking logic?

Follow-Up 2

What if the classifier says intent is faq with 0.54 confidence and the entity extractor finds a strong series_name match?

Follow-Up 3

At what confidence level would you ask a clarifying question, and what would that question look like?

Strong Answer Markers

• Uses PageContext and history before fuzzy matching.
• Explains the separation between intent confidence and entity confidence.
• Avoids hard-coding a single answer for ambiguous names.
• Gives a concise clarification prompt.

Scenario 2: Model Drift Over Time

Primary Prompt

Six months after launch, misroutes have increased for multi-intent questions. How do you detect and fix this?

Follow-Up 1

What production metrics tell you this is classifier drift and not just a seasonal query mix shift?

Follow-Up 2

How would you collect retraining data without waiting for a full annotation project?

Follow-Up 3

How do you deploy a new classifier safely?

Strong Answer Markers

• Looks at downstream outcome metrics, not only model confidence.
• Uses human review on low-confidence or high-impact samples.
• Mentions shadow mode, canarying, and rollback.
• Understands that drift can be data drift, label drift, or policy drift.

Scenario 3: Bad Model Deployment

Primary Prompt

After a new SageMaker model release, 10 percent of requests are clearly misclassified. What is your immediate response?

Follow-Up 1

How do you detect it quickly if average confidence still looks normal?

Follow-Up 2

What is your killswitch or rollback path?

Follow-Up 3

How do you prevent this from happening again?

Strong Answer Markers

• Uses business outcomes such as escalation rate, thumbs-down rate, and resolution rate.
• Rolls back based on canary thresholds, not gut feel.
• Keeps the rule-based stage as a degraded fallback.
• Adds golden-set regression tests before future promotions.

Red Flags

• Treating confidence as calibrated truth without validation.
• Ignoring clarification as a valid answer.
• Saying drift is solved by "retraining sometimes" with no trigger condition.
• Failing to connect intent errors to downstream user harm.

Two-Minute Whiteboard Version

Draw a fork:

1. Rule-based fast path for obvious intents.
2. ML fallback for ambiguous requests.
3. Clarification path when neither intent nor entity confidence is sufficient.