Skip to content

Level-5 Assessment — Production Debugging

Realistic incident exercises. For each: read the Symptom, attempt your own investigation → root cause → fix, then reveal the model. Grade on 4 axes (symptom understood / investigation path / root cause / fix), 1 pt each. Checkpoint for Week 3. Pass = ≥ 3/4 average across attempted incidents. See grading guide.

General method: start at the queue, follow the state field, correlate by trace_id, check the cron's last run. See Request-LifeCycle §5.

Incident 1 — "The order never showed up"

Symptom. A brand says a Swiggy order placed 20 minutes ago isn't visible in Prism (no customer update, no digital bill).

Investigation → Root cause → Fix **Investigate** 1. Confirm ingestion happened: `db.crm_queue.find({/* order id / mobile */}).sort({insertedAt:-1})`. Is there a doc? Check `status`, `source`, `apiSource`. 2. If `status:0` and old → the **drain cron is behind or stuck**. Which ingestion cron owns that source? (uengage 1, petpooja 2, ls_center 3, orderPush 4, rista 6, posist 7). Check its `cron_stats`/`*_cron_status` last run. 3. If **no doc at all** → ingestion never landed. Check whether the source hits the Lambda (`orders-api` → Kafka) or the monolith route; look at API/Lambda logs by `trace_id`. Verify the ingestion token/middleware didn't 401. 4. Check `business_configs.last_uen_order_hit` / `last_pos_order_hit` to see when that outlet last successfully ingested. **Root cause (typical)** - Cron stuck (lock doc never reset) so `status:0` docs pile up; **or** the order hit a source whose consumer/route errored (bad payload, auth), **or** a dual-writer race dropped it. **Fix** - Clear/reset the stuck cron lock and let the drain catch up; for auth/payload errors, fix the token/parser and **re-enqueue** the raw payload. Long-term: idempotent upsert on a stable `source+orderId` key so re-drives are safe. See [Order-Ingestion](../03-Modules/Order-Ingestion/README.md), map-04 §1.

Incident 2 — Points discrepancy (E1.10)

Symptom. A customer's loyalty widget shows Total = 502 but Used + Available + Expired = 500. Support escalates a "2-point" mismatch.

Investigation → Root cause → Fix **Investigate** 1. Reconstruct the ledger from `wallet_transactions` (Mongo) **and** MySQL `wallet`/`wallet_history` — remember points live in **both** stores. 2. Check `crm_points_allocation_queue` for that customer: any `status='pending'`/`failed` rows, retry counts, `next_retry_at`? A half-applied allocation (MySQL updated, Mongo not, or vice-versa) is the classic cause. 3. Check whether the **legacy** MongoDB `loyalty_points_allocate` path (`pointAllocation.js`) *also* touched this customer — double-crediting across the two queues. 4. Look for a downgrade event (`loyalty_milestone_downgrade.js`) that forfeited points without a matching ledger entry. **Root cause** - **Non-atomic Mongo+MySQL writes** in `handleWalletPointsFlow.js` — a crash/partial failure left one store ahead of the other; or double allocation via the two coexisting queues. This is exactly the architecture risk from [Level-4 Q6](Level-4-Architecture.md). **Fix** - Immediate: run the reconciliation for that customer to bring both stores to the correct ledger sum; enforce `Total = Used + Available + Expired`. - Systemic: add **idempotency** (`order.points_allocated` flag), converge on the single MySQL queue (deprecate `loyalty_points_allocate`), and add a **daily reconciliation cron**. See [Loyalty](../03-Modules/Loyalty/README.md), map-04 §5.2/§5.7.

Incident 3 — Campaign "sent" but not delivered (low wallet)

Symptom. A marketer scheduled a WhatsApp campaign; the UI says "sent" but recipients report nothing, and delivery counts are near zero.

Investigation → Root cause → Fix **Investigate** 1. Inspect today's queue: `whatsapp_queue_YYYYMMDD` for that `campaign_id` — what `process_status`? (0 pending, 2 picked, 3 sent/pending-webhook, 4 skipped). Lots of `4=skipped` is a strong signal. 2. Check the **business wallet balance** — sends are **debited per message and blocked on insufficient balance**. Look at `business_configs.wallet_balance` and `wallet_transactions`. 3. Check opt-in/DND: WhatsApp requires `optin=1`; recipients without opt-in are skipped. 4. Check the send cron ran: `sendWhatsappMessageCron` / transactional cron `cron_status`; provider (Gupshup/Meta) errors in logs. **Root cause (this scenario)** - **Insufficient business wallet** → the builder queued messages but the send path skipped/blocked them (`process_status=4`), yet the campaign was marked "sent" at schedule time — a UI-vs-reality gap. (Opt-in gaps and provider throttling are common co-causes.) **Fix** - Top up the business wallet and re-queue eligible recipients; correct the UI to reflect actual delivered vs skipped. Systemic: pre-flight balance check before marking "sent," surface `skipped` reasons, and add per-business rate limiting. See [Campaigns](../03-Modules/Campaigns/README.md), map-04 §4.1/§5.10.

Incident 4 — AOV shows ₹0 (E1.9)

Symptom. In "Your Effective Offers," Average Order Value renders as ₹0 even though the offer clearly drove revenue and redemptions.

Investigation → Root cause → Fix **Investigate** 1. Find the widget's computation. AOV should be `Revenue / Orders` (or, in this widget, effectively `Revenue / Redemptions`). 2. Inspect the inputs feeding it — is `Revenue` populated but the **denominator** (`Redemptions`/`Orders`) zero or undefined at compute time? Or is Revenue itself 0 because cancelled/refunded orders were excluded and nothing remained? 3. Check the aggregation cron/MIS freshness (`aggregationProcess.js` / `dashboard_mis`) and whether the attribution window captured any orders. **Root cause** - The widget divides by a **redemptions/orders count that is zero or missing** (or divides in the wrong order / mislabels the field), producing ₹0 — a **logic bug**, tracked as **E1.9** (AOV in "Your Effective Offers" should be `Revenue / Redemptions`). **Fix** - Correct the formula to `Revenue / Redemptions` with a guard: if denominator is 0, show `—`/`N/A` (never divide-by-zero → 0). Add a data-validation test enforcing the metric per the Product Logic Spec (A4.1). Great **capstone bug** — see [Assignments](Assignments.md#assignment-4). See [Known-Bugs](../13-CommonIssues/Known-Bugs-From-Roadmap.md).

Incident 5 — Cron stuck / nightly numbers stale

Symptom. RFM segments and the dashboard haven't updated for 2 days; some customers are in the wrong segment and journeys aren't firing.

Investigation → Root cause → Fix **Investigate** 1. Check `cron_stats`/`*_cron_status` for `rfmSegments` and the aggregation crons — last successful run timestamp and the `status` flag. A flag stuck at `false`/in-progress means overlap protection is **blocking every subsequent run**. 2. Check the process/logs for an exception in the previous run that killed it *before* it reset the lock (locks are per-document, not lease-based). 3. Confirm it isn't horizontal-scaling double-run corruption (multiple instances racing the same flag). **Root cause** - A prior run **crashed mid-execution without releasing its lock**, so the overlap-protection check skips all new runs; reactive stuck-detection (~20 min alert, ~4 h force-close) either didn't cover this cron or the alert was missed. **Fix** - Manually reset the stuck lock document and re-run; verify `rfm_metrics_tmp` repopulates and segments update. Systemic: replace the boolean lock with an **atomic claim + lease/TTL** so a dead run auto-expires, and add a circuit-breaker/auto-recovery. See [Request-LifeCycle §4](../02-Architecture/Request-LifeCycle.md), map-04 §5.6, [Level-4 Q8](Level-4-Architecture.md).

Incident 6 — WhatsApp volume mismatch (E1.3)

Symptom. The WhatsApp volume report says 12,000 messages sent, but the provider (Gupshup/Meta) invoice shows ~15,000. Delivered counts also look low.

Investigation → Root cause → Fix **Investigate** 1. Count from the source of truth: `whatsapp_queue_YYYYMMDD` grouped by `process_status` for the period. Compare `2/3` (picked/sent) vs `4` (skipped). 2. Check whether reporting counts only one path — remember the **Kafka `whatsapp-messages`/`whatsapp-receipts` consumers are stubbed (log-only)**, so any messages/receipts flowing through that path aren't persisted/counted. 3. Look for a **delivery-receipt processor** — there isn't a visible one, so `process_status=3` "pending webhook" never advances to delivered → low delivered counts. 4. Consider per-day collections: the report may miss a date-partition (`whatsapp_queue_*` by day). **Root cause** - **Split, half-migrated pipeline**: the real send happens in the monolith crons while the Kafka consumers are stubbed, and there's **no receipt-reconciliation cron** — so counts computed from an incomplete set undercount/misattribute (E1.3). Per-day partitioning and no rate limiting compound it. **Fix** - Report from the authoritative queue across **all** date partitions; implement the delivery-receipt processor so `process_status=3 → delivered`; finish (or explicitly retire) the stubbed Kafka consumers so there's one counting path. See [High-Level-Architecture §7](../02-Architecture/High-Level-Architecture.md), map-04 §5.11.

Debugging cheat-sheet (memorize)

Symptom First place to look State field / signal
Order missing crm_queue status:0 stuck; source id; cron cron_stats
Points wrong crm_points_allocation_queue + MySQL wallet + Mongo wallet_transactions pending/failed + retry; two-queue double-credit
Campaign not delivered whatsapp_queue_* / sms_queue_* process_status (esp. 4 skipped); business wallet balance
AOV / metric = 0 widget formula + dashboard_mis zero denominator (E1.9)
Segments stale rfmSegments cron_stats lock stuck false
WhatsApp volume off whatsapp_queue_* all partitions stubbed Kafka consumers + no receipt cron (E1.3)