Common Issues & Runbooks¶
Top production issues as runbooks: symptom → root cause → diagnosis commands → resolution. Pair with the Debugging guide for the underlying tooling and the Known Bugs triage for the PM-acknowledged defects.
Validation legend: ✅ verified · 📄 inferred · ⚠️ known-bug link
RB-1 · Order not showing in dashboard¶
Symptom: merchant placed/received an order; it's not in Order 360 or dashboard counts.
Root causes: ingestion cron stuck; item still status:0 in crm_queue; dual-writer duplicate; aggregation MIS lag.
Diagnose:
// is it queued but unprocessed?
db.crm_queue.find({status:0}).sort({insertedAt:1}).limit(5)
db.crm_queue.aggregate([{$match:{status:0}},{$group:{_id:"$source",n:{$sum:1}}}])
db.cron_stats.find().sort({_id:-1}).limit(20) // ingestion cron heartbeat
status:0 backlog is growing → the ingestion cron is stuck. Check its *_cron_status flag; unstick per Debugging §5, restart if needed.
2. If the item processed but the dashboard is stale → aggregation lag; check dashboard_mis/daily_summary_* freshness (sync SLA: orders <5 min, A4.3).
3. If two near-identical docs exist (one apiSource:"lambda") → dual-writer duplicate (debt D1). Dedup and file against D1.
RB-2 · Loyalty points discrepancy¶
Symptom: customer's Total ≠ Used + Available + Expired (often off by ~2); points not credited. ⚠️ E1.10 / E1.6
Root cause: points ledger spans MongoDB + MySQL with no atomic transaction (debt D2); a mid-flow crash or fraud-rule block leaves stores inconsistent; failed queue rows.
Diagnose:
SELECT status, COUNT(*) FROM crm_points_allocation_queue GROUP BY status;
SELECT * FROM crm_points_allocation_queue WHERE status='failed' ORDER BY id DESC LIMIT 20;
SELECT * FROM crm_points_allocation_queue WHERE status='pending' AND retries>=5;
failed/exhausted rows after fixing the cause (fraud rule, missing customer).
2. Reconcile the wallet: recompute Available from wallet_transactions; correct the ledger.
3. Long-term: idempotency flag + single ledger source (D2). See E1.10 triage.
RB-3 · Campaign delivery failure / low wallet balance¶
Symptom: campaign shows sent but customers didn't receive; or campaign stalls. ⚠️ E1.5
Root causes: business wallet_balance too low (sends deduct wallet); send cron stuck; provider throttle; opt-in/DND filtering.
Diagnose:
db.business_configs.find({parent_business_id:"<id>"},{wallet_balance:1})
db.whatsapp_queue_20260703.countDocuments({process_status:0}) // stuck pending
db.sms_queue_20260703.countDocuments({processed:false})
db.whatsapp_cron_status.find({status:false}) // stuck cron
helpers.triggerWalletLowBalanceEmails).
2. Stuck send cron → unstick the *_cron_status flag (Debugging §5).
3. Provider throttle → back off; no rate limiter exists today (debt D18).
4. Recipients filtered by opt-in/DND is expected behavior, not a bug.
RB-4 · WhatsApp volume report mismatch¶
Symptom: monthly/volume WhatsApp report undercounts vs actual sends. ⚠️ E1.3
Root cause: delivery receipts are not persisted — messages stall at process_status=3 ("pending webhook") and the serverless consumer-receipts Lambda is stubbed (log-only) (debt D8). Reports built on delivered/read counts are therefore wrong.
Diagnose:
Resolve: short-term, countprocess_status IN (2,3) as sent for the report and annotate the gap. Long-term, implement a receipt processor + persist receipts (D8). See E1.3 triage.
RB-5 · Dashboard cross-widget integrity failure¶
Symptom: same metric differs between two widgets (e.g., orders count in KPI vs chart). ⚠️ E1.4 / A4.4
Root causes: two dashboard code paths (dashboardController legacy vs dashboardControllerNew, debt D4); widgets read different sources (live vs MIS); inconsistent period/outlet filters (A4.6).
Diagnose: identify which controller each widget hits; compare its query against live orders; check the period + outlet filter each widget applies; verify MIS freshness.
Resolve: 1. Point both widgets at the same aggregation source. 2. Enforce shared default period + outlet filter (roadmap A4.6). 3. Track toward single-dashboard migration (D4). Validate against A4.1-A4.5.
RB-6 · Cron stuck / overlap¶
Symptom: a periodic job stopped producing output, or you see double sends/double points.
Root cause: overlap guard left status:false after a crash; or a job with no guard ran twice (debt D22). Deploy restarts re-register all 91 crons (D7).
Diagnose:
db.checkin_cron_status.find({status:false})
db.whatsapp_cron_status.find({status:false})
db.cron_stats.find().sort({_id:-1}).limit(30)
pm2 logs crm).
2. Clear the stale lock (status:true or delete the lock doc); next tick resumes.
3. If it double-ran → check for missing idempotency; file against D22/D7.
RB-7 · AOV showing ₹0¶
Symptom: "Your Effective Offers" widget shows AOV = ₹0. ⚠️ E1.9
Root cause hypothesis: AOV is computed as Revenue / Redemptions in the effective-offers query; when Redemptions = 0 (or the numerator/denominator are swapped/null), the result renders ₹0 instead of a guarded fallback.
Diagnose: locate the effective-offers aggregation; inspect the division and whether redemptions can be 0/null; confirm the period filter isn't excluding all redemptions.
Resolve: guard the division (redemptions > 0 ? revenue/redemptions : 0-with-explicit-empty-state), verify the metric matches the Product Logic Spec (A4.1). Full analysis in E1.9 triage.
Escalation matrix¶
| Issue | Owner area | Debt ref | Bug ref |
|---|---|---|---|
| RB-1 order missing | Ingestion | D1, D7 | E1.4 |
| RB-2 points | Loyalty | D2, D6 | E1.10, E1.6 |
| RB-3 campaign | Campaigns | D18 | E1.5 |
| RB-4 WA report | WhatsApp/serverless | D8, D9 | E1.3 |
| RB-5 dashboard | Analytics | D4 | E1.4, A4.4 |
| RB-6 cron | Platform | D7, D22 | E1.7 |
| RB-7 AOV | Analytics | — | E1.9 |