Debugging Guide¶

APM is disabled (Elastic commented in index.js:1-13; New Relic present but inactive). Logs are the primary observability tool. This guide makes the logs actually useful.

Validation legend: ✅ verified · 📄 inferred · 💻 config · ⚠️ gotcha · ⚰️ disabled

1. The logging stack¶

Selector: logger/index.js → dev-logger when NODE_ENV=dev, else prod-logger. ✅

	Dev (`dev-logger.js`)	Prod (`prod-logger.js`)
Console	colorized	colorized
Combined file	`combined-%DATE%.log`	`combined-%DATE%.log`
Error file	`error-%DATE%.log`	`error-%DATE%.log`
Retention (`maxFiles`)	1d ✅	14d ✅
Timestamp	`YYYY-MM-DD HH:mm:ss.SSS`	`YYYY-MM-DD HH:mm:ss`
Zipped archive	yes	yes

⚠️ Dev keeps only 1 day of logs. If you're reproducing locally, grab the log before the next rotation. Prod keeps 14 days — anything older is gone, so triage promptly.

Basic usage:

const logger = require('./logger');
logger.info('msg'); logger.error(err); logger.warn('...');

Context-aware logger (trace-correlated): logger/logger-service.js

const loggerService = require('./logger/logger-service');
const log = loggerService.get('service-name', 'component-name');
log.info('message', { key: 'value' });   // trace_id auto-injected

2. trace_id correlation (the single most useful tool)¶

traceIdMiddleware (wired at index.js:32) uses AsyncLocalStorage to attach one trace_id per request. It reads the x-trace-id header or generates a UUID v4. Every loggerService log inside that request carries the same trace_id.

Grep a whole request across the log:

# find the trace_id from the failing response / access log, then:
grep '"trace_id":"<uuid>"' /path/to/combined-$(date +%F).log

# following a request through gzipped rotations:
zgrep '<uuid>' combined-*.log.gz

# pass your own id from a client to make a request traceable:
curl -H 'x-trace-id: DEBUG-hani-001' https://.../crm_api/... 
grep DEBUG-hani-001 combined-*.log

⚠️ Only code paths that use loggerService.get(...) emit trace_id. Older code using the plain Winston logger won't — those lines you correlate by timestamp + business_id.

3. MySQL pool health — `pool_stats`¶

utilities/poolMonitor.js emits a JSON line every ~60s (POOL_STATS_INTERVAL_MS) for each pool (web limit 15, jobs limit 10):

{"tag":"pool_stats","pool":"web","limit":15,"open":12,"inUse":11,"waiting":9,"acquired":..,"enqueued":..,"created":..}

grep pool_stats combined-$(date +%F).log | tail -50

Read it like this: | Signal | Meaning | Action | |--------|---------|--------| | inUse ≈ limit sustained | pool saturated | slow queries or a leak (connection not released) | | waiting > 0 growing | requests queuing for a connection | API latency spike incoming; find the slow query | | jobs pool saturated | a cron is hogging connections | check which cron is running; crons use jobsPool to protect web | | queueLimit: 0 ⚠️ | unbounded wait queue | under load this grows memory instead of failing fast |

4. Inspecting stuck queue items¶

crm_queue (order ingestion)¶

Orders land with status:0 (pending). If they never flip to processed:

// mongosh — pending backlog by source
db.crm_queue.aggregate([{$match:{status:0}},{$group:{_id:"$source",n:{$sum:1}}}])
// oldest pending item
db.crm_queue.find({status:0}).sort({insertedAt:1}).limit(1)

Source map: 1=uengage, 2=petpooja, 3=ls_center, 4=orderPush, 6=rista, 7=posist. A growing status:0 backlog = the ingestion cron is stuck or erroring (see §5). ⚠️ Remember the dual writer — items with apiSource:"lambda" came from prism-services.

crm_points_allocation_queue (MySQL — loyalty)¶

SELECT status, COUNT(*) FROM crm_points_allocation_queue GROUP BY status;      -- pending/done/failed
SELECT * FROM crm_points_allocation_queue
 WHERE status='pending' AND retries >= 5;                                       -- exhausted retries (max 5, 15-min backoff)
SELECT * FROM crm_points_allocation_queue WHERE status='failed' ORDER BY id DESC LIMIT 20;

failed rows or retries>=5 stuck rows are the usual cause of points-not-credited complaints (bug E1.6).

Channel send queues (date-suffixed)¶

db.whatsapp_queue_20260703.countDocuments({process_status:0})   // 0=pending,2=picked,3=pending-webhook,4=skipped
db.sms_queue_20260703.countDocuments({processed:false})
db.push_noti_queue_20260703.countDocuments({processed:false})

⚠️ You must query today's / tomorrow's dated collection — sends are enqueued into _YYYYMMDD named by execution date.

5. Reading cron health¶

Crons write status/heartbeat into *_cron_status / cron_stats collections and skip if the previous run isn't marked complete (overlap guard):

db.cron_stats.find().sort({_id:-1}).limit(20)
db.checkin_cron_status.find({status:false})     // false = a run started but never finished (stuck)
db.whatsapp_cron_status.find({status:false})
db.sms_cron_status.find(); db.push_cron_status.find()

- status:false lingering > 20 min ⚠️ = stuck cron (an alert email fires, but there is no auto-recovery except dedupCustomers' 4-hour force-close, map-04 §5.6). - To unstick: confirm no live process is running it, then flip the flag status:true (or delete the lock doc) and let the next tick pick up.

6. Symptom → where to look¶

Symptom	First look	Then	Likely root cause
Order missing from dashboard	`crm_queue` `{status:0}` backlog by source	ingestion cron heartbeat in `cron_stats`; `error-*.log`	ingestion cron stuck / dual-writer dup / aggregation lag (E1.4)
Points not credited	`crm_points_allocation_queue` status=`failed`/`retries>=5`	`handleWalletPointsFlow` errors in log by business_id	non-atomic Mongo+MySQL write, fraud-rule block (E1.6, E1.10)
Campaign not delivered	`_queue_YYYYMMDD` pending counts + `_cron_status`	`business_configs.wallet_balance`; provider error in `whatsappMsgProcess` logs	low wallet, stuck send cron, provider throttle (E1.5)
Dashboard number wrong	aggregation collections vs live `orders`	`dashboard_mis` / `daily_summary_*` freshness; which controller (legacy vs New)	stale MIS aggregation, cross-widget mismatch (E1.4, A4.4)
Segment stale	`segment_counts` / `rfm_metrics_tmp` last-write time	`rfm_job_log`; RFM/segment cron heartbeat	nightly RFM/segment cron didn't run (2-3 AM crons)
AOV shows ₹0	the effective-offers query	division `Revenue / Redemptions` with 0 redemptions	E1.9 — see Known Bugs
WhatsApp report mismatch	receipts pipeline	`process_status=3` stuck; stubbed `consumer-receipts` λ	E1.3 — no receipt processor

7. Quick command cheat-sheet¶

# tail live errors
tail -f error-$(date +%F).log
# all logs for one business today
grep '"business_id":"12345"' combined-$(date +%F).log
# pool saturation over the day
grep pool_stats combined-$(date +%F).log | grep '"pool":"web"'
# PM2 (monolith)
pm2 status; pm2 logs crm --err --lines 200
# serverless tier
aws logs tail /aws/lambda/prism-prod-orders-consumer --follow --region ap-south-1

Third-Party Integrations · Deployment · Queues · Common Issues · Database · Known Bugs