Customer Intelligence — RFM & Deep Filtration¶

Validation legend: ✅ verified in code · 📄 Excel-only (product spec, not yet in code) · 💻 code-only (in code, not in spec) · ♻️ duplicate/inconsistent · ⚰️ dead code · ⚠️ risk

1. Overview¶

Customer Intelligence is the "Understand" layer of Prism (see Product Overview §1). It turns the unified Customer 360 built by Customer-CRM into actionable segments, in two complementary engines:

RFM segmentation — a nightly batch that scores every customer on Recency, Frequency, Monetary and assigns them to a lifecycle segment (Champions → Lost). This is the automatic intelligence layer.
Deep Layered Filtration ("37 filters") — an on-demand, async segment-job engine that lets a marketer build an arbitrary audience (visit recency, spend ranges, RFM segment, outlet, city, SKU/category, day/time preference) and export it or hand it off to Campaigns. This is the manual intelligence layer.

Why it exists: Prism's core value proposition is "filter to action in seconds" — turning fragmented order data into a targetable audience. RFM answers "who is slipping and who is a champion?" automatically; the 37 filters answer "give me exactly this cohort" on demand.

Business importance: every downstream retention lever — Campaigns, Automated Journeys, Loyalty targeting — consumes segments produced here. Per the Excel roadmap, Customer 360 + Customer Insight have been merged into one "Customer Intelligence" module (Roadmap A1.2, Done).

⚠️ Headline discrepancy — read this first. The PM Excel specifies an 11-segment model driven by an R × FM lookup grid (Champions, Loyal, Potential Loyalist, New, Promising, Needs Attention, About to Sleep, At Risk, Can't Lose, Hibernating, Lost). The shipped code implements only 7 segments via a simplified if/else on R and FM (constants/rfmSegments.js + crons/rfmSegments.js::getSegment). Both are documented below and the gap is called out explicitly. Treat the 11-segment grid as 📄 Excel-only and the 7-segment model as ✅ code truth.

2. Business Flow¶

2.1 The retention ladder (product view)¶

Every guest sits somewhere on a lifecycle ladder. Prism's whole job is to push guests up and win back those sliding down. Each segment carries a prescribed marketing action (Excel Customer Intelligence 2.x):

Segment (Excel, 📄)	Meaning	Prescribed action
Champions	Recent, frequent, high spend	Exclusive offers, VIP, referral program
Loyal	Consistent regulars	Upsell, cross-sell, loyalty rewards
Potential Loyalist	Recent + growing frequency	Frequency rewards, loyalty enrolment
New	First 1–2 orders, very recent	Welcome offer, encourage 2nd visit
Promising	Recent but low engagement	Category offers, menu highlights
Needs Attention	Above-average, starting to slip	Targeted re-engagement
About to Sleep	Activity declining fast	Urgent limited-time offer / FOMO
At Risk	Was valuable, now fading	High-priority win-back
Can't Lose	Was a top customer, going silent	Aggressive win-back, personal outreach
Hibernating	Inactive, low historical value	Low-cost reactivation or accept churn
Lost	Gone, lowest scores	Hail-mary offer OR clean from lists

2.2 Example user journey (Marketing Manager)¶

Marketer opens the Customer Intelligence tab, wants "customers who spent > ₹2,000 lifetime, visited in the last 30 days, in Mumbai, who like Biryani".
They configure filters and hit Submit → POST /crm_api/customer/insights/submit creates an async segment job (segment_jobs, status pending). ✅
customerInsightsCron (every minute, picks 3 oldest pending jobs) computes the audience and writes matching mobiles to segment_results and a count to segment_counts. ✅
Marketer polls POST /customer/insights/status, sees an estimated pick / completion time (queue-position math: 3 jobs/min). ✅
On completion they export CSV (/customer/insights/export) or (📄 roadmap B1.2) send a campaign to that audience without leaving the screen.
Separately, overnight, rfmSegments recomputes the RFM segment of every customer so RFM-based filters and the segment-count dashboard are fresh next morning. ✅

2.3 Edge cases¶

Never-ordered customers → forced to recencyDays = 999999 → R = 1 → segment lost. ✅ (Excel 1.1 also mandates this.)
Clustered data → some quintiles (e.g. F=2, F=4) may not appear; quintile cutoffs are relative to each brand's own base (📄 1.2, and code uses $bucketAuto-style percentile boundaries ✅).
Placeholder mobiles (0000000000, 1111111111, …, all-repeated digits, or outside 6000000000–9999999999) are excluded from RFM. ✅
Cancelled/refunded orders: the RFM cron aggregates orders without an explicit cancelled-status filter ⚠️ — see §9 Business Rules. The Excel business rule says cancelled orders must be excluded from revenue (📄), so this is a spec/code gap.
RFM not enabled: the cron only processes businesses with business_configs.rfm_enabled: true. A brand must request enablement (/rfm/request/enable). ✅

3. Technical Flow¶

3.1 RFM nightly batch (automatic)¶

crons/rfmSegments.js  (cron: "30 0 * * *", 00:30 IST)   ✅
  └─ getEnabledBusinesses()               → business_configs {rfm_enabled:true}
     for each parent business:
       ├─ startJobLog()                     → rfm_job_log (status: started)
       ├─ buildCustomerBaseTemp()           → rfm_metrics_tmp {recencyDays, lto}   (never-ordered = 999999)
       ├─ buildOrderWindowMetricsTemp()     → orders aggregate within [now-window_days, now]
       │                                       → frequencyOrders (count), monetaryValue (sum amount)
       ├─ getBucketBounds() × R,F,M         → percentile boundaries (with end-spike / start-spike fixes)
       ├─ updateCustomersAndBuildSegmentCounts()
       │     r = scoreFromBounds(recency, …, higherIsBetter=false)   // lower days ⇒ higher R
       │     f = scoreFromBounds(freq,    …, higherIsBetter=true)
       │     m = scoreFromBounds(money,   …, higherIsBetter=true)
       │     fm = Math.ceil((f+m)/2)                                  ✅ matches Excel 1.4
       │     sg = getSegment(r, fm, lto)                              // 7-segment if/else
       │     → customers.bulkWrite: set rfm{r,f,m,fm,sc,sg,wd,ca}     ✅
       ├─ updateBusinessAggregator()        → business_aggreator {rfm_segment_count, rfm_updated_at}
       ├─ endJobLog(status: completed|failed)
       └─ cleanupTempRun()                  → rfm_metrics_tmp.deleteMany({run_id})

The exact 7-segment mapping (crons/rfmSegments.js::getSegment, ✅):

function getSegment(r, fm, lto) {
    if (r === 5 && lto === 1) return "new_customers"; // first-time, recent
    if (r >= 4) {
        if (fm === 1) return "promising";
        if (fm <= 3) return "regular";
        return "champions";        // fm 4-5
    }
    if (r === 3) return "need_attention";
    if (r === 2) return "at_risk";
    return "lost";                 // r === 1
}

Scoring helper (scoreFromBounds, ✅): maps a value onto its percentile bucket, producing 1–5. For "higher-is-better" metrics (F, M) it returns the bucket score directly; for recency it inverts (6 - score) so fewer days ⇒ higher R. Never-ordered / zero-order customers are re-bucketed by explicit end-spike (recency 999999 → R=1) and start-spike (freq/money 0 → score 1) fixes so the 999999 sentinel and zero-value crowd don't distort the real quintiles.

3.2 Deep filtration (on-demand, async job queue)¶

UI → POST /crm_api/customer/insights/submit           controllers/customerInsightsController.submitJob
        └─ insert segment_jobs {job_id(uuidv4), filters, status:'pending', submitted_at}   ✅
                              │
crons/customerInsightsCron.js  processingCron  ("* * * * *")   ✅
        ├─ pick 3 oldest {status:'pending'} → set status:'processing', picked_at
        ├─ processJob(): build $match from filters → aggregate customers
        │     → segment_results (matching mobiles, batched 1000)
        │     → segment_counts   {count, sms_count, whatsapp_count, push_count}
        └─ set segment_jobs.status:'completed' (or 'failed' + error_message)

        archiveCron  ("0 2 * * *")  → delete jobs/results/counts older than 2 days   ✅

UI → POST /customer/insights/status     → getJobStatus (queue ETA math)
UI → POST /customer/insights/customers  → getJobCustomers (paginated, limit 10)
UI → POST /customer/insights/export     → exportJobCustomers (CSV; only if 'completed')

3.3 RFM read path (dashboard)¶

GET counts:   POST /rfm/segment/counts    → business_aggreator.rfm_segment_count   ✅
GET customers:POST /rfm/segment/customers → customers.find({'rfm.sg': segment}) paginated (limit ≤ 500)
GET segments: POST /rfm/segment           → rfm_segments collection + business_configs {rfm_enabled, window_days}
Config:       POST /update/rfm/window     → business_configs.window_days
Enable:       POST /rfm/request/enable    → email uengage ops + set rfm_request_mail_sent

4. Architecture Diagram¶

4.1 RFM batch flowchart¶

flowchart TD
    subgraph nightly["rfmSegments.js — 00:30 IST nightly"]
        A[getEnabledBusinesses<br/>business_configs.rfm_enabled=true] --> B[buildCustomerBaseTemp<br/>recencyDays, lto]
        B --> C[buildOrderWindowMetricsTemp<br/>freq, money over window_days]
        C --> D[getBucketBounds R/F/M<br/>percentile quintiles + spike fixes]
        D --> E[score r,f,m → fm=Ceil F+M /2 → getSegment]
        E --> F[(customers.rfm.*)]
        E --> G[(business_aggreator.rfm_segment_count)]
        A --> H[(rfm_job_log)]
        E --> I[(rfm_metrics_tmp — temp, cleaned)]
    end
    F --> RD[POST /rfm/segment/customers]
    G --> RC[POST /rfm/segment/counts]

4.2 Segment-job sequence¶

sequenceDiagram
    autonumber
    participant UI as Marketer UI
    participant API as insights API (Express)
    participant JQ as segment_jobs
    participant CRON as customerInsightsCron (1 min)
    participant RES as segment_results / segment_counts
    UI->>API: POST /customer/insights/submit {filters}
    API->>JQ: insert {job_id, status:pending}
    API-->>UI: {status:true, job_id}
    loop every minute
        CRON->>JQ: pick 3 oldest pending → processing
        CRON->>RES: aggregate customers → write mobiles + counts
        CRON->>JQ: status = completed
    end
    UI->>API: POST /customer/insights/status
    API-->>UI: {status, estimated_pick_time, count}
    UI->>API: POST /customer/insights/export
    API-->>UI: {link: CSV}

4.3 Segment lifecycle (7-segment code model, state view)¶

stateDiagram-v2
    [*] --> new_customers: R=5 & LTO=1
    new_customers --> promising: R>=4, FM=1
    promising --> regular: R>=4, FM 2-3
    regular --> champions: R>=4, FM 4-5
    champions --> need_attention: R=3
    regular --> need_attention: R=3
    need_attention --> at_risk: R=2
    at_risk --> lost: R=1
    lost --> [*]

5. Folder Structure¶

Real files that make up this module (paths relative to repo root uengage-crm/):

Layer	File	Role	Validation
Controller	`controllers/rfmController.js`	RFM read/config HTTP API	✅
Controller	`controllers/customerInsightsController.js`	37-filter async segment-job API	✅
Cron	`crons/rfmSegments.js`	Nightly RFM scoring engine	✅
Cron	`crons/customerInsightsCron.js`	Segment-job processor (1 min) + archive (2 AM)	✅
Constants	`constants/rfmSegments.js`	Canonical list of the 7 segments	✅
Process lib	`processLibrary/segmentCountV2.js`	Advanced multi-field segment counting	✅
Process lib	`processLibrary/getOnlySegmentCount.js`	Large optimized segment-count query helper (~1.1 MB)	✅
Routes	`routes/crm.js` (lines ~1762–1778)	Registers all `/rfm/` and `/customer/insights/` routes under `/crm_api`	✅
Middleware	`middlewares/authMiddleware.js`	Applied to every endpoint here	✅
Utility	`utilities/mongodb.js` (`getMongoDB`)	Native driver handle used by these controllers/crons	✅
Utility	`utilities/loggerUtils.js` (`logErrorWithPrefixAndSuffix`)	Structured error logging	✅

Upstream: customers, orders collections are produced by Customer-CRM and Order-Ingestion. This module is a pure consumer of those.

6. Database¶

All MongoDB (native driver via getMongoDB()), scoped by parent_business_id.

Collection	Purpose	Key fields	Written by	Read by
`customers`	Customer 360 master; holds computed `rfm` sub-doc	`mobileNo`, `parent_business_id`, `LTO`, `LTV`, `avg_order_size`, `lastOrderDate`, `rfm.{r,f,m,fm,sc,sg,wd,ca}`	rfmSegments cron (bulkWrite)	rfmController, insights cron
`orders`	Order facts (window source for F & M)	`parent_business_id`, `mobile_number`, `amount`, `invoiced_at`	Order-Ingestion	rfmSegments, insights cron
`business_configs`	Per-tenant RFM config	`parent_business_id`, `is_parent`, `rfm_enabled`, `window_days` (default 90), `rfm_request_mail_sent`	rfmController	rfmSegments, rfmController
`business_aggreator`	Per-tenant rollups incl. RFM counts	`businessId`, `is_parent`, `rfm_segment_count` (per-segment + total), `rfm_updated_at`	rfmSegments cron	rfmController `getSegmentCounts`
`rfm_segments`	RFM segment reference/definitions	(definition docs)	(seed/reference)	rfmController `getSegment`
`rfm_metrics_tmp`	Ephemeral per-run working set	`run_id`, `mobileNo`, `recencyDays`, `frequencyOrders`, `monetaryValue`, `lto`, `expireAt`	rfmSegments cron	rfmSegments cron (then deleted)
`rfm_job_log`	RFM run audit log	`parent_business_id`, `run_id`, `window_days`, `started_at`, `status`, `ended_at`, `error_stack`	rfmSegments cron	ops/debug
`segment_jobs`	Async filter-job queue	`job_id` (uuidv4), `parent_business_id`, `business_id`, `filters`, `status` (pending/processing/completed/failed), timestamps	insights controller + cron	both
`segment_results`	Matched mobiles per job	`job_id`, `mobile_number`, `parent_business_id`, `created_at`	insights cron	getJobCustomers/export
`segment_counts`	Per-job counts + channel breakdown	`job_id`, `count`, `sms_count`, `whatsapp_count`, `push_count`	insights cron	getJobStatus
`segment_filters`	Saved reusable filter sets	`parent_business_id`, `business_id`, `filter_name`, `filters`, `updated_at`	saveFilters	getSaveFilters
`customers_outlets`	Customer↔outlet mapping (multi-outlet filters)	`customer_id`, `business_id`, `business_state`, `businessCity`, `lastOrderDate`	Customer-CRM/dedup	insights cron ($lookup)
`order_items`	Line items (SKU/category filters)	`mobile_number`, `menu_name`, `item_name`, `sku_id`, `section_name`	Order-Ingestion	insights cron (pre-compute)

Keys / relationships: mobileNo (10-digit) is the customer key; parent_business_id scopes every query (multi-tenant). job_id (UUID) links segment_jobs → segment_results/segment_counts.

Indexes: ⚠️ Most collections have no explicit indexes (see map-03 §5.1). The RFM cron does full-collection aggregations per tenant; customers.parent_business_id and orders.(parent_business_id, invoiced_at) are the highest-value missing indexes.

Common queries:

// segment counts for dashboard
db.business_aggreator.findOne({ businessId: String(parentId), is_parent: true },
  { projection: { rfm_segment_count: 1, rfm_updated_at: 1 } })

// customers in a segment (paginated)
db.customers.find({ parent_business_id, 'rfm.sg': 'champions' }).limit(500)

7. APIs¶

All routes mount under /crm_api (index.js: app.use("/crm_api", crm_routes)), method POST, and require authMiddleware (JWT). Representative payloads marked verify are inferred from body destructuring.

7.1 `POST /crm_api/rfm/segment` — segment definitions + config ✅¶

Auth: authMiddleware
Request: { "business_id": "12345" } (verify)
Response: { status, segments, rfm_enabled, rfm_request_mail_sent, window_days }
Reads: rfm_segments, business_configs
Failure: returns {status:false, message} on error (note: logged under a copy-pasted prefix getSegmentCounts ♻️).

7.2 `POST /crm_api/rfm/segment/counts` — per-segment counts ✅¶

Request: { "parent_business_id": "1", "business_id": "0" }
Validation: parent_business_id required. When parent_business_id == '0', uses business_id as the parent (single-outlet convention).
Response: { status, segment_counts, last_updated } from business_aggreator.

7.3 `POST /crm_api/rfm/segment/customers` — customers in a segment ✅¶

Request: { parent_business_id, business_id, segment, page, limit }
Validation: limit capped at 500.
Response: { status, data:[{mobile_number, customer_name, gender, email, aov, lto, ltv, last_order_date}], total, page, limit }

7.4 `POST /crm_api/rfm/segment/export` — CSV export ✅¶

Response: { status, link } → CSV under /public/rfm_exports/{segment}_{parentId}_{ts}.csv.

7.5 `POST /crm_api/update/rfm/window` — set RFM window ✅¶

Request: { business_id, window_days }
Validation: window_days must be a positive number → else {status:false}.

7.6 `POST /crm_api/rfm/request/enable` — request RFM enablement ✅¶

Request: { parent_business_id, email }
Effect: emails uEngage ops (https://www.uengage.in/addoapi/sendCRMMail), sets rfm_request_mail_sent.

7.7 `POST /crm_api/customer/insights/submit` — create segment job ✅¶

Request: { parent_business_id, business_id, filters:{…} }
Response: { status:true, job_id } — job queued in segment_jobs (pending).

7.8 `POST /crm_api/customer/insights/status` — poll jobs ✅¶

Response: { status, data:[{job_id, status, count, estimated_pick_time, estimated_completion_time, …}] } — ETA computed from global pending queue (3 jobs/min).

7.9 `POST /crm_api/customer/insights/customers` — paginated results ✅¶

Request: { job_id, business_id, page } — fixed limit=10.

7.10 `POST /crm_api/customer/insights/export` — CSV of matched mobiles ✅¶

Only if job.status === 'completed', else {status:false, job_status}. CSV: mobile_number column, /public/segment_exports/customers_{job_id}.csv.

7.11 `POST /crm_api/save/filters` & `POST /crm_api/get/save/filters` — saved filters ✅¶

Upsert / list into segment_filters.

Supported filter keys (💻 in `customerInsightsCron`)¶

customer_segment (RFM, one of the 7), outlets[], states[], cities[], items[] (menu_name/item_name/sku_id), timeOfDay[] (morning 5–11 / afternoon 12–15 / evening 16–19 / night 20–4), lto{min,max}, ltv{min,max}, aov{min,max} (on avg_order_size), visitedDays (30/60/custom), notVisitedDays, dayType (weekdays/weekends). Filters combine with AND; multi-select values are OR within a filter (matches Excel 3.x semantics 📄).

8. Code Walkthrough¶

`crons/rfmSegments.js` (the engine)¶

getEnabledBusinesses(db) — selects business_configs {rfm_enabled:true}. Tenant loop guarantees isolation.
normalizeWindowDays() — clamps window_days (default 90) per Excel-configurable window (📄 1.x / ✅).
buildCustomerBaseTemp() — writes recencyDays (or 999999 if never ordered) and lto into rfm_metrics_tmp.
buildOrderWindowMetricsTemp() — aggregates orders in [now-window, now] → frequencyOrders, monetaryValue.
getBucketBounds(field) — computes 5-quintile boundaries; Fix A (recency end-spike, 999999 → R=1) and Fix B (freq/money start-spike, 0 → score 1) keep sentinel/zero crowds from polluting quintiles.
scoreFromBounds(value, bounds, higherIsBetter) — value → 1–5 bucket score, inverted for recency.
getSegment(r, fm, lto) — the 7-segment if/else (see §3.1).
updateCustomersAndBuildSegmentCounts() — customers.bulkWrite sets rfm.{r,f,m,fm,sc,sg,wd,ca} and tallies counts.
updateBusinessAggregator() — writes rfm_segment_count + rfm_updated_at.
cleanupTempRun() — deletes the run's temp docs.

Call hierarchy: cron.schedule("30 0 * * *") → per business → startJobLog → buildCustomerBaseTemp → buildOrderWindowMetricsTemp → getBucketBounds×3 → updateCustomersAndBuildSegmentCounts → updateBusinessAggregator → endJobLog → cleanupTempRun.

`controllers/rfmController.js`¶

getSegment, getSegmentCounts, getSegmentCustomers, exportSegmentCustomers, updateRFMWindow, requestRfmEnable. Helper resolveParentId implements the parent_business_id=='0' single-outlet convention. escapeCsvField guards CSV injection.

`controllers/customerInsightsController.js`¶

submitJob, getJobStatus (queue-ETA math), getJobCustomers, exportJobCustomers, saveFilters, getSaveFilters.

`processLibrary/segmentCountV2.js`¶

Advanced multi-field segment counting (RFM, order-value ranges, last-order-date, comm preferences) used by dashboard segment tooling; complements the insights cron's $match builder.

9. Business Rules¶

Rule	Detail	Validation
7-segment code vs 11-segment spec	Code ships 7 (`champions, regular, promising, need_attention, at_risk, lost, new_customers`); Excel wants 11 via R×FM grid.	✅ code / 📄 spec — ⚠️ gap
FM = Ceil((F+M)/2)	Collapses F×M to a single 1–5 axis.	✅ (matches Excel 1.4)
Never-ordered → R=1 → lost	`recencyDays=999999`, end-spike fix.	✅ (Excel 1.1)
Quintiles are per-brand relative	Boundaries from each tenant's own distribution; sparse data may skip buckets.	✅ / 📄 1.2
RFM only for enabled tenants	`rfm_enabled:true` gate; brands request via `/rfm/request/enable`.	✅
Configurable window	`window_days` default 90, per parent business.	✅
new_customers requires LTO=1	Only truly first-order customers, even if R=5.	✅
Cancelled-order exclusion	Excel requires excluding cancelled from revenue; RFM cron does not filter order status.	📄 spec, ⚠️ not enforced in RFM cron
Placeholder-mobile exclusion	Repeated-digit / out-of-range mobiles skipped.	✅
Tenant scoping	Every query filters by `parent_business_id`; `'0'` means treat `business_id` as parent.	✅
Insights limits	Job results page `limit=10`; RFM customers `limit≤500`; jobs archived after 2 days.	✅
Killer feature (offer→segment)	Excel 3.24 "Create promo mapped exclusively to a filtered segment" is Not Live.	📄
Send-from-segment	Excel 3.23 one-click campaign from segment is Not Live (roadmap B1.2).	📄

Computed-fields schedule (Excel 3.26–3.40, 📄; where implemented ✅):

Field	Cadence (spec)	Where in code
Days since last visit, Total visits, Total spend, AOV	Real-time / per-order	Customer-CRM ingestion (`customers` update) ✅
R/F/M scores + RFM segment, Spend bucket, Customer status, Churn risk	Daily nightly batch	`rfmSegments` (RFM ✅); spend bucket / status / churn 📄 mostly
Day preference, Time preference, Favourite item/category	Weekly batch	insights cron computes on-the-fly for filters ✅; persisted weekly 📄
Loyalty tier + points	Real-time per txn	Loyalty ✅
NPS category	Real-time per feedback	Feedback-NPS ✅

10. Performance¶

Batch, not real-time: RFM is a nightly bulkWrite per tenant — reads served from precomputed customers.rfm / business_aggreator.rfm_segment_count (O(1) dashboard reads). ✅
Temp collection + cleanup: rfm_metrics_tmp isolates working state per run_id and is deleted after each run (has expireAt TTL as a safety net). ✅
Job concurrency cap: insights cron processes 3 jobs/minute, batching results in 1000-doc inserts — bounds load and gives predictable ETAs. ✅
Pagination: insights results limit=10, RFM customers limit≤500 prevent large payloads. ✅
Bottlenecks / risks ⚠️:
No indexes on customers.parent_business_id, orders.(parent_business_id, invoiced_at) → full-scan aggregations grow with tenant size.
All crons run in the web process — RFM/insight load couples with API latency (see High-Level Architecture §9).
Outlet filter uses $lookup (can't be pre-computed due to BSON size), the heaviest filter path.

11. Logging¶

Source: utilities/loggerUtils.logErrorWithPrefixAndSuffix(stack, fnName, controllerName) in every controller catch; logger.info(...) (Winston) in the RFM cron ("RFM cron picked N businesses", per-tenant start lines).
Audit log: rfm_job_log is the durable per-run record (started_at, status, ended_at, error_stack, window_start/end).
Trace correlation: run_id = {parentBusinessId}_{epochMs} ties temp docs + job log + logs for one run. ♻️ Note some controller catches use a copy-pasted prefix (getSegment logs as getSegmentCounts) — grep by controller name, not just prefix.
Key messages: RFM cron picked N businesses, RFM cron started for parent_business_id=…, window_days=….

12. Monitoring¶

Signal	Where	Healthy
RFM freshness	`business_aggreator.rfm_updated_at`	Updated within last 24h for every enabled tenant
RFM run status	`rfm_job_log.status`	`completed`; no lingering `started` past 01:30 IST
Segment sanity	`rfm_segment_count` per tenant	`lost` should not be ~100% (would signal window/data issue)
Job queue depth	`segment_jobs {status:'pending'}`	Small; drains at 3/min. Growing backlog ⇒ cron stalled
Job failures	`segment_jobs {status:'failed'}` + `error_message`	Near-zero
Cron liveness	web-process `cron_stats` / PM2	Cron heartbeats present

"Healthy" = every enabled tenant has a completed rfm_job_log and a same-day rfm_updated_at, and segment_jobs pending count trends to zero each minute.

13. Troubleshooting¶

Symptom	Root cause	Resolution
Dashboard RFM counts stale	RFM cron didn't run / tenant not `rfm_enabled`	Check `rfm_job_log` for the tenant; confirm `business_configs.rfm_enabled:true`; check PM2 process up
"Everyone is Lost"	Empty/short window, or all orders outside `window_days`, or cancelled orders inflating recency	Verify `orders.invoiced_at` populated; check `window_days`; inspect quintile bounds in logs
Segment job stuck `pending`	insights cron down or queue congested	Count pending jobs; restart cron; check `error_message` on `failed` jobs
Export returns error	Job not `completed`	Poll `/status` until `completed`, then export
RFM counts ≠ insights count for same segment	RFM uses precomputed `rfm.sg`; insights recomputes RFM for a custom `visitedDays` window	Expected — different windows; align windows to compare
New customer never shows in `new_customers`	Requires R=5 and LTO=1; second order flips them to `promising`/`regular`	By design

Commands:

// last RFM run for a tenant
db.rfm_job_log.find({ parent_business_id: 123 }).sort({ started_at:-1 }).limit(1)
// pending job backlog
db.segment_jobs.countDocuments({ status: 'pending' })
// segment distribution for a tenant
db.customers.aggregate([{ $match:{ parent_business_id:123 }},{ $group:{ _id:'$rfm.sg', n:{ $sum:1 }}}])

14. FAQs¶

Why 7 segments in code but 11 in the product spec? The Excel is the target (R×FM grid); the shipped cron uses a simpler R-first if/else. Reconciling them is open work. Always trust constants/rfmSegments.js for what actually exists.
Is FM really Ceil((F+M)/2)? Yes — Math.ceil((f+m)/2) in rfmSegments.js line ~421. ✅
Where do quintile cutoffs come from? Each tenant's own R/F/M distribution (percentile bounds), recomputed nightly — not fixed thresholds.
How do never-ordered customers score? recencyDays=999999 → R=1 → lost. ✅
Real-time or batch? RFM is nightly batch (00:30 IST). Segment jobs are async (≈1 min). AOV/LTV/LTO are real-time (Customer-CRM).
Do cancelled orders count? In the RFM cron currently yes (no status filter) ⚠️ — a known spec/code gap.
How is an audience handed to campaigns? Today via CSV export; one-click send is roadmap (📄 3.23 / B1.2).

15. Cheat Sheet¶

ENGINE:  crons/rfmSegments.js   cron "30 0 * * *" (00:30 IST)   [7-segment model]
JOBS:    crons/customerInsightsCron.js  process "* * * * *" (3/min), archive "0 2 * * *"
SEGMENTS (code): champions, regular, promising, need_attention, at_risk, lost, new_customers
MATH:    R,F,M ∈ 1..5 (per-brand quintiles) ; FM = Ceil((F+M)/2)
         r5&lto1→new; r>=4:{fm1→promising, fm<=3→regular, else→champions}; r3→need_attention; r2→at_risk; r1→lost
COLLECTIONS: customers.rfm.{r,f,m,fm,sc,sg,wd,ca} | business_aggreator.rfm_segment_count
             segment_jobs → segment_results / segment_counts | segment_filters | rfm_job_log | rfm_metrics_tmp(temp)
ROUTES (POST, /crm_api, authMiddleware):
  /rfm/segment  /rfm/segment/counts  /rfm/segment/customers  /rfm/segment/export
  /update/rfm/window  /rfm/request/enable
  /customer/insights/submit|status|customers|export   /save/filters  /get/save/filters
FILTERS: customer_segment, outlets, states, cities, items(sku), timeOfDay, lto, ltv, aov,
         visitedDays, notVisitedDays, dayType  (AND across, OR within)
GOTCHAS: 7≠11 segments ⚠️ | cancelled orders NOT excluded in RFM cron ⚠️ | window_days default 90
         | never-ordered=R1 | job export only when 'completed' | limit: insights=10, rfm=500

Upstream: Customer-CRM (Customer 360) — produces customers; Order-Ingestion — produces orders/order_items.
Downstream: Campaigns, Automated-Journeys, Loyalty — consume segments/audiences.
Sibling: Dashboard-Analytics — surfaces RFM counts.
Cross-cutting: Database · Queues · API · Security · Terminologies · High-Level Architecture.

Knowledge Tests¶

Level 1 — MCQs¶

What is the FM score formula? a) (F+M) b) Floor((F+M)/2) c) Ceil((F+M)/2) d) max(F,M) — c ✅ (Math.ceil((f+m)/2)).
A never-ordered customer receives which R score? a) 5 b) 3 c) 1 d) null — c (recencyDays=999999 → R=1).
How many segments does the code actually assign? a) 11 b) 7 c) 5 d) 25 — b (constants/rfmSegments.js). 11 is the Excel spec only.
Which condition yields new_customers? a) R=5 only b) FM=1 only c) R=5 AND LTO=1 d) R≥4 AND FM=5 — c.
The customer-insights cron picks how many pending jobs per run? a) 1 b) 3 c) 10 d) all — b (drives the ETA math).

Level 2 — Scenarios¶

A brand complains their "Champions" count dropped to zero overnight. Their window_days was just changed from 90 to 7. Explain the likely cause and where you'd confirm it. (Hint: shorter window ⇒ fewer in-window orders ⇒ lower F/M ⇒ fewer high-FM customers; confirm in rfm_job_log window + quintile bounds in logs.)
Marketing built a filter for "AOV > 500 AND Champions" and the count differs from the RFM dashboard's Champions count. Why can these legitimately differ?

Level 3 — Code reading¶

Open crons/rfmSegments.js. Trace getSegment(r, fm, lto) and scoreFromBounds. For a customer with recencyDays=10, frequencyOrders=1, monetaryValue=2000 in a base where those land at R=5, F=2, M=5 — compute FM and the resulting segment. (FM=Ceil((2+5)/2)=4; R=5, LTO assume 1 → new_customers; if LTO>1 → champions.)

Level 4 — Architecture¶

The Excel mandates an 11-segment R×FM grid but the code has 7. Design a migration to the grid without breaking existing customers.rfm.sg consumers (Campaigns/Journeys/Dashboard). Address: dual-write vs cutover, backfill, and how segment IDs map. Reference High-Level Architecture.

Level 5 — Production debugging¶

business_aggreator.rfm_updated_at for a large tenant is 3 days stale while smaller tenants are fresh. rfm_job_log shows status:'started' with no ended_at for that tenant each night. Diagnose (unindexed full-scan aggregation timing out / OOM in the shared web process) and propose immediate + structural fixes.

Practical Assignments¶

Trace a segment job end-to-end: submit a job via /customer/insights/submit, watch segment_jobs flip pending→processing→completed, and read the resulting segment_results/segment_counts. Document each state transition and timing.
Add a filter: extend customerInsightsCron to support a gender filter on customers.gender, wire it through the $match builder, and verify counts. (Add the key to the filter docs in §7.)
Reproduce the 7-vs-11 gap: pick 5 real customers, compute their Excel-grid segment (R×FM lookup 2.12–2.16) by hand, then read their customers.rfm.sg — tabulate where the code's 7-segment result diverges from the spec's 11-segment result.