High-Level Architecture¶

1. Component topology¶

flowchart TB
    subgraph ext["External"]
        POS[POS: PetPooja, UrbanPiper, Posist, LS Central, Rista, uEngage POS]
        EDGE[uEngage Edge / App / Website]
        AGG[Aggregators: Swiggy, Zomato]
        WA[WhatsApp / Meta / Gupshup]
        FCM[FCM / Firebase]
        SMSP[SMS providers / DLT]
        S3[(AWS S3)]
        GMB[Google / Zomato reviews]
    end

    subgraph svc["prism-services · Lambda (ap-south-1)"]
        OAPI["orders-api<br/>httpApi POST /crm_api/injestion/{pos}"]
        OCONS["orders-consumer<br/>MSK batch=10"]
        WSEND["whatsapp api-send"]
        WREC["whatsapp api-receipts"]
        WCS["whatsapp consumer-send / consumer-receipts<br/>(stubbed)"]
    end

    subgraph kafka["Amazon MSK (Kafka)"]
        T1[[prism-ingest-orders]]
        T2[[whatsapp-messages]]
        T3[[whatsapp-receipts]]
    end

    subgraph crm["uengage-crm · Express monolith (PM2)"]
        RT[Routes]
        CT[Controllers]
        CF[commonFunctions]
        PL[processLibrary]
        CR[Crons x91]
        SK[Socket.IO]
    end

    subgraph store["Datastores"]
        MG[(MongoDB uengage)]
        SQ[(MySQL addo_*)]
        RD[(Redis ElastiCache)]
    end

    POS --> OAPI
    EDGE --> OAPI
    AGG --> OAPI
    OAPI --> T1 --> OCONS --> MG
    WSEND --> T2 --> WCS
    WA --> WREC --> T3 --> WCS

    POS -->|direct| RT
    EDGE -->|direct| RT
    RT --> CT --> CF
    CR --> CF
    CR --> PL
    CF --> MG & SQ & RD
    PL --> MG & SQ
    CT --> MG & SQ
    SK --> MG

    CF --> WA & FCM & SMSP & S3
    CR --> GMB
    OCONS -->|update| MG

2. Why this shape? (the architectural story)¶

Prism started as a single Express monolith (uengage-crm) that does everything — ingestion, APIs, dashboards, campaigns, loyalty, and 91 in-process crons. As order volume grew, a serverless ingestion tier (prism-services) was added in front to absorb spiky, high-throughput POS traffic without risking the monolith. Kafka (MSK) decouples the burst of incoming orders from the slower downstream processing.

So the system is mid-migration and deliberately transitional: - Edge ingestion → serverless + Kafka (new, scalable). - Everything else → monolith + Mongo/MySQL + crons (legacy, being migrated to Next.js on the frontend; backend still monolithic). - The crm_queue collection is the seam between the two worlds.

3. Data plane vs control plane¶

Plane	What	Where
Ingestion (data in)	Orders + customers from all sources land in `crm_queue`	prism-services and monolith ingestion controllers
Processing	Drain queue → build Customer 360/Order data, points, journeys	monolith crons + commonFunctions
Intelligence	RFM, segments, MIS/aggregation	monolith crons + processLibrary
Activation (data out)	Campaigns/journeys → WhatsApp/SMS/Push	monolith channel queues + crons → providers
Presentation	Dashboards, reports, APIs	monolith controllers + `/stats` metrics API

4. Kafka topics¶

Topic	Producer	Consumer	Payload	Batch
`prism-ingest-orders`	`orders-api` λ	`orders-consumer` λ → `crm_queue`	base64-encoded order JSON	10
`whatsapp-messages`	`whatsapp api-send` λ	`consumer-send` λ (⚠️ log-only)	message JSON	20
`whatsapp-receipts`	`whatsapp api-receipts` λ (from Meta/Gupshup webhooks)	`consumer-receipts` λ (⚠️ log-only)	receipt JSON	20

⚠️ The WhatsApp consumers are stubbed — they log but do not persist. The monolith's own WhatsApp crons/queues currently do the real send/receipt work. Reconciling these is a roadmap item. See Technical Debt.

5. Datastore responsibilities¶

Store	Holds	Access pattern
MongoDB `uengage` (maxPool 400, SECONDARY_PREFERRED)	`crm_queue`, ingestion dumps, campaign MIS, journeys, ratings/NPS, dashboard MIS, socket chat state, config	high-write queues + aggregation reads
*MySQL `addo_`** (webPool 15 + jobsPool 10)	orders, order items, users/roles/auth, customers, wallet + `wallet_history` + `wallet_rules`, `crm_points_allocation_queue`, promo engine, menus	transactional, joined reads
Redis ElastiCache	caches, transient state	short-lived

Full catalog: Database.

6. Order ingestion sequence (canonical flow)¶

sequenceDiagram
    autonumber
    participant SRC as POS/App/Aggregator
    participant API as orders-api λ
    participant K as Kafka prism-ingest-orders
    participant CN as orders-consumer λ
    participant Q as MongoDB crm_queue
    participant CFG as MongoDB business_configs
    participant CRON as Monolith ingestion crons
    participant DB as Customer/Order data (Mongo+MySQL)

    SRC->>API: POST /crm_api/injestion/{pos} (order payload)
    API->>K: publish base64(order), source tag
    API-->>SRC: 200 (accepted)
    K->>CN: deliver batch (≤10)
    CN->>Q: insert {data, source:<n>, status:0, insertedAt, apiSource:'lambda'}
    CN->>CFG: update last_uen_order_hit
    Note over CRON,Q: asynchronously…
    CRON->>Q: find {status:0}
    CRON->>DB: upsert customer + order, recompute LTV/LTO/AOV
    CRON->>Q: mark status=processed

Note the dual-writer reality: some sources hit orders-api (Lambda), others hit the monolith's /injestion/* routes directly (see route inventory). Both converge on crm_queue.

7. WhatsApp send sequence¶

sequenceDiagram
    autonumber
    participant MK as Marketer / Journey
    participant CRM as Monolith campaign queue
    participant WQ as WhatsApp queue collection
    participant CRON as sendWhatsappMessageCron
    participant PROV as Gupshup / Meta
    participant REC as api-receipts λ
    participant K as Kafka whatsapp-receipts

    MK->>CRM: create campaign/journey (audience + template)
    CRM->>WQ: enqueue messages (status=pending)
    CRON->>WQ: poll pending
    CRON->>PROV: send template message
    PROV-->>CRON: message id / accepted
    CRON->>WQ: mark sent
    PROV->>REC: delivery/read webhook
    REC->>K: publish receipt
    Note over K: consumer-receipts currently log-only ⚠️

8. Cross-cutting concerns¶

Multi-tenancy: parent_business_id + child_business_id scope every operation.
Async backbone: in-process node-cron; queue collections act as work buffers. See Queues.
Observability: Winston rotating logs + pool_stats; APM present but disabled. See Debugging.
Security: 6 auth middlewares; several hardcoded secrets. See Security.

9. Known architectural risks (summary)¶

Risk	Impact	Ref
Dual ingestion writers to `crm_queue`	duplicate orders, race conditions	Tech Debt
Loyalty points across Mongo and MySQL, non-atomic	balance drift (cf. bug E1.10)	Loyalty
91 crons in the web process	cron load ↔ API latency coupling	Queues
Stubbed WhatsApp consumers	incomplete migration	above
Hardcoded secrets / open CORS / no rate limit	security exposure	Security
No DLQ/retry on Kafka consumers	lost/duplicated messages	prism-services map

Request Lifecycle · Module Architecture · Queues · Database

Assessment¶

Level-4 architecture questions: ../15-Assessments/Level-4-Architecture.md