Skip to content

System Overview

The 10-minute technical mental model. Read Product Overview first for what Prism does; this page is how it's built.

1. The whole system in one picture

flowchart TB
    subgraph clients["Clients"]
        POS[POS systems]
        APP[Brand App / Web / Edge]
        AGG[Aggregators]
        WAIN[WhatsApp inbound]
    end

    subgraph edge["prism-services · AWS Lambda + Kafka"]
        OAPI[orders-api λ]
        OCONS[orders-consumer λ]
        WSEND[whatsapp api-send λ]
        WRcv[whatsapp api-receipts λ]
        WSC[whatsapp consumers λ]
        KAFKA[(MSK Kafka<br/>prism-ingest-orders<br/>whatsapp-messages<br/>whatsapp-receipts)]
    end

    subgraph mono["uengage-crm · Express monolith (PM2)"]
        ROUTES[Routes /crm_api, /stats, /integration/v1 ...]
        CTRL[57 Controllers]
        LIB[commonFunctions + processLibrary]
        CRONS[91 Crons in-process]
        SOCK[Socket.IO chat]
    end

    subgraph data["Datastores"]
        MONGO[(MongoDB 'uengage'<br/>queues, MIS, campaigns)]
        MYSQL[(MySQL 'addo_*'<br/>orders, users, wallet, promo)]
        REDIS[(Redis ElastiCache)]
    end

    POS & APP & AGG --> OAPI --> KAFKA --> OCONS --> MONGO
    POS & APP --> ROUTES
    WAIN --> WRcv --> KAFKA
    WSEND --> KAFKA --> WSC
    ROUTES --> CTRL --> LIB
    CRONS --> LIB
    LIB --> MONGO & MYSQL & REDIS
    CRONS --> MONGO & MYSQL
    SOCK --> MONGO

2. Two deployables, one database

uengage-crm (this repo) prism-services
Kind Long-running Express monolith Serverless functions
Runtime Node.js (mongoose 6, mysql2, ioredis) Node.js 22.x on AWS Lambda
Process mgr PM2 (crm_ecosystem.config.js) Serverless Framework
Triggers HTTP + 91 in-process crons + Socket.IO API Gateway (httpApi) + Kafka (MSK)
Writes to MongoDB uengage, MySQL, Redis MongoDB uengage (crm_queue, business_configs)
Region — ap-south-1

Both share the same MongoDB uengage database and the crm_queue collection. prism-services is the scalable "front door" for high-volume ingestion; the monolith does everything else and drains the queue via crons. ⚠️ The monolith also has its own ingestion controllers writing the same collection — see Technical Debt.

3. The stack (uengage-crm)

  • Web: Express 4, body-parser, open CORS. Entry index.js.
  • Datastores:
  • MongoDB (utilities/mongodb.js) — DB name hardcoded uengage, maxPoolSize: 400, readPreference: SECONDARY_PREFERRED. Both native driver + Mongoose.
  • MySQL (config/mysqlPool.js) — two pools: webPool (limit 15, for request handlers) and jobsPool (limit 10, for crons) so background jobs never starve web traffic. Monitored by utilities/poolMonitor.js (emits pool_stats JSON every 60s).
  • Redis (utilities/redis.js) — ioredis to AWS ElastiCache (host currently hardcoded).
  • Async backbone: node-cron. All 91 crons are require()d at startup in index.js and self-register — they run inside the same process as the web server.
  • Real-time: Socket.IO (socket.js) for the WhatsApp live-chat agent console (state in Mongo whatsapp_chat_*).
  • Logging: Winston + daily-rotate (logger/), plus a trace-id-aware logger-service.js (@uengage.io/js-logger, AsyncLocalStorage). See Debugging.
  • APM: New Relic + Elastic APM present but currently disabled/commented — monitoring is effectively logs + pool stats today.

4. The stack (prism-services)

  • orders-api (API Gateway) → publishes to Kafka prism-ingest-orders.
  • orders-consumer (MSK trigger, batchSize 10) → decodes and inserts into crm_queue with a numeric source id, then bumps business_configs.last_uen_order_hit.
  • whatsapp service: api-send + api-receipts (HTTP) publish to whatsapp-messages / whatsapp-receipts; consumers currently log-only (stubbed) — no DB writes yet.
  • Details + diagrams: High-Level Architecture.

5. How a request/order actually flows (short version)

  1. Order in → orders-api (Lambda) or monolith /crm_api/injestion/{pos} → crm_queue (status: 0).
  2. Crons drain crm_queue → parse → upsert customer + order → recompute LTV/LTO/AOV → enqueue loyalty points → trigger digital bill + eligible journeys → mark processed.
  3. Nightly/periodic crons compute RFM segments, MIS/aggregations, tier downgrades.
  4. Marketer acts → campaign/journey → channel queue (WhatsApp/SMS/Push crons) → provider → delivery/read/click tracked → ROI attributed → dashboards.

Full detail: Request Lifecycle.

6. Multi-tenancy & identity (memorize these)

  • Tenant scope: parent_business_id (brand/HQ) + child_business_id (outlet). Almost every query filters on these.
  • Customer key: the 10-digit mobile number — the join key across MongoDB and MySQL.
  • crm_queue.status is the ingestion state machine.

7. Auth at a glance (6 layers)

Middleware Who Mechanism
authMiddleware Business/staff (web+mobile) MySQL token / auth_tokens / bcrypt
adminMiddleware Admins MySQL + business_id == 1
appMiddleware Customer app (loyalty) Mongo addo_contacts_mapping token
jwtMiddleware / jwtWhatsappMiddleware Internal/WhatsApp JWT HS256 (hardcoded secrets ⚠️)
crm_offer_middleware POS / 3rd-party Mongo authorization_token
injestion_middleware / thirdPartyMiddleware Ingestion hardcoded static token ⚠️

Details + the security debt list: Security.

8. What surprises new joiners

  • Crons live in the web process — a heavy cron can affect API latency; that's why the separate jobsPool exists.
  • Two dashboards (dashboardController.js legacy + dashboardControllerNew.js) and three campaign pipelines coexist (migration in progress). See Module Architecture.
  • Loyalty points live in both Mongo and MySQL without atomic transactions.
  • Secrets are hardcoded in several middlewares and .env.dev is committed.

Assessment

../15-Assessments/00-Getting-Started-Assessment.md