Level-4 Assessment — Architecture¶
System-level reasoning and trade-offs. Each question has a model answer; grade against it (2 = captures the key trade-off/risk, 1 = partial). Checkpoint for Week 3. Pass = ≥ 75% of max. See grading guide.
Q1. Dual repos. Prism is split into uengage-crm (monolith) and prism-services (serverless). Explain the split, why it exists, and one risk it introduces.
Model answer
The monolith historically does *everything* (APIs, dashboards, campaigns, loyalty, ~91 in-process crons). `prism-services` (Lambda + MSK Kafka, ap-south-1) was added **in front** to absorb spiky, high-throughput POS ingestion without risking the monolith — a deliberate mid-migration state. **Risk:** the two write to the **same MongoDB `uengage` DB and the same `crm_queue`**, so there is a **dual-writer** problem — no dedup/idempotency between the Lambda path and the monolith's ingestion controllers → possible duplicate orders / races / unclear source-of-truth. (Also: shared DB couples the two deployables' schemas.)Q2. The crm_queue seam. Why is crm_queue described as "the seam," and what property must the queue-drain design have to be safe?
Model answer
`crm_queue` is the single collection where **both** ingestion worlds converge (serverless `orders-consumer` λ + monolith ingestion controllers). It decouples "data in" from "processing." To be safe the drain must be **idempotent** (processing the same order twice must not double-count LTV/AOV or double-allocate points) and must correctly advance `status` (0 → processed). Today idempotency isn't enforced everywhere, and there's **no TTL** so `deleteCrmQueue.js` (every 2 days) manually prunes `status=1` + source-6 non-terminal events.Q3. Kafka / MSK. What does Kafka buy Prism, and what is missing from the consumer design?
Model answer
Kafka (`prism-ingest-orders`, batch 10) **decouples burst ingestion from slower downstream work** and lets the serverless tier absorb spikes. **Missing:** no DLQ / retry on the consumers — a failed message can crash the Lambda and be lost or duplicated on rebalance; the API uses `Date.now()` as the key (no idempotency); and the two WhatsApp topics (`whatsapp-messages`, `whatsapp-receipts`) have **stubbed, log-only consumers**, so that half of the pipeline isn't real yet.Q4. Dual datastore. Justify the MongoDB/MySQL split. Give the "rule of thumb" and one concrete failure the split causes.
Model answer
**MongoDB `uengage`** (maxPool 400, SECONDARY_PREFERRED) holds high-write queues, MIS/aggregations, journeys, ratings/NPS, config, chat — fast-changing, analytical, schema-flexible. **MySQL `addo_*`** (webPool 15 + jobsPool 10) holds transactional/joined data — orders, users/auth, and crucially **money**: `wallet`, `wallet_history`, `wallet_rules`, `crm_points_allocation_queue`. Rule of thumb: **Mongo = buffers/analytics, MySQL = money & identity.** Concrete failure: loyalty points span *both* stores with **no distributed transaction** → balance drift (E1.10). Also business-id type inconsistency (String vs Number) in Mongo breaks naive filters.Q5. Dual dashboards & three campaign pipelines. Why do these coexist, and how should an engineer decide where to make a change?
Model answer
They're the surface of an in-progress Next.js migration/merge. Dashboards: `dashboardController.js` (legacy, 50+ fns) vs `dashboardControllerNew.js` (unified, Redis-cached, Socket.io). Campaigns: `campaignController.js` (traditional SMS/FCM/Email/WhatsApp) vs `campaignControllerDynamic.js` (custom-code, `Custom_Code_Campaign_MIS`) vs `automatedCampaignController.js` (journeys). Decision rule: **trace the actual route/consumer the feature uses in production before editing**, prefer the "new/unified" path for new work, and check the Roadmap merges (Campaigns+Journeys → "Central Marketing Hub", A2.1) so you don't build on a soon-to-be-deprecated pipeline.Q6. Non-atomic points. Walk through a crash scenario in points allocation and explain the mitigation Prism uses (or should use).
Model answer
Flow: order → `crm_points_allocation_queue` (MySQL) → `crmPointsAllocationQueueCron.js` (2 min) → `handleWalletPointsFlow.js` updates MySQL `wallet` + inserts `wallet_transactions` + updates Mongo `customers`/`orders` + milestone. A crash **between** the MySQL wallet update and the Mongo order-mark leaves state inconsistent (points added but order not flagged, or vice-versa) → the E1.10 discrepancy class. Mitigation: **idempotency** (check an `order.points_allocated` flag before allocating), the queue's **5-retry / 15-min backoff**, and a **daily reconciliation job** enforcing `Total = Used + Available + Expired`. True 2-phase commit across Mongo+MySQL isn't available, so idempotency + reconciliation is the pragmatic answer (Saga pattern).Q7. In-process crons. What are the operational consequences of running ~91 crons inside the web process, and what mitigations already exist?
Model answer
Consequences: **cron CPU/DB load is coupled to API latency** (a heavy nightly job can slow requests); a crash in cron code can affect the web server; scaling the web tier also multiplies crons unless guarded. Existing mitigations: **separate MySQL `jobsPool` (10)** so crons don't starve `webPool` (15); **overlap protection** via `*_cron_status`/`cron_stats` lock docs (skip if previous run not marked done); reactive stuck detection (~20 min → alert email) + ~4 h force-close. Gaps: no circuit breaker / auto-recovery, single-process coupling remains. The clean fix is extracting crons into a dedicated worker deployment.Q8. Cron overlap protection. Design-review the current lock approach. Where does it break under horizontal scaling?
Model answer
Each cron checks a `status:false` flag in a status document before running and sets `true`/`false` around execution. This works for a **single process** but is **not a distributed lock**: if the monolith is scaled to N instances, N copies of each cron register and race on the same flag (read-modify-write is not atomic) → double execution (double points, duplicate sends). A correct design uses an atomic `findOneAndUpdate` claim with a lease/TTL, or moves crons to a single worker instance / external scheduler. See [Request-LifeCycle §4](../02-Architecture/Request-LifeCycle.md), map-04 §5.6.Q9. Data-plane vs control-plane. Map Prism's components onto the five planes and name the collection/cron that anchors each.
Model answer
**Ingestion:** `crm_queue` (Lambda `orders-consumer` + monolith ingestion crons). **Processing:** ingestion crons + `handleWalletPointsFlow` → Customer 360/Order data + `crm_points_allocation_queue`. **Intelligence:** `rfmSegments.js` (3 AM) → `rfm_metrics_tmp`; `aggregationProcess.js` → ~14 MIS collections. **Activation:** `sms_queue_*`/`whatsapp_queue_*`/`push_noti_queue_*` drained by their crons → providers. **Presentation:** `dashboard_mis` + `dashboardControllerNew`/`/stats`. See [High-Level-Architecture §3](../02-Architecture/High-Level-Architecture.md).Q10. WhatsApp reconciliation. Two WhatsApp code paths exist. Describe both and why volume reports (E1.3) can be wrong.
Model answer
Path A (real): monolith enqueues to `whatsapp_queue_YYYYMMDD`; `sendWhatsappMessageCron` (3 s, marketing) / `sendWhatsappTransactionalCron` (1 s, transactional) send via Gupshup/Meta and advance `process_status` (0→2→3→4). Path B (stubbed): Kafka `whatsapp-messages`/`whatsapp-receipts` consumers **log only**. There's **no visible delivery-receipt processor cron**, so `process_status=3` "pending webhook" items aren't fully reconciled. Counting sent/delivered from an incomplete pipeline (plus per-day collections and no rate limiting) yields inaccurate volume reports (E1.3). See map-04 §5.11.Q11. Design exercise. You're asked to make ingestion exactly-once. Sketch the minimal changes across prism-services and the monolith.