Skip to content

Deployment

How each repo ships. Two very different pipelines: the monolith runs long-lived under PM2; the serverless tier deploys via Serverless Framework + GitHub Actions on push to main.

Validation legend: ✅ verified · 📄 inferred · 💻 config file · ⚠️ risk/gotcha · ⚰️ disabled

1. Pipeline overview

flowchart LR
    subgraph mono["uengage-crm · monolith"]
        direction TB
        DEV1[developer]
        SRV[(prod server)]
        ENV1[.env.prod on server]
        PM2["PM2<br/>crm_ecosystem.config.js<br/>app: crm"]
        DEV1 -->|git pull| SRV
        ENV1 -.reads at start.-> PM2
        SRV -->|pm2 start/restart --env production| PM2
        PM2 -->|node index.js<br/>PORT 4000| RUN1[running monolith + 91 crons]
    end

    subgraph svc["prism-services · serverless"]
        direction TB
        DEV2[developer]
        MAIN[(main branch)]
        GHA["GitHub Actions<br/>deploy-prod.yml"]
        SLS["serverless deploy<br/>--stage prod"]
        CF["CloudFormation<br/>prism-prod / prism-whatsapp-prod"]
        DEV2 -->|push to main| MAIN
        MAIN -->|trigger| GHA
        GHA -->|node 22, yarn install| SLS
        SLS --> CF
        CF --> L1[Lambdas + API Gateway]
        CF --> L2[MSK event source mappings]
    end

2. Monolith (uengage-crm) — PM2

Process manager: PM2, driven by crm_ecosystem.config.js (single app crm, script ./index.js, watch: false).

Environments (two blocks in the ecosystem file):

dev (env_development) prod (env_production)
PORT 3000 4000 ✅ (crm_ecosystem.config.js:36)
NODE_ENV dev prod
Mongo URI mongodb://0.0.0.0:27017 (inline) process.env.DB_URL_STRING (from .env.prod)
SQL creds process.env.* process.env.*
CONNECTION_LIMIT 20 20
Redis host ElastiCache (hardcoded) ElastiCache (hardcoded) ⚠️
MAIL_* recipients single (prerna@) multi-recipient lists

Env sourcing ✅: crm_ecosystem.config.js:4 does dotenv.config({ path: .env.prod }) at PM2 start/restart time. .env.prod is gitignored and lives on the server next to the config. logger/index.js selects dev vs prod logger by NODE_ENV.

npm scripts (package.json): - start → NODE_ENV=prod node index.js - start:dev → NODE_ENV=dev nodemon index.js - start:prod → NODE_ENV=prod node index.js

Deploy sequence (typical): 

# on the prod server
git pull origin main
npm install                      # if deps changed
pm2 start crm_ecosystem.config.js --env production   # first boot
pm2 restart crm --update-env --env production         # subsequent deploys
pm2 logs crm                     # tail
pm2 save                         # persist process list across reboots

Gotchas ⚠️ - 91 crons boot in-process with the web server (index.js). A deploy restart re-registers all of them; overlapping/duplicate cron runs can occur on the restart boundary. index.js:40 warns to comment crons if attaching a DB to prod. See Queues. - No zero-downtime story documented — a pm2 restart briefly drops the listener. PM2 cluster mode is not configured (single instance). - Redis host is hardcoded, so a Redis migration requires a code change, not an env change.

3. prism-services — Serverless Framework + GitHub Actions

Trigger ✅: push to main (.github/workflows/deploy-prod.yml:5).

CI steps (deploy-prod.yml): 1. actions/checkout@v3 2. actions/setup-node@v3 → Node 22, npm cache 3. yarn (install deps) 4. serverless/github-action@v3.2 → deploy --stage prod -c serverless-orders.yml

Runtime/infra (from serverless YMLs): - Region ap-south-1 (Mumbai) - Runtime nodejs22.x - VPC: security group sg-0a1941fefacf9f748; subnets subnet-90c721dd, subnet-193c9a70, subnet-211ee85a - MSK cluster arn:aws:kafka:ap-south-1:...:cluster/flash/... - Event source mappings: prism-ingest-orders (batch 10, window 1s), whatsapp-messages / whatsapp-receipts (batch 20, window 5s), startingPosition: LATEST

CloudFormation stacks produced: - prism-prod (orders-api, orders-consumer) - prism-whatsapp-prod (api-send, api-receipts, consumer-send, consumer-receipts)

AWS creds: GitHub repo secrets SERVERLESS_AWS_ACCESS_KEY_ID / SERVERLESS_AWS_SECRET_ACCESS_KEY (deploy-prod.yml:28-29).

3.1 ⚠️ WhatsApp service is NOT in CI

deploy-prod.yml deploys only serverless-orders.yml. There is no step for serverless-whatsapp.yml ✅ (verified — the workflow has a single deploy step). Consequences: - Changes to the WhatsApp Lambdas do not ship automatically on push to main — they require a manual serverless deploy --stage prod -c serverless-whatsapp.yml. - Easy to have main and deployed WhatsApp Lambdas drift silently. - Fix: add a second deploy step (or matrix) for the whatsapp YML.

3.2 ⚠️ Hardcoded creds in YMLs

MONGODB_URI (with password) and KAFKA_BROKER_URL are plaintext in the serverless YMLs (serverless-orders.yml:51, serverless-whatsapp.yml:67,89). Anyone with repo read access has prod Mongo. Fix: move to AWS Secrets Manager / SSM Parameter Store and reference via ${ssm:...}. See Technical Debt.

4. Rollback & ops notes

Monolith

  • Rollback = redeploy previous commit: git checkout <prev-sha> && pm2 restart crm --update-env --env production.
  • Crash loop: pm2 logs crm --err; check Mongo/Redis reachability (Mongo failure exit(1)s the process; Redis failure only logs — see index.js startup).
  • Roll back a single bad cron: comment its require() in index.js and restart (many crons already commented, index.js:69,77,105,…). ⚰️
  • Verify health: pm2 status, then watch pool_stats JSON logs (MySQL pool) and combined-%DATE%.log. See Debugging.

prism-services

  • Rollback: re-run the workflow on a previous main commit, or locally serverless deploy --stage prod -c serverless-orders.yml from the older checkout. Serverless/CloudFormation keeps the prior stack; a failed deploy auto-rolls-back the CFN change set.
  • Rollback a single Lambda: serverless rollback --stage prod or serverless deploy function -f orders-consumer.
  • ⚠️ No DLQ/retry: a poison message re-drives the consumer until it ages off Kafka; watch CloudWatch for repeated consumer errors. Since startingPosition: LATEST, redeploying does not replay history.
  • ⚠️ Dual-writer awareness: the orders-consumer writes crm_queue alongside the monolith — a deploy of one tier does not coordinate with the other.

Ops checklist per release

Step Monolith prism-services
Pre confirm .env.prod present on server confirm GitHub secrets set
Deploy pm2 restart --update-env push to main (orders) + manual whatsapp
Verify pm2 status + logs + pool_stats CloudWatch logs per Lambda
Rollback checkout prev SHA + restart serverless rollback / redeploy prev