Producer/Consumer Architecture in ZinTrust

Overview

ZinTrust employs a Split Architecture deployment model when running on serverless edge platforms like Cloudflare Workers. This architecture decouples job production (enqueueing) from job consumption (processing) to respect runtime limitations.

graph TD
    User[User / Client] -->|HTTP Request| Edge[Cloudflare Workers Edge]
    Edge -->|Queue.add / monitor reads| RPC[Redis RPC]
    RPC -->|BullMQ + Redis| Redis[(Redis)]
    RPC -->|Process Job| Consumer[Node.js worker process]
    Consumer -->|Store Result| DB[(Database)]

• Producer (Edge): Runs on Cloudflare Workers. Handles ephemeral HTTP requests, enqueues background jobs, and returns fast responses.
• Consumer (Origin): Runs in a Node.js runtime that can maintain persistent Redis/BullMQ connections. This can be Docker, a VM, Fargate, a process manager, or the zin redis-rpc server. It does not have to be a Cloudflare Container.

Why Split?

Cloudflare Workers and similar edge runtimes have specific limitations unsuitable for traditional queue workers:

1. No Persistent TCP: Workers cannot keep a Redis connection open indefinitely for BLPOP/subscription.
2. Request Lifecycle: Workers are meant to spin down immediately after sending a response. Background processing is limited.
3. Global Scope: BullMQ and similar libraries often rely on global state or socket pooling incompatible with V8 isolate freeze/thaw cycles.

Configuration

To support this, ZinTrust uses the RUNTIME_MODE and feature flags.

Producer (Cloudflare)

Settings for wrangler.jsonc or .env.producer:

RUNTIME_MODE=cloudflare-workers
WORKER_ENABLED=false
QUEUE_ENABLED=true
USE_REDIS_PROXY=true
REDIS_RPC_URL=https://queues.example.com

Consumer (Container)

Settings for docker-compose.yml or .env:

RUNTIME_MODE=containers
WORKER_ENABLED=true
WORKER_AUTO_START=true
QUEUE_ENABLED=true

For Docker container stacks, keep DOCKER_WORKER unset (or false) by default.

Redis RPC

Install @zintrust/redis-rpc in the backend project that owns Redis credentials:

npm install @zintrust/redis-rpc
zin redis-rpc

Redis RPC becomes active in callers only when both USE_REDIS_PROXY=true and REDIS_RPC_URL are set. That explicit two-variable gate prevents accidental proxy selection during local direct-Redis development.

Local Development

You can simulate this split architecture locally using the CLI:

zin dev --mode=split

This spawns two processes:

1. Web: The API server (configured as Producer)
2. Worker: The background process (configured as Consumer)