Event-Driven FastAPI: Building a 10M events/day Automation Engine
Architecture notes from OmniFlow — idempotency keys, outbox pattern, backoff strategies and horizontal worker autoscaling that survived Black Friday.
@app.post("/webhooks/dispatch", status_code=202)
async def dispatch(evt: Event, idem: str = Header(...)):
if await redis.sismember("idem", idem):
return {"deduped": True}
await outbox.save(evt, idem) # transactional
await broker.publish(evt, routing_key=evt.tenant_tier)Why events, not REST calls
OmniFlow orchestrates webhooks across thousands of tenants. Synchronous REST chaining meant one slow downstream took down the whole chain. Moving to events decoupled producers from consumers and let us retry intelligently.
The core loop is simple: FastAPI edge accepts and validates, Postgres outbox guarantees durability, RabbitMQ routes by tenant tier, Python workers execute with exponential backoff + jitter.
Lessons from 10M events/day
Idempotency is non-negotiable. Every event carries an idempotency key stored in Redis with a 24h TTL. Retries without it double-charge customers.
We run three queue priorities: realtime (webhooks, <50ms dispatch), default (automations), and bulk (imports, reports). Autoscaling is driven by queue age, not depth — depth spikes during batches but age tells you if SLAs are burning.
verifiedKey takeaways
- arrow_rightOutbox pattern > dual-write every time.
- arrow_rightScale on queue age, not queue depth.
- arrow_rightDedup at the edge saves your database.