Independent lab · not an employer deliverable

FlowForge

Event-Driven Systems Observatory

A control room for producers, brokers, consumers, retries, and the dead-letter path.

Problem

Event-driven systems fail in motion: duplicate deliveries, poison messages, and retries that hide the original fault. FlowForge makes those states visible so operators can reason about at-least-once processing without reading raw broker logs.

Interactive demo

Seeded event log with operator controls. Counts and timelines are simulated example data.

FlowForge · event observatorySeeded demo log

Producers

gateway · batch-runner · webhooks

Broker

topics · partitions · consumer groups

Consumers

surveillance · usage-index · scheduler · catalog · reports

Storage

PostgreSQL · Oracle · Redis keys

Ops

retries · DLQ · replay

completed

in flight

failed

DLQ

ID	Type	Consumer	Status	Retries
evt_18f2	TradeIngested	surveillance	completed	0/3
evt_44aa	LibraryUsage	usage-index	processing	0/3
evt_90c1	CatalogSynced	catalog	failed	2/3
evt_12be	ReportRequested	reports	queued	0/3
evt_77d0	ScheduleRequested	scheduler	completed	1/3
evt_03ea	TradeIngested	surveillance	queued	0/3
evt_dlq1	CatalogSynced	catalog	dead-lettered	3/3
evt_61af	LibraryUsage	usage-index	completed	0/3

Inspector

evt_90c1

failed

Producer · webhooks

Consumer · catalog

Key · shop:441:product:19

Updated · 2026-10-29T06:20:00Z

{
  "merchant": 441,
  "sku": "AX-19",
  "action": "upsert"
}

2026-10-29T06:12:00Z · queued — Shopify webhook received.
2026-10-29T06:14:00Z · processing — First attempt.
2026-10-29T06:16:00Z · retried — Google Shopping 429 — retry 1.
2026-10-29T06:20:00Z · failed — Schema mismatch on product payload.

Dead-letter queue (1)

Architecture

Producers publish to a broker. Workers pull messages, enforce idempotency keys, and write to storage. Failures retry with a bounded policy, then land in a dead-letter queue. Replay re-enqueues an eligible message under the same key. This demo is a client-side simulation of that control loop.

Producer → broker queue → consumer (idempotent handler) → store / side effects → retry policy → dead-letter queue → optional replay.

Decisions

— Surface status as a first-class field (queued, processing, completed, retried, failed, dead-lettered).
— Keep retry and replay as explicit operator actions rather than hidden background magic.
— Derive dashboard counts from the same in-memory event log the inspector uses.

Trade-offs

— At-least-once delivery is operationally simpler than exactly-once, and it requires idempotent consumers.
— Aggressive retries recover transient faults and can amplify load during an outage (backpressure matters).
— Replaying from a DLQ restores availability, and it can reintroduce side effects if handlers are not safe.

Failure modes

— Poison payloads exhaust retries and isolate in the DLQ.
— Duplicate deliveries with a missing idempotency key double-apply a side effect.
— Unbounded concurrency without backpressure overwhelms downstream storage.

Evaluation

The observatory is evaluated by whether a retry, fail, and replay sequence produces a coherent history for a single event—not by fabricated throughput claims.

Production extensions

Proposed — not implemented in this portfolio.

— Replace the in-memory broker with Kafka and transactional outbox writes.
— Add consumer lag, retry budgets, and tracing spans per hop.
— Gate replay behind authorization and a dry-run payload diff.

Python · Go · Kafka · asynchronous workers · PostgreSQL · Redis · idempotency · retries