Messaging Architecture

All communication between TRADEOS.tech services flows through a single message bus: Redis Streams. This is a deliberate architectural choice with significant operational and correctness implications.

Why Redis Streams

The core requirements for the TRADEOS.tech message bus were:

1. Local-first — no cloud dependencies; all data stays on the operator's machine
2. Ordered and persistent — messages must survive process crashes without loss
3. Replayable — any historical sequence of events must be replayable for audit and debugging
4. Consumer groups — critical services must not double-process the same message
5. Low memory footprint — must run alongside all other services on a single machine

Redis Streams satisfies all five. Apache Kafka would also satisfy them technically but requires 1–4 GB of JVM heap — incompatible with single-machine deployment. NATS JetStream is a viable alternative but adds a component when Redis is already required for state management.

Message persistence

Redis is configured with append-only file (AOF) persistence. Every write to every stream is synced to disk before Redis acknowledges the write. If the process crashes, in-flight messages are recovered from the AOF log on restart. No messages are lost.

Stream design

The trade path follows a strict pipeline of named streams:

market_data.raw
  └─► signals.raw
        └─► signals.qualified
              └─► signals.modified
                    └─► trade.intents
                          └─► trade.approved
                                └─► fills
                                      └─► signal.outcomes
                                            └─► metrics.snapshot

Each arrow represents a stream. Each service reads from one stream and writes to the next. No service skips a stage. No service reads from a stage it does not own.

Consumer groups and delivery guarantees

Services on the critical path (feasibility, execution) use Redis consumer groups with manual acknowledgment (XACK). A message is not considered processed until the service explicitly acknowledges it. If the service crashes before acknowledging, the message is reassigned to another consumer in the group (or the same service on restart) — providing at-least-once delivery on the trade path.

Signal-reading services (signal engine, signal modifier) use independent read offsets rather than consumer groups, allowing multiple signal processors to consume the same market data concurrently.

Memory management

Each stream is bounded by a maximum length (MAXLEN). When the limit is reached, the oldest entries are trimmed. This bounds memory usage while retaining enough history for the replay and debugging use cases. Critical streams (trade approval, fills) are configured with larger buffers than informational streams.

Operational inspection

The entire system state is visible through standard Redis commands. Any stream can be inspected, tailed, or replayed in a terminal:

# Watch the live signal stream
docker compose exec redis redis-cli XREAD COUNT 10 STREAMS signals.qualified 0

# View recent fills
docker compose exec redis redis-cli XREVRANGE fills + - COUNT 20

# Check stream depths
docker compose exec redis redis-cli XLEN trade.intents

This means debugging requires no special tooling — any operator familiar with Redis can observe the full system state in real time.

What this means operationally

• No silent message loss — AOF persistence means a crash never loses a message in transit
• No cloud egress — all messages flow through localhost Redis; nothing leaves the machine
• Deterministic replay — stream history can be fed back through the pipeline to reproduce historical decisions exactly (see Deterministic Replay)
• Simple failure model — if Redis goes down, the pipeline pauses cleanly; no message is processed twice or lost; services resume automatically when Redis comes back