Data Infrastructure

TRADEOS.tech uses a layered data architecture that separates real-time streaming state from time-series history, with clear ownership boundaries and retention policies for each layer.

Data layers

Layer 1 — Redis Streams (real-time event bus)

Redis Streams (DB 0) is the primary inter-service communication medium for all real-time events:

Stream	Publisher	Consumers
market_data	Ingestion	Signal Engine
signals.raw	Signal Engine	Signal Modifier
signals.modified	Signal Modifier	Feasibility
intents	Feasibility	Execution
orders	Execution	Execution (internal)
fills.raw	Execution	TCA Monitor, Dashboard
positions	Execution	Dashboard, Metrics API
risk_events	Feasibility	Dashboard, Autonomous Agent
audit	All services	Autonomous Agent

Stream entries are retained up to a configured maxlen (trimmed at write time). Limits are tuned per environment — development runs with smaller buffers, production with larger ones for deeper history.

Consumer groups ensure exactly-once processing: each consumer explicitly acknowledges processed messages, and unacknowledged messages are automatically re-delivered after a timeout.

Layer 2 — Redis Cache (live state)

Redis DB 1 holds the current state of the system — the latest snapshot of values that services need to read frequently:

• Current market prices per symbol
• Current account balance and portfolio value
• Open position state
• Hero signals list
• Active policy profile parameters
• Win-rate and IC tracker state

This layer uses key-value and hash structures. Values are updated by the service that owns them and read by any service that needs them. TTLs are applied to volatile values to prevent stale state from persisting across restarts.

Layer 3 — Redis Ephemeral (volatile flags)

Redis DB 2 uses a volatile-ttl eviction policy. It holds short-lived state:

• Kill switch flags (autonomous agent, manual halt)
• Rate limit counters
• Circuit breaker state
• Deduplication keys for idempotent operations

All entries in DB 2 have explicit TTLs. This layer is not a source of truth — values here expire naturally and are re-established when needed.

Layer 4 — PostgreSQL (persistent records)

PostgreSQL holds all durable business records:

• Trade history — every filled order with full attribution chain
• Position snapshots — periodic portfolio snapshots for P&L history
• Backtest runs — job state, parameters, and results
• Policy profiles — risk parameter configurations (Conservative, Balanced, Aggressive)
• System config — live-tunable parameters (hero signals, thresholds) managed by the autonomous agent
• Signal outcomes — TCA results linking signals to their trade outcomes

Schema is managed via Alembic migrations. All migrations are idempotent and can be run multiple times safely.

Layer 5 — ClickHouse (OHLCV data lake)

ClickHouse holds the historical OHLCV time-series used by:

• Signal enrichment — ATR regime classification, volume percentile features, opening range breakout detection
• Backtest engine — replay of historical market data for strategy validation
• Autonomous agent — weekly re-optimization pulls per-symbol volatility regimes and momentum features
• Algorithm warm-start — bootstrap data for the ML algorithm suite (VAE, Gaussian Process, Thompson Sampling)

Data is ingested hourly from Coinbase public REST endpoints. Retention is enforced via ClickHouse TTL policies configured per resolution tier. The ClickHouse instance is managed as a separate project (tradeos-datalake) and connected via an external Docker network.

Data retention policy

Data type	Retention	Storage
Market data ticks	Rolling buffer, env-configured	Redis Streams
Signal events	Rolling buffer, env-configured	Redis Streams
Live prices / state	Until overwritten	Redis Cache
Kill switches / flags	TTL-based (short-lived)	Redis Ephemeral
Trade records	Indefinite	PostgreSQL
OHLCV candles (1m)	Configurable retention window	ClickHouse
OHLCV candles (1h)	Configurable retention window	ClickHouse
Backtest results	Indefinite	PostgreSQL

Observability

Each service exposes a /healthz (or /health) endpoint that returns its current status and key dependency health. The metrics API aggregates health across all services and exposes it to the dashboard.

Structured logging is used throughout. All log events include a service name, timestamp, correlation ID, and severity level. Critical events (risk limit breaches, circuit breaker activations, execution failures) are also written to the risk_events stream for real-time dashboard visibility.