dreamstack/STREAM_COMPOSITION.md

DreamStack Stream Composition

Compose independent apps through live signal streams. Each app is a node in a reactive dataflow graph — it can source signals, receive them, derive new ones, and re-stream the results.

Core Concepts

Source — an app that publishes signals to the relay:

stream counter on "ws://localhost:9100/peer/counter" {
  mode: signal,
  output: count, doubled
}

Receiver — an app that subscribes to a stream:

let counter = stream from "ws://localhost:9100/stream/counter"
text "Count: {counter.count}"

Composition — an app that is both receiver AND source:

let counter = stream from ".../stream/counter"
let events  = counter.count + stats.total       -- derived from upstream
stream metrics on ".../peer/metrics" { output: events }

URL Convention

Role	URL Pattern	Description
Source/Peer	ws://relay/peer/{channel}	Publishes signals to a named channel
Receiver	ws://relay/stream/{channel}	Subscribes to a named channel
Signaling	ws://relay/signal/{channel}	WebRTC signaling exchange

Each channel is independent. A source on /peer/counter is received at /stream/counter.

The select Clause

Filter which signals a receiver cares about:

-- Receive everything (default)
let counter = stream from "ws://.../stream/counter"

-- Receive only specific fields
let counter = stream from "ws://.../stream/counter" {
  select: count, doubled
}

What happens:

1. Client-side — _connectStream filters incoming state to only selected keys
2. Relay-side — Receiver sends a 0x33 SubscribeFilter frame; relay strips unwanted fields from signal frames before forwarding (saving bandwidth)

stream Declaration

stream <name> on "<relay-url>" {
  mode: signal,           -- signal | pixel
  output: field1, field2  -- which signals to publish (private signals excluded)
}

Key	Values	Description
mode	signal (default)	Signal mode streams JSON state; pixel mode streams RGBA frames
output	comma-separated names	Only listed signals are sent; unlisted ones stay local

Private signals — any signal not in output is never transmitted:

let count = 0           -- public (in output)
let doubled = count * 2 -- public (in output)
let message = "hello"   -- private (not in output)

stream counter on "..." { output: count, doubled }

stream from Expression

let <name> = stream from "<receiver-url>"
let <name> = stream from "<receiver-url>" { select: field1, field2 }

Returns a reactive signal proxy. Access fields with dot notation:

text "Count: {counter.count}"
let events = counter.count + stats.total

The proxy auto-updates when new frames arrive from the relay. All dependent derived signals and UI elements update reactively.

Chained Composition

Apps can be stacked in layers. Each layer receives, derives, and re-streams:

Layer 0 (Sources):
  Counter ──┐
  Clock  ───┤
  Stats  ───┘
              └──► Layer 1 (Aggregator):
                     compose-metrics
                     derives: uptime, events, status
                     re-streams on /peer/metrics
                                                    └──► Layer 2 (Dashboard):
                                                           compose-master
                                                           receives: metrics + mood

Layer 1 — compose-metrics.ds:

let counter = stream from ".../stream/counter"
let clock   = stream from ".../stream/clock"
let stats   = stream from ".../stream/stats"

let uptime = clock.seconds
let events = counter.count + stats.total
let status = "nominal"

stream metrics on ".../peer/metrics" {
  mode: signal, output: uptime, events, status
}

Layer 2 — compose-master.ds:

let metrics = stream from ".../stream/metrics" { select: uptime, events }
let mood    = stream from ".../stream/mood"    { select: mood, energy }

Bitstream Protocol

All frames share a 16-byte header:

┌──────┬───────┬──────┬───────────┬───────┬────────┬────────┐
│ type │ flags │ seq  │ timestamp │ width │ height │ length │
│  u8  │  u8   │ u16  │    u32    │  u16  │  u16   │  u32   │
└──────┴───────┴──────┴───────────┴───────┴────────┴────────┘

Signal Frame Types

Code	Name	Direction	Payload
0x30	SignalSync	source → relay → receivers	Full JSON state { count: 3, doubled: 6 }
0x31	SignalDiff	source → relay → receivers	Changed fields only { count: 4 }
0x32	SchemaAnnounce	source → relay	{ signals: ["count","doubled"], mode: "signal" }
0x33	SubscribeFilter	receiver → relay	{ select: ["count"] }

Pixel Frame Types

Code	Name	Payload
0x01	Pixels	Raw RGBA (width × height × 4 bytes)
0x03	DeltaPixels	XOR delta, RLE-compressed
0xF0	Keyframe	Reset receiver state

Flags

Bit	Name	Meaning
0	FLAG_INPUT	Frame is input (receiver → source)
1	FLAG_KEYFRAME	Full state, no delta
2	FLAG_COMPRESSED	Payload is compressed

Relay

cargo run -p ds-stream

The relay is a production-grade WebSocket router:

• Routes frames from sources/peers to receivers on the same channel
• Diff merging: Consolidates accumulated signal diffs into ONE catchup frame for late joiners
• Caches schema announcements (0x32) per channel
• Applies per-receiver signal filters (0x33) before forwarding
• Auto-creates channels on first connection
• Cleans up idle channels after grace period
• Source reconnect grace: cache preserved during reconnection window

Runtime Features

Diff Batching

Multiple signal changes in a single event loop tick are coalesced into one WebSocket frame:

-- Click fires 3 signal changes, but sends only 1 WS frame:
button "Happy" { click: mood = "happy"; color = "green"; energy += 10 }

Connection Status

The stream from proxy injects reactive connection metadata:

let counter = stream from "ws://localhost:9100/stream/counter"

-- Available fields:
-- counter._connected   boolean (true when WS is open)
-- counter._latency     RTT in milliseconds (ping/pong every 5s)
-- counter._frames      total frames received
-- counter._reconnects  reconnect attempt count

RTT Tracking

Both sources and receivers measure round-trip latency via periodic ping/pong (every 5s). The _latency field updates reactively on each measurement.

Relay Diff Merging

Late-joining receivers receive a single consolidated sync frame instead of replaying hundreds of individual diffs:

-- Before: receiver gets sync + 500 individual diffs
-- After:  receiver gets 1 merged sync frame with all changes applied

Running the Demo

# Terminal 1: Start relay
cargo run -p ds-stream

# Terminal 2-5: Start source apps
dreamstack stream examples/streaming-counter.ds --port 3000
dreamstack stream examples/streaming-clock.ds   --port 3002
dreamstack stream examples/streaming-stats.ds   --port 3003
dreamstack stream examples/streaming-mood.ds    --port 3004

# Terminal 6: Start aggregator (Layer 1)
dreamstack stream examples/compose-metrics.ds   --port 3006

# Terminal 7: Build and serve dashboard
dreamstack build examples/streaming-dashboard.ds -o /tmp/build-dash
python3 -m http.server 3007 -d /tmp/build-dash

Open http://localhost:3007 to see all streams composited into one dashboard.

Examples

File	Role	Channel	Signals
streaming-counter.ds	Source	/peer/counter	count, doubled
streaming-clock.ds	Source	/peer/clock	hours, minutes, seconds
streaming-stats.ds	Source	/peer/stats	total, average, max
streaming-mood.ds	Source	/peer/mood	mood, energy, color
streaming-dashboard.ds	Receiver	3 streams	Components + live data
compose-dashboard.ds	Receiver	4 streams	Flat dashboard
compose-metrics.ds	Receiver+Source	3 in, 1 out	uptime, events, status
compose-master.ds	Receiver	2 streams	Chained dashboard
timer-multi-action.ds	Timer	standalone	ticks, status, elapsed