dreamstack/docs/panel-ir-spec.md

DreamStack Panel IR — Dynamic UI Streaming to ESP32

Stream DreamStack apps to ESP32-P4 touchscreen panels as a compact Panel IR over WebSocket. The on-device LVGL runtime renders the UI locally. Touch is instant. New UIs arrive dynamically — no reflashing.

Architecture

HUB (Pi / laptop)                              ESP32-P4 PANEL (Waveshare 10.1")
──────────────────                              ─────────────────────────────────

DreamStack compiler                             Panel Runtime (~500 lines C)
  ds compile app.ds --target panel                LVGL 9 graphics library
  ↓                                               ↓
ir_emitter.rs → Panel IR (JSON)  ──→ WiFi 6 ──→  Parse IR → create widgets
                                                  Bind signal reactivity
                                                  Handle touch locally (< 5ms)

Signal update: { "t":"sig", "s":{"0":75} } ──→   Update bound labels instantly
Touch event:   { "t":"evt", "n":3, "e":"click" } ←── LVGL callback fires
New screen:    Full IR blob              ──→       Destroy old UI, build new

Why not pixel streaming?

	Pixel streaming	Panel IR (this spec)
Touch latency	30-50ms (network round-trip)	< 5ms (local)
WiFi disconnect	Screen freezes	App stays interactive
Bandwidth	20-50 Kbps continuous	~100 bytes/event
Dynamic UIs	✅	✅
Hub CPU load	Renders for all panels	Near zero
Offline	❌	✅ (for local signals)

Message Protocol

Three message types over WebSocket (JSON):

1. ui — Full UI tree

Sent when app loads, screen changes, or hub pushes a new app.

{
  "t": "ui",
  "signals": [
    { "id": 0, "v": 72, "type": "int" },
    { "id": 1, "v": true, "type": "bool" },
    { "id": 2, "v": "Kitchen", "type": "str" }
  ],
  "root": {
    "t": "col", "gap": 10, "pad": 20,
    "c": [
      { "t": "lbl", "id": 0, "text": "{2}: {0}°F", "size": 24 },
      { "t": "btn", "id": 1, "text": "Lights",
        "on": { "click": { "op": "toggle", "s": 1 } } },
      { "t": "sld", "id": 2, "min": 60, "max": 90, "bind": 0 }
    ]
  },
  "derived": [
    { "id": 3, "expr": "s0 * 2", "deps": [0] }
  ]
}

2. sig — Signal update (hub → panel)

{ "t": "sig", "s": { "0": 75 } }

3. evt — Event (panel → hub)

{ "t": "evt", "n": 1, "e": "click" }

IR Node Types

IR type	DreamStack source	LVGL widget	Key props
col	column [...]	lv_obj + LV_FLEX_FLOW_COLUMN	gap, pad, bg
row	row [...]	lv_obj + LV_FLEX_FLOW_ROW	gap, pad
stk	stack [...]	lv_obj (absolute pos)	—
lbl	text "..."	lv_label	text, color, size
btn	button "..." {}	lv_btn + lv_label	text, on.click
inp	input	lv_textarea	placeholder, bind
sld	slider	lv_slider	min, max, bind
img	image	lv_img	src
bar	progress	lv_bar	min, max, bind
sw	toggle	lv_switch	bind
lst	list [...]	lv_list	—
pnl	panel [...]	lv_obj (styled card)	title, bg, radius

Signal Binding

Text fields use {N} for signal interpolation:

"text": "{2}: {0}°F"    →    "Kitchen: 72°F"
                              signal 2  signal 0

When signal 0 changes to 75, the runtime re-expands the template and calls lv_label_set_text(). Only affected labels update.

Sliders and inputs use "bind": N for two-way binding. When the user moves a slider, the runtime updates signal N locally AND sends a sig message to the hub.

Event Actions

Simple operations execute locally on the ESP32 (instant). Complex logic forwards to the hub.

Action	IR	Runs on
Set value	{ "op": "set", "s": 0, "v": 5 }	ESP32 (local)
Toggle bool	{ "op": "toggle", "s": 1 }	ESP32 (local)
Increment	{ "op": "inc", "s": 0 }	ESP32 (local)
Decrement	{ "op": "dec", "s": 0 }	ESP32 (local)
Navigate	{ "op": "nav", "screen": "settings" }	Hub → sends new IR
Remote call	{ "op": "remote", "name": "fetch_weather" }	Hub → result as sig

Styling

Each node can have optional style props:

{
  "t": "lbl", "id": 0, "text": "Hello",
  "style": {
    "bg": "#1a1a2e",
    "fg": "#e0e0ff",
    "size": 18,
    "radius": 8,
    "pad": 12,
    "align": "center",
    "font": "bold"
  }
}

The runtime maps these to lv_obj_set_style_*() calls.

Implementation Plan

Phase 1: Compiler Backend

New file: compiler/ds-codegen/src/ir_emitter.rs

pub fn emit_panel_ir(program: &Program, graph: &SignalGraph) -> String {
    // Walk AST → build IR JSON
    // Map signal names → integer IDs
    // Convert OnHandler → action opcodes
    // Convert string interpolations → {N} references
}

Modify: compiler/ds-cli/src/main.rs — add --target panel flag

Phase 2: ESP32 LVGL Runtime

New files in devices/waveshare-p4-panel/main/:

File	Lines	Purpose
ds_runtime.h	~60	API: build, destroy, signal_update, event_send
ds_runtime.c	~400	JSON parser (cJSON), widget factory, signal table, text formatter, event dispatch

Core functions:

void ds_ui_build(const char *ir_json);     // Parse IR, create LVGL tree
void ds_ui_destroy(void);                  // Tear down for screen change
void ds_signal_update(uint16_t id, const char *value);  // Update + refresh
void ds_event_send(uint16_t node_id, const char *event); // Send to hub

Phase 3: Hub Server

New file: engine/ds-stream/src/panel_server.rs

WebSocket server that:

1. Compiles .ds → IR on startup
2. Sends IR to connecting panels
3. Forwards signal diffs bidirectionally
4. Watches .ds file → recompile → push new IR (hot reload)

Phase 4: Pixel fallback (already built)

The existing ds_codec.c + ds_protocol.h remain as a fallback for cases where pixel streaming IS needed (streaming non-DreamStack content, camera feeds, etc).

Hardware

Component	Role	Price
Waveshare ESP32-P4-WIFI6 10.1"	Panel display + runtime	~$85
Raspberry Pi 5 (or any Linux box)	Hub: compile + serve IR	~$60
Total POC		~$145

Example: Complete Flow

1. Write the app

// kitchen.ds
let temperature = 72
let lights_on = true

view main {
  text { "Kitchen: {temperature}°F" }
  button "Lights" {
    on click { lights_on = !lights_on }
  }
  slider { min: 60, max: 90, bind: temperature }
}

2. Compile to IR

ds compile kitchen.ds --target panel > kitchen.ir.json

3. Hub serves it

ds serve --panel kitchen.ds --port 9100
# Panel connects → receives IR → renders UI
# Tap button → lights_on toggles locally
# Move slider → temperature updates locally + syncs to hub
# Hub pushes weather API data → { "t":"sig", "s":{"0": 68} }
# Panel label updates: "Kitchen: 68°F"

4. Push a new app

ds serve --panel settings.ds --port 9100
# Hub sends new IR → panel destroys kitchen UI → builds settings UI
# No reflash. Instant.