dreamstack/docs/panel-ir-spec.md

# DreamStack Panel IR — Dynamic UI Streaming to ESP32

Stream DreamStack apps to ESP32-P4 touchscreen panels as a compact **Panel IR** over WebSocket. The on-device LVGL runtime renders the UI locally. Touch is instant. New UIs arrive dynamically — no reflashing.

## Architecture

```
HUB (Pi / laptop)                              ESP32-P4 PANEL (Waveshare 10.1")
──────────────────                              ─────────────────────────────────

DreamStack compiler                             Panel Runtime (~500 lines C)
  ds compile app.ds --target panel                LVGL 9 graphics library
  ↓                                               ↓
ir_emitter.rs → Panel IR (JSON)  ──→ WiFi 6 ──→  Parse IR → create widgets
                                                  Bind signal reactivity
                                                  Handle touch locally (< 5ms)

Signal update: { "t":"sig", "s":{"0":75} } ──→   Update bound labels instantly
Touch event:   { "t":"evt", "n":3, "e":"click" } ←── LVGL callback fires
New screen:    Full IR blob              ──→       Destroy old UI, build new
```

## Why not pixel streaming?

| | Pixel streaming | Panel IR (this spec) |
|---|---|---|
| Touch latency | 30-50ms (network round-trip) | **< 5ms (local)** |
| WiFi disconnect | Screen freezes | **App stays interactive** |
| Bandwidth | 20-50 Kbps continuous | ~100 bytes/event |
| Dynamic UIs | ✅ | ✅ |
| Hub CPU load | Renders for all panels | **Near zero** |
| Offline | ❌ | ✅ (for local signals) |

## Message Protocol

Three message types over WebSocket (JSON):

### 1. `ui` — Full UI tree

Sent when app loads, screen changes, or hub pushes a new app.

```json
{
  "t": "ui",
  "signals": [
    { "id": 0, "v": 72, "type": "int" },
    { "id": 1, "v": true, "type": "bool" },
    { "id": 2, "v": "Kitchen", "type": "str" }
  ],
  "root": {
    "t": "col", "gap": 10, "pad": 20,
    "c": [
      { "t": "lbl", "id": 0, "text": "{2}: {0}°F", "size": 24 },
      { "t": "btn", "id": 1, "text": "Lights",
        "on": { "click": { "op": "toggle", "s": 1 } } },
      { "t": "sld", "id": 2, "min": 60, "max": 90, "bind": 0 }
    ]
  },
  "derived": [
    { "id": 3, "expr": "s0 * 2", "deps": [0] }
  ]
}
```

### 2. `sig` — Signal update (hub → panel)

```json
{ "t": "sig", "s": { "0": 75 } }
```

### 3. `evt` — Event (panel → hub)

```json
{ "t": "evt", "n": 1, "e": "click" }
```

## IR Node Types

| IR type | DreamStack source | LVGL widget | Key props |
|---|---|---|---|
| `col` | `column [...]` | `lv_obj` + `LV_FLEX_FLOW_COLUMN` | gap, pad, bg |
| `row` | `row [...]` | `lv_obj` + `LV_FLEX_FLOW_ROW` | gap, pad |
| `stk` | `stack [...]` | `lv_obj` (absolute pos) | — |
| `lbl` | `text "..."` | `lv_label` | text, color, size |
| `btn` | `button "..." {}` | `lv_btn` + `lv_label` | text, on.click |
| `inp` | `input` | `lv_textarea` | placeholder, bind |
| `sld` | `slider` | `lv_slider` | min, max, bind |
| `img` | `image` | `lv_img` | src |
| `bar` | `progress` | `lv_bar` | min, max, bind |
| `sw`  | `toggle` | `lv_switch` | bind |
| `lst` | `list [...]` | `lv_list` | — |
| `pnl` | `panel [...]` | `lv_obj` (styled card) | title, bg, radius |

## Signal Binding

Text fields use `{N}` for signal interpolation:

```
"text": "{2}: {0}°F"    →    "Kitchen: 72°F"
                              signal 2  signal 0
```

When signal 0 changes to 75, the runtime re-expands the template and calls `lv_label_set_text()`. Only affected labels update.

Sliders and inputs use `"bind": N` for two-way binding. When the user moves a slider, the runtime updates signal N locally AND sends a `sig` message to the hub.

## Event Actions

Simple operations execute locally on the ESP32 (instant). Complex logic forwards to the hub.

| Action | IR | Runs on |
|---|---|---|
| Set value | `{ "op": "set", "s": 0, "v": 5 }` | ESP32 (local) |
| Toggle bool | `{ "op": "toggle", "s": 1 }` | ESP32 (local) |
| Increment | `{ "op": "inc", "s": 0 }` | ESP32 (local) |
| Decrement | `{ "op": "dec", "s": 0 }` | ESP32 (local) |
| Navigate | `{ "op": "nav", "screen": "settings" }` | Hub → sends new IR |
| Remote call | `{ "op": "remote", "name": "fetch_weather" }` | Hub → result as sig |

## Styling

Each node can have optional style props:

```json
{
  "t": "lbl", "id": 0, "text": "Hello",
  "style": {
    "bg": "#1a1a2e",
    "fg": "#e0e0ff",
    "size": 18,
    "radius": 8,
    "pad": 12,
    "align": "center",
    "font": "bold"
  }
}
```

The runtime maps these to `lv_obj_set_style_*()` calls.

## Implementation Plan

### Phase 1: Compiler Backend

**New file:** `compiler/ds-codegen/src/ir_emitter.rs`

```rust
pub fn emit_panel_ir(program: &Program, graph: &SignalGraph) -> String {
    // Walk AST → build IR JSON
    // Map signal names → integer IDs
    // Convert OnHandler → action opcodes
    // Convert string interpolations → {N} references
}
```

**Modify:** `compiler/ds-cli/src/main.rs` — add `--target panel` flag

### Phase 2: ESP32 LVGL Runtime

**New files in** `devices/waveshare-p4-panel/main/`:

| File | Lines | Purpose |
|---|---|---|
| `ds_runtime.h` | ~60 | API: build, destroy, signal_update, event_send |
| `ds_runtime.c` | ~400 | JSON parser (cJSON), widget factory, signal table, text formatter, event dispatch |

Core functions:
```c
void ds_ui_build(const char *ir_json);     // Parse IR, create LVGL tree
void ds_ui_destroy(void);                  // Tear down for screen change
void ds_signal_update(uint16_t id, const char *value);  // Update + refresh
void ds_event_send(uint16_t node_id, const char *event); // Send to hub
```

### Phase 3: Hub Server

**New file:** `engine/ds-stream/src/panel_server.rs`

WebSocket server that:
1. Compiles `.ds` → IR on startup
2. Sends IR to connecting panels
3. Forwards signal diffs bidirectionally
4. Watches `.ds` file → recompile → push new IR (hot reload)

### Phase 4: Pixel fallback (already built)

The existing `ds_codec.c` + `ds_protocol.h` remain as a fallback for cases where pixel streaming IS needed (streaming non-DreamStack content, camera feeds, etc).

## Hardware

| Component | Role | Price |
|---|---|---|
| Waveshare ESP32-P4-WIFI6 10.1" | Panel display + runtime | ~$85 |
| Raspberry Pi 5 (or any Linux box) | Hub: compile + serve IR | ~$60 |
| **Total POC** | | **~$145** |

## Example: Complete Flow

### 1. Write the app
```
// kitchen.ds
let temperature = 72
let lights_on = true

view main {
  text { "Kitchen: {temperature}°F" }
  button "Lights" {
    on click { lights_on = !lights_on }
  }
  slider { min: 60, max: 90, bind: temperature }
}
```

### 2. Compile to IR
```bash
ds compile kitchen.ds --target panel > kitchen.ir.json
```

### 3. Hub serves it
```bash
ds serve --panel kitchen.ds --port 9100
# Panel connects → receives IR → renders UI
# Tap button → lights_on toggles locally
# Move slider → temperature updates locally + syncs to hub
# Hub pushes weather API data → { "t":"sig", "s":{"0": 68} }
# Panel label updates: "Kitchen: 68°F"
```

### 4. Push a new app
```bash
ds serve --panel settings.ds --port 9100
# Hub sends new IR → panel destroys kitchen UI → builds settings UI
# No reflash. Instant.
```