dreamstack/docs/explorations.md

# DreamStack Hardware Explorations

> Research notes on form factors, display technologies, and touch input methods for DreamStack-powered surfaces.

---

## 1. USB Dongle (Chromecast-like)

The DreamStack relay protocol + delta codec already does 90% of the work. A dongle receives the bitstream and outputs to HDMI.

### Path A: ESP32-S3 HDMI Dongle (~$15 DIY)

| Component | Part | Cost |
|-----------|------|------|
| SoC | ESP32-S3-WROOM-1 (N16R8) | ~$4 |
| HDMI output | CH7035B or ADV7513 HDMI encoder IC | ~$3 |
| USB-C power | Standard power-only connector | ~$0.50 |
| PCB + passives | Custom PCB (JLCPCB) | ~$5 for 5 boards |
| HDMI connector | Type-A male or mini-HDMI | ~$1 |

- ESP32-S3 LCD parallel interface → HDMI encoder IC → HDMI out
- WiFi connects to DreamStack relay, receives delta-compressed frames
- Resolution limit: ~480×320 smooth, 800×600 at lower FPS
- Input via BLE HID remote or HDMI CEC (pin 13)

### Path B: Linux Stick / Allwinner (~$25-40)

MangoPi MQ-Pro / Radxa Zero form factor:

| Component | Part | Cost |
|-----------|------|------|
| SoC | Allwinner H616/H618 (HDMI built-in) | ~$15 module |
| RAM | 512MB DDR3 onboard | included |
| WiFi | RTL8723CS | ~$3 |
| Storage | 8GB eMMC or SD | ~$3 |

- Runs minimal Linux (Buildroot), headless browser or C receiver writing to `/dev/fb0`
- Native HDMI — no encoder IC needed
- Full DreamStack JS runtime in headless Chromium/WPE-WebKit
- CEC for remote control

### Path C: Pi Zero 2 W (~$15, recommended MVP)

Best for proving the concept immediately — $15, mini-HDMI out, WiFi, runs DreamStack natively.

```
Laptop                    WiFi/LAN           Pi Zero 2 W (in 3D-printed HDMI case)
──────                    ─────────          ─────────────────────────────────────
dreamstack dev app.ds                        headless browser / ds_runtime
  → relay-bridge ──────── WebSocket ───────→   → HDMI out to TV
                          ←── CEC/BLE ←────  remote control
```

### Off-the-shelf stick computers

| Device | Price | HDMI | WiFi | Notes |
|--------|-------|------|------|-------|
| Raspberry Pi Zero 2 W | $15 | Mini-HDMI ✅ | ✅ | Best form factor |
| MangoPi MQ-Pro (RISC-V) | $20 | HDMI ✅ | ✅ | Stick form factor |
| Radxa Zero | $25 | Micro-HDMI ✅ | ✅ | Amlogic S905Y2 |
| T-Dongle S3 (LilyGO) | $12 | No (LCD only) | ✅ | ESP32-S3, tiny LCD |

---

## 2. Projected Touch Wall

### Architecture

```
SOURCE (laptop/Pi)              RELAY (:9100)              WALL
───────────────                 ─────────────              ────
DreamStack app                  WebSocket hub              UST Projector
800×1280 canvas                                            + touch overlay

pixels → XOR delta → RLE ────→ relay ─────────────────→   decode → project
                                ←── touch {x,y,phase} ←── touch sensor
```

### Ultra-Short-Throw Projectors

| Product | Price (new) | Price (used) | Notes |
|---------|-------------|-------------|-------|
| Xiaomi Laser Cinema 2 | ~$1,200 | ~$400 | Good value |
| BenQ V7050i | ~$2,500 | ~$800 | 4K HDR |
| JMGO U2 | ~$1,000 | ~$300 | Budget friendly |
| Epson LS500 | ~$2,000 | ~$600 | Bright |

Wall prep: screen paint (Silver Ticket / Rust-Oleum, ~$30).

---

## 3. Touch Input Technologies

Ranked from fastest to slowest latency:

### 3a. Piezoelectric Sensors (<1ms, ~$7)

Stick 3-4 piezo discs on the **back** of the wall. Finger taps create vibrations; time-difference-of-arrival (TDOA) triangulates X,Y.

| Part | Cost |
|------|------|
| 4× piezo disc (35mm) | ~$2 |
| ESP32-S3 (built-in ADC, 40kHz+ sampling) | ~$5 |

```
Wall (drywall, glass, wood, whiteboard)
┌───────────────────────────────────────┐
│  P1 ●                       ● P2     │
│               👆 TAP                 │
│  P3 ●                       ● P4     │
└───────────────────────────────────────┘
       └────── ESP32 (TDOA → x,y) ─────┘
```

**Pros:** Near-instant, invisible, dirt cheap, works through paint
**Cons:** Only detects **taps** (not drag/hover), needs hard surface

---

### 3b. Capacitive Wire/Paint Grid (1-3ms, ~$21 DIY)

Grid of conductors (copper tape or conductive paint) behind the wall. Measures capacitance change when a finger approaches.

| Part | Cost |
|------|------|
| Copper tape grid (30 channels) or graphite paint | ~$10 |
| MPR121 capacitive controller ×3 | ~$6 |
| ESP32 | ~$5 |

**Supports:** Touch ✅ Drag ✅ Multi-touch ✅ Hover (~1-2cm) ✅
**Resolution:** Depends on grid pitch — 3cm pitch ≈ 30×40 nodes over 100"

#### How Mutual Capacitance Works

Two layers of conductors (rows + columns) cross each other, separated by a thin insulator. Each intersection forms a capacitor. A finger near any intersection absorbs electric field, reducing measured capacitance.

The controller scans one row at a time (AC drive), reads all columns simultaneously. Full scan of 30×40 grid: ~0.5-1ms. Continuous scanning gives automatic drag/swipe detection. Sub-pixel interpolation from adjacent node readings gives ~1mm accuracy from 5mm pitch.

#### Recommended ICs

| IC | Grid Size | Touch Points | Price |
|----|-----------|-------------|-------|
| MTCH6303 (Microchip) | 15×49 | 10 | ~$5 |
| IQS7211A (Azoteq) | 15×22 | 5 | ~$3 |
| GT911 (Goodix) | 26×14 | 5 | ~$2 |
| FT5x06 (FocalTech) | 24×14 | 5 | ~$2 |

---

### 3c. FTIR — Frustrated Total Internal Reflection (3-8ms, ~$110-250)

Acrylic sheet on wall with IR LEDs on edges (total internal reflection). Finger touch "frustrates" the reflection → bright spots detected by IR camera.

| Part | Cost |
|------|------|
| 4mm acrylic sheet (100") | ~$80-150 |
| IR LED strip (850nm) on edges | ~$10 |
| IR camera (120fps, no IR filter) | ~$15 |
| ESP32-S3 or Pi | ~$5-75 |

**Pros:** Multi-touch, precise, pressure-sensitive (brighter blob = more pressure)
**Cons:** Needs smooth flat surface (acrylic)

---

### 3d. IR Touch Frame (8-15ms, ~$250-500)

Aluminum frame with IR LEDs + sensors on 4 edges. Finger breaks IR beams → X,Y.

| Size | Price | Touch Points |
|------|-------|-------------|
| 65" | ~$250 | 6-10 pt |
| 82" | ~$350 | 10-20 pt |
| 100" | ~$500 | 10-20 pt |
| 120"+ | ~$800+ | 20+ pt |

Premium: Neonode zForce (~6-8ms, 200Hz). Budget: generic Chinese frames (~15-30ms, 100Hz).

**Pros:** USB HID plug-and-play, works on any surface
**Cons:** Physical border/bezel on wall

---

### 3e. Depth Camera (15-30ms, ~$80-450)

| Camera | FPS | Latency | Range | Price |
|--------|-----|---------|-------|-------|
| **Intel RealSense D405** | 90fps | ~11ms | 7cm-50cm | ~$80 |
| RealSense D435i | 90fps | ~11ms | 10cm-10m | ~$200 |
| OAK-D SR (Short Range) | 60fps | ~12ms | 2cm-100cm | ~$150 |
| OAK-D Pro | 30fps depth | ~15ms | 20cm-15m | ~$200 |
| Stereolabs ZED Mini | 100fps | ~10ms | 10cm-12m | ~$300 |
| Stereolabs ZED X Mini | 120fps | ~8ms | 10cm-15m | ~$450 |
| Orbbec Gemini 2 | 60fps | ~16ms | 15cm-10m | ~$130 |

**RealSense D405** is ideal for wall touch — 90fps hardware stereo depth, 7cm minimum distance, global shutter. No ML needed for touch detection: just threshold the depth map (`depth < 5mm → TOUCH`, `< 150mm → HOVER`).

Layer MediaPipe on top (parallel) for gesture classification.

---

### 3f. Hybrid: Best of All Worlds

| Input | Method | Latency |
|-------|--------|---------|
| Tap detection + pressure | Piezo (4 corners) | <1ms |
| Touch + drag + hover | Capacitive grid | 1-3ms |
| Hand gestures (air) | RealSense D405 | ~15ms |

---

## 4. Gesture / Hand Tracking

### DIY Approaches (Ultraleap alternative)

#### Stereo IR Camera + MediaPipe (~$30-50)

Two OV2710 IR USB cameras (stereo pair, ~$15 each) + 850nm IR LED strip (~$5). MediaPipe Hands on Pi 5 or Jetson: 21 landmarks per hand, 30-120fps. Stereo triangulation gives Z. **Latency: ~20-30ms.**

#### Single Depth Camera (~$80-150)

Use RealSense D405 or OAK-D SR (see above). Hardware depth gives Z-distance from wall.

#### ESP32-S3 + IR Matrix (~$20, lowest latency)

IR LEDs flood the area in front of the wall. 2-3 IR cameras do blob detection at 120fps on ESP32-S3. Z estimated from blob size. No ML needed. **Latency: 5-10ms.**

---

## 5. Conductive Paint Recipes

For capacitive grid electrodes painted directly on walls.

### Graphite Paint (easiest, ~$5)

| Ingredient | Amount | Source |
|-----------|--------|--------|
| Graphite powder (<45μm) | 3 tbsp | Art supply, Amazon (~$8/lb) |
| PVA glue (white school glue) | 2 tbsp | Any store |
| Water | 1 tbsp | Tap |

~60% graphite, 30% glue, 10% water by volume. **Resistance: ~500-2000 Ω/sq.** Good enough for capacitive sensing.

### Carbon Black + Acrylic (~$15)

20-25% carbon black powder (conductive grade) in 75-80% acrylic medium. **Resistance: ~200-800 Ω/sq.** Better adhesion. Wear mask + gloves.

### Nickel Paint (~$20)

MG Chemicals 841, premade. **Resistance: ~5-50 Ω/sq.** Mid-range.

### Silver Paint (~$30-50)

Premade: Bare Conductive (~$25/50ml), MG Chemicals 842.
DIY: 70-80% silver flake powder (<10μm), 15-20% acrylic medium, 5-10% butyl acetate.
**Resistance: ~0.5-5 Ω/sq.** Near-wire conductivity.

### For capacitive sensing: graphite is sufficient

Capacitive touch doesn't need low resistance — just enough conductivity to couple with a finger. Paint lines with tape masking at 3-5cm spacing.

---

## 6. Pixel Paint — Paint-On Displays

### Electroluminescent (EL) Paint Display

Real and buildable. A stack of painted layers that glow when AC voltage is applied.

```
Layer stack (painted in order):

  5. Clear topcoat
  4. Transparent conductor (PEDOT:PSS)         ← rows
  3. Phosphor layer (ZnS:Cu in acrylic)        ← glows
  2. Dielectric (BaTiO₃ in acrylic)            ← insulator
  1. Base conductor (silver/carbon paint)       ← columns
  ─── Wall surface ───
```

Row/column intersection = one pixel. AC across a specific row+column → only that intersection glows (passive matrix).

| Layer | Material | Cost/m² |
|-------|----------|---------|
| Base conductor (columns) | Silver paint, painted in strips | ~$50 |
| Dielectric | Barium titanate (BaTiO₃) in acrylic | ~$30 |
| Phosphor | ZnS:Cu powder in acrylic | ~$20 |
| Top conductor (rows) | PEDOT:PSS | ~$40 |
| Driver electronics | HV507 shift registers + ESP32 | ~$30 |
| **Total** | | **~$170/m²** |

#### Resolution at different pitches

| Pixel Pitch | Pixels (100" wall) | Comparable To |
|------------|---------------------|---------------|
| 20mm | 110×65 = 7,150 | LED sign |
| 10mm | 220×130 = 28,600 | Scoreboard |
| 5mm | 440×260 = 114,400 | ~400×260 display ✅ |
| 2mm | 1100×650 = 715,000 | Near SD |

At 5mm pitch: 440×260 — enough for DreamStack UIs, dashboards, snake game.

#### Color

- ZnS:Cu → green (brightest)
- ZnS:Cu,Mn → amber/orange
- ZnS:Cu,Al → blue-green
- Full RGB requires 3 sub-pixels per pixel (3× driver count)
- Monochrome green is practical and looks great

#### Built-in touch (free!)

The row/column electrodes double as capacitive sensing electrodes via time-multiplexing:
1. **Sense phase** (1ms): measure capacitance = touch position
2. **Drive phase** (15ms): apply AC = illuminate pixels

Same paint layers, no extra hardware.

#### Driver IC

HV507 — 64-channel high-voltage shift register. Drives 100V+ outputs from 3.3V SPI. Chain several for full display.

### Other Display Paint Technologies (Future)

| Technology | Status | Color | Speed |
|-----------|--------|-------|-------|
| Electrochromic (PEDOT:PSS, WO₃) | Real | Grayscale | 1-30s (too slow for video) |
| Thermochromic + resistive grid | Hackable | Limited | 1-5s |
| Perovskite spray-on LEDs | Lab only | Full color | ~ms |
| QD-LED inkjet | Lab only | Full color | ~ms |

Perovskite / QD-LED spray-on is the future (~2028-2030) but not available today.

---

## 7. Off-the-Shelf Solutions

### Capacitive Touch Overlays (stick-on film)

| Product | Max Size | Touch Points | Latency | Price |
|---------|----------|-------------|---------|-------|
| **Displax Skin Ultra** | 105" | 40 | ~6ms | ~$800-1500 |
| Visual Planet TouchFoil | 100"+ | 40 | ~8ms | ~$600-1200 |
| PQ Labs iTouch Plus | 150"+ | 32 | ~8ms | ~$400-900 |
| AliExpress "PCAP touch foil" | 100"+ | 10 | ~10-15ms | ~$200-400 |

Displax Skin Ultra: transparent polymer film with nano-wire grid, adhesive-backed, works through 6mm of material, USB HID, detects hover at ~2cm. Stick on wall, plug USB, done.

### All-in-One Interactive Projectors

| Product | Size | Touch | Latency | Price (new) |
|---------|------|-------|---------|-------------|
| **Epson BrightLink 770Fi** | 100" | 10pt + pen | ~10ms | ~$2,500 |
| Epson BrightLink 735Fi | 100" | 10pt + pen | ~10ms | ~$2,000 |
| BenQ LW890UST | 100" | 10pt | ~12ms | ~$1,800 |
| Boxlight Mimio MiXX | 100" | 20pt | ~8ms | ~$2,200 |

**Used education projectors** (schools constantly upgrade):

| Used Option | Price |
|-------------|-------|
| Epson BrightLink 695Wi/696Ui | $300-600 |
| BenQ MW855UST+ with PointWrite | $400-700 |
| Promethean UST + ActivBoard | $300-500 |

### Interactive Flat Panels (giant touchscreen monitors)

| Product | Size | Price (new) | Price (used) |
|---------|------|-------------|-------------|
| **SMART Board MX** | 65-86" | $3,000-6,000 | $500-1,500 |
| Promethean ActivPanel | 65-86" | $3,000-5,000 | $600-1,200 |
| ViewSonic ViewBoard | 65-98" | $2,000-8,000 | $500-1,500 |
| Samsung Flip | 55-85" | $2,000-4,000 | $800-2,000 |
| Microsoft Surface Hub 2S | 50-85" | $5,000-12,000 | $1,500-3,000 |

---

## 8. Recommended Builds

### Budget: $675

| Component | Source | Price |
|-----------|--------|-------|
| Used SMART Board 65" | eBay | ~$600 |
| Pi 5 | Official | ~$75 |

Plug HDMI + USB, run DreamStack, done.

### Mid-Range: $700

| Component | Price |
|-----------|-------|
| UST projector (used) | ~$300 |
| PCAP touch foil 100" (AliExpress) | ~$300 |
| Pi 5 | ~$75 |
| Screen paint | ~$30 |

### Premium: $1,050

| Component | Price |
|-----------|-------|
| UST projector (used) | ~$400 |
| 100" IR touch frame | ~$350 |
| RealSense D405 (gestures + hover) | ~$80 |
| Pi 5 | ~$75 |
| Piezo sensors (4 corners, tap confirm) | ~$7 |
| Screen paint | ~$30 |

Touch at 8-15ms + hover/gestures at 15ms + tap confirmation at <1ms.

### DIY Maximum: ~$200 + wall paint

| Component | Price |
|-----------|-------|
| Conductive graphite paint (capacitive grid) | ~$10 |
| MPR121/MTCH6303 cap-touch IC | ~$5 |
| ESP32-S3 | ~$5 |
| UST projector (used) | ~$300 |

Paint your own touch grid on the wall, 1-3ms latency, no frame needed.

---

## 9. DreamStack Integration

All touch methods feed into the existing relay protocol:

```
Touch sensor (any method above)
  → ESP32 or Pi reads touch events
  → Encodes as DreamStack protocol:
       0x01 Pointer move    (x, y)
       0x02 Pointer down    (x, y, buttons)
       0x03 Pointer up
       0x10 KeyDown         (keyCode)
       0x20 Hover           (x, y, z_distance)    ← new
       0x21 Swipe           (direction, velocity)  ← new
       0x22 Pinch/Grab      (state)                ← new
  → WebSocket → DreamStack relay
  → App receives as signal updates
```

DreamStack syntax for handling:

```
on hover(ev) -> opacity = lerp(0.5, 1.0, ev.z)
on swipe(ev) -> navigate(if ev.dir == "left" then "/next" else "/prev")
on grab(ev) -> scale = if ev.closed then 0.9 else 1.0
```