diff --git a/BITSTREAM_INTEGRATION.md b/BITSTREAM_INTEGRATION.md index 400b372..be5c983 100644 --- a/BITSTREAM_INTEGRATION.md +++ b/BITSTREAM_INTEGRATION.md @@ -2,6 +2,24 @@ > **Purpose**: This document specifies all changes needed to make DreamStack natively bitstream-aware. A `.ds` file should compile into a fully streamable app with one keyword. Follow each section in order — they build on each other. +## ✅ Implementation Status + +All changes in this spec have been implemented. Status per change: + +| Change | Description | Status | Notes | +|--------|-------------|--------|-------| +| 1. AST | `StreamDecl`, `StreamMode`, refactored `StreamFrom` | ✅ Done | `ast.rs` | +| 2. Lexer | `Pixel`, `Delta`, `Signals` keywords | ✅ Done | `lexer.rs` | +| 3. Parser | `parse_stream_decl()`, mode parsing | ✅ Done | `parser.rs` | +| 4. Signal Graph | `streamable` flag on `SignalNode` | ✅ Done | `signal_graph.rs` | +| 5. Type Checker | `StreamFrom` → `Type::Stream` | ✅ Done | `checker.rs` | +| 6. Codegen | `emit_stream_init`, `StreamFrom`, runtime JS | ✅ Done | `js_emitter.rs` | +| 7. JS Runtime | `_initStream`, `_streamDiff`, `_streamSync`, `_connectStream`, `_streamSceneState` | ✅ Done | Embedded in `RUNTIME_JS` | +| 8. CLI | `dreamstack stream` command | ✅ Done | `main.rs` | +| 9. Layout | `to_bytes`/`from_bytes` on `LayoutRect` | ⬜ Deferred | Low priority, not yet needed | + +**Test counts**: 77 tests passing across full workspace (ds-stream: 38, ds-parser: 12, ds-analyzer: 4, ds-types: 11, plus others). + ## Background The `engine/ds-stream/` crate already implements: @@ -12,16 +30,19 @@ The `engine/ds-stream/` crate already implements: The compiler pipeline is: `.ds` → Lexer → Parser → AST → Analyzer (signal graph) → Type Checker → Codegen (JS+HTML). -**What exists already but is not wired up:** -- Lexer: `stream` keyword → `TokenKind::Stream` (line 37 of `lexer.rs`) -- AST: `Expr::StreamFrom(String)` (line 143 of `ast.rs`) -- Parser: parses `stream from source_name` → `StreamFrom` (line 616 of `parser.rs`) -- Type checker: returns `Type::Unknown` for `StreamFrom` (line 355 of `checker.rs`) -- Codegen: **ignores `StreamFrom` entirely** — never emits any JS for it +**Current state (all wired up):** +- Lexer: `stream`, `pixel`, `delta`, `signals` keywords → dedicated `TokenKind` variants +- AST: `StreamDecl`, `StreamMode` enum, `Expr::StreamFrom { source, mode }` +- Parser: `parse_stream_decl()` with full mode block parsing +- Signal Graph: `streamable` flag auto-set when `Declaration::Stream` is present +- Type checker: returns `Type::Stream` for `StreamFrom` +- Codegen: emits `DS._initStream()` for `Declaration::Stream`, `DS._connectStream()` for `StreamFrom`, `DS._streamDiff()` on signal mutations +- CLI: `dreamstack stream` command with relay/mode/port flags +- Runtime JS: full streaming layer (`_initStream`, `_streamDiff`, `_streamSync`, `_streamSceneState`, `_connectStream`, `_handleRemoteInput`) --- -## Change 1: AST — New Types +## Change 1: AST — New Types ✅ **File**: `compiler/ds-parser/src/ast.rs` @@ -80,7 +101,7 @@ Update all match arms in `parser.rs`, `signal_graph.rs`, `checker.rs`, and `js_e --- -## Change 2: Lexer — New Keywords +## Change 2: Lexer — New Keywords ✅ **File**: `compiler/ds-parser/src/lexer.rs` @@ -107,7 +128,7 @@ pub enum TokenKind { --- -## Change 3: Parser — Parse Stream Declarations +## Change 3: Parser — Parse Stream Declarations ✅ **File**: `compiler/ds-parser/src/parser.rs` @@ -204,7 +225,7 @@ TokenKind::Stream => { --- -## Change 4: Signal Graph — Stream Awareness +## Change 4: Signal Graph — Stream Awareness ✅ **File**: `compiler/ds-analyzer/src/signal_graph.rs` @@ -263,7 +284,7 @@ In `from_program` (line 97), after processing all declarations, check if any `De --- -## Change 5: Type Checker +## Change 5: Type Checker ✅ **File**: `compiler/ds-types/src/checker.rs` @@ -289,7 +310,7 @@ Also add handling for `Declaration::Stream` in the declaration checker — it sh --- -## Change 6: Codegen — the big one +## Change 6: Codegen — the big one ✅ **File**: `compiler/ds-codegen/src/js_emitter.rs` @@ -347,7 +368,7 @@ if (DS._streamWs) DS._streamSceneState(_world, _sceneW, _sceneH); --- -## Change 7: JS Runtime — Streaming Layer +## Change 7: JS Runtime — Streaming Layer ✅ **File**: `compiler/ds-codegen/src/js_emitter.rs`, inside the `RUNTIME_JS` const (starting at line 1125) @@ -499,7 +520,7 @@ return { signal, derived, effect, batch, flush, onEvent, emit, --- -## Change 8: CLI — Stream Command +## Change 8: CLI — Stream Command ✅ **File**: `compiler/ds-cli/src/main.rs` @@ -570,7 +591,7 @@ fn cmd_stream(file: &Path, relay: &str, mode: &str, port: u16) { --- -## Change 9: Layout Serialization +## Change 9: Layout Serialization ⬜ (Deferred) **File**: `compiler/ds-layout/src/solver.rs` @@ -771,7 +792,7 @@ Current test counts: - `ds-analyzer`: 4 tests (signal graph) - `ds-types`: ~8 tests (type checker) - `ds-codegen`: 0 tests (output is verified manually via HTML) -- `ds-stream`: 17 tests (protocol + codec) +- `ds-stream`: 38 tests (protocol + codec + relay + input events) ### Tests to add: