···1+# Widget Support Plan of Record
2+3+*Created: 2026-02-12*
4+5+## Goals
6+7+Add Jupyter-style interactive widget support to js_top_worker, enabling OCaml
8+code running in the toplevel to create interactive UI elements (sliders, buttons,
9+dropdowns, etc.) that communicate bidirectionally with the frontend.
10+11+## Design Principles
12+13+1. **Zero new dependencies in the worker** - Every dependency compiled into the
14+ worker can conflict with libraries the user wants to load at runtime. Widget
15+ support must not add any new OCaml dependencies to the worker.
16+17+2. **Broad OCaml version compatibility** - The project currently targets OCaml
18+ >= 4.04. New features must not raise this floor unnecessarily.
19+20+3. **Build on the message protocol, not RPC** - The worker already uses the
21+ message-based protocol (`message.ml` / `js_top_worker_client_msg.ml`). Widget
22+ communication extends this protocol rather than the legacy JSON-RPC layer.
23+24+4. **Remove, don't accumulate** - The legacy rpclib-based communication layer
25+ should be removed as part of this work, reducing the dependency footprint.
26+27+## Why Not CBOR?
28+29+The architecture document previously listed CBOR as a planned transport format.
30+After investigation, we've decided against it:
31+32+- **Dependency risk**: Even the lightweight `cbor` opam package brings in
33+ `ocplib-endian`. Any dependency in the worker namespace can conflict with
34+ user-loaded libraries.
35+- **Unnecessary complexity**: The existing message protocol uses `Js_of_ocaml`'s
36+ native JSON handling (`Json.output` / `Json.unsafe_input`), which has zero
37+ additional dependencies.
38+- **Binary data via Typed Arrays**: For binary payloads (images, etc.),
39+ `js_of_ocaml`'s `Typed_array` module provides native browser typed array
40+ support without any extra libraries.
41+- **JSON is the browser's native format** - No encoding/decoding overhead when
42+ passing structured data via `postMessage`.
43+44+## Communication Architecture
45+46+### Current State (Two Parallel Layers)
47+48+```
49+1. Legacy RPC (to be removed):
50+ Client (js_top_worker_client.ml) <-> JSON-RPC <-> Server (Toplevel_api_gen)
51+ Dependencies: rpclib, rpclib-lwt, rpclib.json, ppx_deriving_rpc
52+53+2. Message protocol (to be extended):
54+ Client (js_top_worker_client_msg.ml) <-> JSON messages <-> Worker (worker.ml)
55+ Dependencies: js_of_ocaml (already required)
56+```
57+58+### Target State
59+60+```
61+Client (js_top_worker_client_msg.ml) <-> JSON messages <-> Worker (worker.ml)
62+ | |
63+ |-- Request/Response (existing: eval, complete, errors, ...) |
64+ |-- Push messages (existing: output_at streaming) |
65+ |-- Widget messages (NEW: comm_open, comm_update, comm_msg, ...) |
66+```
67+68+All communication uses the existing `message.ml` protocol extended with widget
69+message types. No new serialization libraries.
70+71+## Widget Protocol Design
72+73+### Message Types (Worker -> Client)
74+75+```
76+CommOpen { comm_id; target; state } -- Widget created by OCaml code
77+CommUpdate { comm_id; state } -- Widget state changed
78+CommClose { comm_id } -- Widget destroyed
79+```
80+81+### Message Types (Client -> Worker)
82+83+```
84+CommMsg { comm_id; data } -- Frontend event (click, value change)
85+CommClose { comm_id } -- Frontend closed widget
86+```
87+88+### Widget State Format
89+90+Widget state is a JSON object with well-known keys, following the Jupyter widget
91+convention where practical:
92+93+```json
94+{
95+ "widget_type": "slider",
96+ "value": 50,
97+ "min": 0,
98+ "max": 100,
99+ "step": 1,
100+ "description": "Threshold",
101+ "disabled": false
102+}
103+```
104+105+The `widget_type` field replaces Jupyter's `_model_module` / `_model_name` /
106+`_view_module` / `_view_name` quartet, since we don't need the npm module
107+indirection - our widget renderers are built into the client.
108+109+### Alignment with Jupyter Protocol
110+111+We adopt the **concepts** from the Jupyter widget protocol but simplify the
112+implementation:
113+114+| Jupyter Concept | Our Equivalent |
115+|-----------------|----------------|
116+| comm_open | CommOpen message |
117+| comm_msg method:"update" | CommUpdate message |
118+| comm_msg method:"custom" | CommMsg message |
119+| comm_close | CommClose message |
120+| _model_module + _model_name | widget_type string |
121+| buffer_paths (binary) | Typed_array via js_of_ocaml |
122+| Display message | CommOpen includes display flag |
123+124+We do **not** implement:
125+- echo_update (single frontend, no multi-client sync needed)
126+- request_state / request_states (state is authoritative in worker)
127+- Version negotiation (internal protocol, not cross-system)
128+129+## OCaml Widget API
130+131+User-facing API available as an OCaml library in the toplevel:
132+133+```ocaml
134+module Widget : sig
135+ type t
136+137+ (** Create a widget. Returns it and displays it. *)
138+ val slider : ?min:int -> ?max:int -> ?step:int ->
139+ ?description:string -> int -> t
140+141+ val button : ?style:string -> string -> t
142+143+ val text : ?placeholder:string -> ?description:string -> string -> t
144+145+ val dropdown : ?description:string -> options:string list -> string -> t
146+147+ val checkbox : ?description:string -> bool -> t
148+149+ val html : string -> t
150+151+ (** Read current value *)
152+ val get : t -> Yojson.Safe.t (* or a simpler JSON type *)
153+154+ (** Update widget state *)
155+ val set : t -> string -> Yojson.Safe.t -> unit
156+157+ (** Register event handler *)
158+ val on_change : t -> (Yojson.Safe.t -> unit) -> unit
159+ val on_click : t -> (unit -> unit) -> unit
160+161+ (** Display / close *)
162+ val display : t -> unit
163+ val close : t -> unit
164+end
165+```
166+167+**Important**: This API library (`widget` or similar) runs *inside* the toplevel
168+and must have minimal dependencies. It communicates with the frontend by pushing
169+messages through the same channel as `Mime_printer`.
170+171+## Code Removal Plan
172+173+### Files to Remove
174+175+| File | Reason |
176+|------|--------|
177+| `idl/transport.ml`, `transport.mli` | JSON-RPC transport wrapper |
178+| `idl/js_top_worker_client.ml`, `.mli` | RPC-based Lwt client |
179+| `idl/js_top_worker_client_fut.ml` | RPC-based Fut client |
180+| `idl/_old/` directory | Historical RPC reference code |
181+182+### Dependencies to Remove
183+184+| Package | Used By |
185+|---------|---------|
186+| `rpclib` | transport.ml, RPC clients |
187+| `rpclib-lwt` | impl.ml (IdlM module) |
188+| `rpclib.json` | transport.ml, RPC clients |
189+| `ppx_deriving_rpc` | toplevel_api.ml code generation |
190+| `xmlm` | transitive via rpclib |
191+| `cmdliner` | transitive via rpclib |
192+193+### Files to Refactor
194+195+| File | Change |
196+|------|--------|
197+| `lib/impl.ml` | Remove `IdlM` / `Rpc_lwt` usage, use own types |
198+| `idl/toplevel_api.ml` | Keep type definitions, remove RPC IDL machinery |
199+| `idl/toplevel_api_gen.ml` | Replace with hand-written types (no ppx_deriving_rpc) |
200+| `idl/dune` | Remove rpclib library deps and ppx rules |
201+| `lib/dune` | Remove rpclib-lwt dep |
202+| `test/node/*.ml` | Migrate from RPC Server/Client to message protocol |
203+| `test/browser/client_test.ml` | Use message-based client |
204+| `example/unix_worker.ml` | Use message protocol over Unix socket |
205+| `example/unix_client.ml` | Use message protocol over Unix socket |
206+207+### Impact on `toplevel_api_gen.ml`
208+209+The generated file is 92k+ lines (from ppx_deriving_rpc). The types it defines
210+are used extensively in `impl.ml` and `worker.ml`. The plan:
211+212+1. Extract the **type definitions** into a new lightweight module (no ppx)
213+2. Hand-write any needed serialization for the message protocol
214+3. Remove `ppx_deriving_rpc` dependency entirely
215+4. Delete `toplevel_api_gen.ml`
216+217+## Testing Strategy
218+219+### Existing Test Infrastructure
220+221+| Backend | Location | Framework |
222+|---------|----------|-----------|
223+| Unit tests | `test/libtest/` | ppx_expect |
224+| Node.js | `test/node/` | js_of_ocaml + Node |
225+| Unix (cram) | `test/cram/` | Cram tests with unix_worker |
226+| Browser | `test/browser/` | Playwright + Chromium |
227+228+### Widget Testing Approach
229+230+#### 1. Unit Tests (`test/libtest/`)
231+232+- Widget state management logic
233+- Message serialization/deserialization for widget messages
234+- CommManager state tracking (open/update/close lifecycle)
235+236+#### 2. Node.js Tests (`test/node/`)
237+238+- Widget creation produces correct CommOpen messages
239+- Widget state updates produce CommUpdate messages
240+- Event handler registration and dispatch
241+- Widget close cleanup
242+- Multiple simultaneous widgets
243+- Widget interaction with regular exec output (mime_vals + widgets)
244+245+#### 3. Cram Tests (`test/cram/`)
246+247+- Unix worker handles widget messages over socket
248+- Widget lifecycle via command-line client
249+- Widget messages interleaved with regular eval output
250+251+#### 4. Browser Tests (`test/browser/`)
252+253+- **End-to-end widget rendering**: OCaml creates widget -> message sent ->
254+ client renders DOM element -> user interaction -> event sent back -> OCaml
255+ handler fires
256+- **Widget types**: Test each widget type (slider, button, text, dropdown,
257+ checkbox, html)
258+- **State synchronization**: Frontend changes propagated to worker and back
259+- **Multiple widgets**: Several widgets active simultaneously
260+- **Widget cleanup**: Closing widgets removes DOM elements
261+- **Integration with existing features**: Widgets alongside code completion,
262+ error reporting, MIME output
263+264+### Test Utilities
265+266+A shared test helper module for widget testing:
267+268+```ocaml
269+(* test/test_widget_helpers.ml *)
270+val assert_comm_open : worker_msg -> comm_id:string -> widget_type:string -> unit
271+val assert_comm_update : worker_msg -> comm_id:string -> key:string -> unit
272+val assert_comm_close : worker_msg -> comm_id:string -> unit
273+val simulate_event : comm_id:string -> data:string -> client_msg
274+```
275+276+## Example Widgets
277+278+### Priority 1: Core Widgets (Implement First)
279+280+These are the most commonly used Jupyter widgets and cover the fundamental
281+interaction patterns:
282+283+| Widget | State | Events | Jupyter Equivalent |
284+|--------|-------|--------|--------------------|
285+| IntSlider | value, min, max, step | on_change(int) | IntSlider |
286+| Button | description, style | on_click | Button |
287+| Text | value, placeholder | on_change(string) | Text |
288+| Dropdown | value, options | on_change(string) | Dropdown |
289+| Checkbox | value, description | on_change(bool) | Checkbox |
290+| HTML | value (html string) | none | HTML |
291+292+### Priority 2: Composition Widgets
293+294+| Widget | Purpose | Jupyter Equivalent |
295+|--------|---------|-------------------|
296+| HBox / VBox | Layout containers | HBox / VBox |
297+| Output | Capture stdout/display | Output |
298+| FloatSlider | Decimal slider | FloatSlider |
299+300+### Priority 3: Domain-Specific Widgets
301+302+| Widget | Purpose | Inspired By |
303+|--------|---------|-------------|
304+| Plot | Simple 2D charts (SVG) | bqplot (simplified) |
305+| Table | Data grid display | ipydatagrid (read-only) |
306+| Image | Display image bytes | Image widget |
307+308+### Example: Interactive Slider
309+310+```ocaml
311+(* User code in toplevel *)
312+let threshold = Widget.slider ~min:0 ~max:100 ~description:"Threshold" 50;;
313+314+Widget.on_change threshold (fun v ->
315+ let n = Widget.Int.of_json v in
316+ Printf.printf "Threshold changed to: %d\n" n
317+);;
318+```
319+320+### Example: Button with Output
321+322+```ocaml
323+let count = ref 0;;
324+let label = Widget.html (Printf.sprintf "<b>Count: %d</b>" !count);;
325+let btn = Widget.button "Increment";;
326+327+Widget.on_click btn (fun () ->
328+ incr count;
329+ Widget.set label "value"
330+ (`String (Printf.sprintf "<b>Count: %d</b>" !count))
331+);;
332+```
333+334+### Example: Linked Widgets
335+336+```ocaml
337+let slider = Widget.slider ~min:0 ~max:255 ~description:"Red" 128;;
338+let preview = Widget.html {|<div style="width:50px;height:50px"></div>|};;
339+340+Widget.on_change slider (fun v ->
341+ let r = Widget.Int.of_json v in
342+ Widget.set preview "value"
343+ (`String (Printf.sprintf
344+ {|<div style="width:50px;height:50px;background:rgb(%d,0,0)"></div>|} r))
345+);;
346+```
347+348+## Implementation Phases
349+350+### Phase 1: RPC Removal & Type Cleanup
351+352+Remove the legacy RPC layer and establish clean type definitions.
353+354+### Phase 2: Widget Message Protocol
355+356+Extend `message.ml` and `worker.ml` with widget message types.
357+358+### Phase 3: Widget Manager (Worker Side)
359+360+Implement the comm manager that tracks widget state and routes events.
361+362+### Phase 4: OCaml Widget API
363+364+Create the user-facing `Widget` module available in the toplevel.
365+366+### Phase 5: JavaScript Widget Renderer
367+368+Implement widget rendering in the JavaScript client.
369+370+### Phase 6: Testing & Examples
371+372+Full test coverage across all backends, example widgets, documentation.
373+374+## Open Questions
375+376+1. **JSON representation in OCaml API**: Use `Yojson.Safe.t`? A custom minimal
377+ JSON type? Raw `Js.Unsafe.any`? (Yojson adds a dependency; custom type is
378+ more work; Unsafe.any is untyped.)
379+380+2. **Widget library loading**: Should the Widget module be preloaded in the
381+ worker, or loaded on demand via `#require "widget"`?
382+383+3. **Layout model**: How much of Jupyter's CSS-based layout model to support?
384+ Full flexbox control per-widget, or simpler HBox/VBox only?
385+386+4. **Persistence**: Should widget state survive cell re-execution? Jupyter
387+ widgets are destroyed and recreated; we could do the same or preserve state.