···11+# Scrollycode Monorepo Integration Plan
22+33+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
44+55+**Goal:** Consolidate odoc (with extension-plugins), js_top_worker, x-ocaml, and a standalone scrollycode extension into the monorepo, then add mobile responsiveness and x-ocaml/js_top_worker integration for interactive code tutorials.
66+77+**Architecture:** Four workstreams executed in sequence: (1) monorepo consolidation via monopam, (2) mobile responsive CSS with diff view, (3) x-ocaml + js_top_worker integration for type-on-hover and live execution, (4) "Open in Playground" overlay. The monorepo work unblocks everything else by getting all packages building together.
88+99+**Tech Stack:** OCaml, dune, monopam (git subtree monorepo manager), odoc extension API, x-ocaml web component, js_top_worker (browser OCaml toplevel), CodeMirror 6, CSS media queries, IntersectionObserver.
1010+1111+---
1212+1313+## Workstream 1: Monorepo Consolidation
1414+1515+### Task 1.1: Merge extension-plugins into odoc staging
1616+1717+The monorepo tracks odoc on `jonludlam/staging`. The `extension-plugins` branch has 6 commits staging doesn't have (custom tags + dune-site plugin system). Staging has 41 commits extension-plugins doesn't have. They diverge from `d8460cda`.
1818+1919+**Files:**
2020+- Modify: `~/workspace/odoc/` (git merge)
2121+2222+**Step 1: Create a working branch from staging**
2323+2424+```bash
2525+cd ~/workspace/odoc
2626+git fetch jonludlam staging extension-plugins
2727+git checkout -b staging-with-extensions jonludlam/staging
2828+```
2929+3030+**Step 2: Merge extension-plugins into the new branch**
3131+3232+```bash
3333+git merge jonludlam/extension-plugins
3434+```
3535+3636+Expected: Merge conflicts in ~5-10 files. Key conflict areas:
3737+- `src/parser/ast.ml` — both add Custom tag support
3838+- `src/odoc/bin/main.ml` — both add extension CLI code
3939+- `src/model/comment.ml` — tag handling logic
4040+- `src/document/comment.ml` — custom tag rendering
4141+- Build files (`dune-project`, `odoc.opam`, various `dune` files)
4242+4343+**Step 3: Resolve conflicts**
4444+4545+For each conflict:
4646+- Staging's extension infrastructure is more mature (41 commits of iteration)
4747+- Extension-plugins adds the dune-site dynamic loading that staging may not have
4848+- Prefer staging's versions where they cover the same functionality
4949+- Keep extension-plugins' dune-site plugin loading (`src/odoc/bin/main.ml` — `Sites.Plugins.Extensions.load_all ()`)
5050+5151+**Step 4: Build and test**
5252+5353+```bash
5454+dune build @all 2>&1 | head -50
5555+dune runtest 2>&1 | tail -20
5656+```
5757+5858+Expected: Clean build. If tests fail, fix incrementally.
5959+6060+**Step 5: Cherry-pick the scrollycode-demos commit**
6161+6262+```bash
6363+git cherry-pick 8f6c4537f
6464+```
6565+6666+This adds `src/extensions/scrollycode/` and `test/scrollycode-demos/` — the scrollycode extension (which will be extracted in Task 1.2).
6767+6868+**Step 6: Push the merged branch**
6969+7070+```bash
7171+git push jonludlam staging-with-extensions
7272+```
7373+7474+Then update `~/workspace/mono/sources.toml` to track this branch:
7575+7676+```toml
7777+[odoc]
7878+url = "git+https://github.com/jonludlam/odoc.git"
7979+upstream = "git+https://github.com/ocaml/odoc.git"
8080+branch = "staging-with-extensions"
8181+```
8282+8383+**Step 7: Sync odoc in monorepo**
8484+8585+```bash
8686+cd ~/workspace/mono
8787+monopam sync odoc
8888+```
8989+9090+**Step 8: Commit**
9191+9292+```bash
9393+git add -A && git commit -m "chore: sync odoc with extension-plugins merge"
9494+```
9595+9696+---
9797+9898+### Task 1.2: Extract scrollycode extension into standalone repo
9999+100100+Create `odoc-scrollycode-extension` as a standalone package in the monorepo, following the pattern of `odoc-rfc-extension` and `odoc-admonition-extension`.
101101+102102+**Files:**
103103+- Create: `mono/odoc-scrollycode-extension/dune-project`
104104+- Create: `mono/odoc-scrollycode-extension/src/dune`
105105+- Create: `mono/odoc-scrollycode-extension/src/scrollycode_extension.ml` (move from odoc)
106106+- Create: `mono/odoc-scrollycode-extension/test/` (move demo files from odoc)
107107+108108+**Step 1: Create directory structure**
109109+110110+```bash
111111+cd ~/workspace/mono
112112+mkdir -p odoc-scrollycode-extension/src odoc-scrollycode-extension/test
113113+```
114114+115115+**Step 2: Create dune-project**
116116+117117+Write `odoc-scrollycode-extension/dune-project`:
118118+119119+```
120120+(lang dune 3.18)
121121+(using dune_site 0.1)
122122+(name odoc-scrollycode-extension)
123123+(generate_opam_files true)
124124+125125+(package
126126+ (name odoc-scrollycode-extension)
127127+ (synopsis "Scrollycode tutorial extension for odoc")
128128+ (description
129129+ "Provides @scrolly.warm, @scrolly.dark, and @scrolly.notebook tags
130130+ for creating scroll-driven code tutorials in odoc documentation")
131131+ (depends
132132+ (ocaml (>= 4.14))
133133+ odoc
134134+ odoc_model
135135+ odoc_document))
136136+```
137137+138138+**Step 3: Create src/dune**
139139+140140+```
141141+(library
142142+ (public_name odoc-scrollycode-extension.impl)
143143+ (name scrollycode_extension)
144144+ (libraries odoc.extension_api odoc_model odoc_document))
145145+146146+(plugin
147147+ (name odoc-scrollycode-extension)
148148+ (libraries odoc-scrollycode-extension.impl)
149149+ (site (odoc extensions)))
150150+```
151151+152152+**Step 4: Move the extension source**
153153+154154+```bash
155155+cp odoc/src/extensions/scrollycode/scrollycode_extension.ml \
156156+ odoc-scrollycode-extension/src/scrollycode_extension.ml
157157+```
158158+159159+**Step 5: Move demo files to test/**
160160+161161+```bash
162162+cp odoc/test/scrollycode-demos/*.mld odoc-scrollycode-extension/test/
163163+cp odoc/test/scrollycode-demos/odoc_scrolly.ml odoc-scrollycode-extension/test/
164164+cp odoc/test/scrollycode-demos/odoc_scrolly_main.ml odoc-scrollycode-extension/test/
165165+cp odoc/test/scrollycode-demos/dune odoc-scrollycode-extension/test/
166166+```
167167+168168+Update `test/dune` to use the public library name:
169169+170170+```
171171+(executable
172172+ (name odoc_scrolly)
173173+ (libraries
174174+ cmdliner
175175+ odoc_model
176176+ odoc_odoc
177177+ odoc_extension_api
178178+ odoc-scrollycode-extension.impl))
179179+```
180180+181181+**Step 6: Remove scrollycode from odoc tree**
182182+183183+```bash
184184+rm -rf odoc/src/extensions/scrollycode/
185185+rm -rf odoc/test/scrollycode-demos/
186186+```
187187+188188+**Step 7: Build to verify**
189189+190190+```bash
191191+dune build odoc-scrollycode-extension 2>&1
192192+```
193193+194194+Expected: Clean build.
195195+196196+**Step 8: Commit**
197197+198198+```bash
199199+git add -A && git commit -m "feat: extract odoc-scrollycode-extension into standalone package"
200200+```
201201+202202+**Step 9: Fork into its own repo**
203203+204204+```bash
205205+monopam fork odoc-scrollycode-extension
206206+```
207207+208208+This creates `src/odoc-scrollycode-extension/` with extracted git history and re-adds `mono/odoc-scrollycode-extension/` as a proper subtree.
209209+210210+---
211211+212212+### Task 1.3: Join x-ocaml into monorepo
213213+214214+**Step 1: Join x-ocaml**
215215+216216+```bash
217217+cd ~/workspace/mono
218218+monopam join https://github.com/art-w/x-ocaml.git
219219+```
220220+221221+**Step 2: Verify the subtree exists**
222222+223223+```bash
224224+ls mono/x-ocaml/src/
225225+```
226226+227227+Expected: `backend.ml`, `cell.ml`, `client.ml`, `editor.ml`, `jtw_client.ml`, `merlin_ext.ml`, `webcomponent.ml`
228228+229229+**Step 3: Update root dune-project if needed**
230230+231231+Check if `ocamlformat-lib` or other x-ocaml deps need adding to the root package's `depends` list.
232232+233233+**Step 4: Build to verify**
234234+235235+```bash
236236+dune build x-ocaml 2>&1 | head -20
237237+```
238238+239239+**Step 5: Commit**
240240+241241+```bash
242242+git add -A && git commit -m "chore: join x-ocaml into monorepo"
243243+```
244244+245245+---
246246+247247+### Task 1.4: Sync js_top_worker
248248+249249+**Step 1: Sync js_top_worker (12 commits behind)**
250250+251251+```bash
252252+cd ~/workspace/mono
253253+monopam sync js_top_worker
254254+```
255255+256256+**Step 2: Build to verify everything compiles together**
257257+258258+```bash
259259+dune build @all 2>&1 | tail -20
260260+```
261261+262262+**Step 3: Commit**
263263+264264+```bash
265265+git add -A && git commit -m "chore: sync js_top_worker with upstream"
266266+```
267267+268268+---
269269+270270+## Workstream 2: Mobile Responsive Layout
271271+272272+### Task 2.1: Add diff computation to scrollycode extension
273273+274274+Compute diffs between consecutive steps at HTML generation time.
275275+276276+**Files:**
277277+- Modify: `odoc-scrollycode-extension/src/scrollycode_extension.ml`
278278+279279+**Step 1: Add a simple LCS-based line diff function**
280280+281281+After the `highlight_ocaml` function (~line 325), add:
282282+283283+```ocaml
284284+(** {1 Diff Computation} *)
285285+286286+type diff_line =
287287+ | Same of string
288288+ | Added of string
289289+ | Removed of string
290290+291291+(** Simple LCS-based line diff between two code strings *)
292292+let diff_lines old_code new_code =
293293+ let old_lines = String.split_on_char '\n' old_code in
294294+ let new_lines = String.split_on_char '\n' new_code in
295295+ let n = List.length old_lines in
296296+ let m = List.length new_lines in
297297+ let old_arr = Array.of_list old_lines in
298298+ let new_arr = Array.of_list new_lines in
299299+ (* LCS table *)
300300+ let dp = Array.make_matrix (n + 1) (m + 1) 0 in
301301+ for i = 1 to n do
302302+ for j = 1 to m do
303303+ if old_arr.(i-1) = new_arr.(j-1) then
304304+ dp.(i).(j) <- dp.(i-1).(j-1) + 1
305305+ else
306306+ dp.(i).(j) <- max dp.(i-1).(j) dp.(i).(j-1)
307307+ done
308308+ done;
309309+ (* Backtrack to produce diff *)
310310+ let result = ref [] in
311311+ let i = ref n and j = ref m in
312312+ while !i > 0 || !j > 0 do
313313+ if !i > 0 && !j > 0 && old_arr.(!i-1) = new_arr.(!j-1) then begin
314314+ result := Same old_arr.(!i-1) :: !result;
315315+ decr i; decr j
316316+ end else if !j > 0 && (!i = 0 || dp.(!i).(!j-1) >= dp.(!i-1).(!j)) then begin
317317+ result := Added new_arr.(!j-1) :: !result;
318318+ decr j
319319+ end else begin
320320+ result := Removed old_arr.(!i-1) :: !result;
321321+ decr i
322322+ end
323323+ done;
324324+ !result
325325+```
326326+327327+**Step 2: Add diff HTML generation function**
328328+329329+```ocaml
330330+(** Generate HTML for a diff view of a step *)
331331+let generate_diff_html ~theme ~step_index ~title ~prose ~prev_code ~curr_code =
332332+ let diff = match prev_code with
333333+ | None -> List.map (fun l -> Added l) (String.split_on_char '\n' curr_code)
334334+ | Some prev -> diff_lines prev curr_code
335335+ in
336336+ (* Generate diff lines HTML with green/red backgrounds *)
337337+ ...
338338+```
339339+340340+**Step 3: Commit**
341341+342342+```bash
343343+git add -A && git commit -m "feat(scrollycode): add LCS-based line diff computation"
344344+```
345345+346346+---
347347+348348+### Task 2.2: Generate mobile stacked layout HTML
349349+350350+**Files:**
351351+- Modify: `odoc-scrollycode-extension/src/scrollycode_extension.ml`
352352+353353+**Step 1: Add a `generate_mobile_html` function**
354354+355355+This generates the stacked layout with diff blocks per step. It's wrapped in a `<div class="sc-mobile">` that's hidden on desktop and shown on mobile via media query.
356356+357357+**Step 2: Update `generate_html` to include both layouts**
358358+359359+The main `generate_html` function now emits:
360360+- `<div class="sc-desktop">` — existing side-by-side layout (hidden on mobile)
361361+- `<div class="sc-mobile">` — stacked layout with diffs (hidden on desktop)
362362+363363+**Step 3: Add responsive CSS to all three themes**
364364+365365+Each theme's CSS gets a `@media (max-width: 700px)` block:
366366+367367+```css
368368+@media (max-width: 700px) {
369369+ .sc-desktop { display: none !important; }
370370+ .sc-mobile { display: block !important; }
371371+ .sc-mobile-step { margin: 2rem 1rem; }
372372+ .sc-diff-block { ... }
373373+ .sc-diff-added { background: rgba(0, 180, 0, 0.12); }
374374+ .sc-diff-removed { background: rgba(255, 0, 0, 0.1); text-decoration: line-through; }
375375+ .sc-diff-context { opacity: 0.5; }
376376+}
377377+@media (min-width: 701px) {
378378+ .sc-mobile { display: none !important; }
379379+}
380380+```
381381+382382+**Step 4: Rebuild and test at mobile viewport**
383383+384384+```bash
385385+# Rebuild
386386+dune build odoc-scrollycode-extension/test/odoc_scrolly.exe
387387+# Run pipeline
388388+ODOC=./_build/default/odoc-scrollycode-extension/test/odoc_scrolly.exe
389389+for f in warm_parser dark_repl notebook_testing index; do
390390+ $ODOC compile --package scrolly-demos -o /tmp/scrolly-build/page-${f}.odoc \
391391+ odoc-scrollycode-extension/test/${f}.mld
392392+done
393393+# ... link and html-generate ...
394394+```
395395+396396+Verify in Playwright at 390x844 (iPhone 14 portrait) — should show stacked layout with diffs.
397397+398398+**Step 5: Commit**
399399+400400+```bash
401401+git add -A && git commit -m "feat(scrollycode): add mobile responsive stacked layout with diff view"
402402+```
403403+404404+---
405405+406406+## Workstream 3: x-ocaml + js_top_worker Integration
407407+408408+### Task 3.1: Implement jtw_client.ml in x-ocaml
409409+410410+Wire up the js_top_worker backend in x-ocaml. The current `src/jtw_client.ml` is a stub.
411411+412412+**Files:**
413413+- Modify: `x-ocaml/src/jtw_client.ml`
414414+- Modify: `x-ocaml/src/dune` (add js_top_worker-client dependency)
415415+416416+**Step 1: Understand the protocol bridge**
417417+418418+x-ocaml uses `X_protocol` with binary Marshal. js_top_worker uses JSON-RPC. The bridge must:
419419+- Convert `X_protocol.Eval` → js_top_worker `exec` RPC
420420+- Convert `X_protocol.Merlin(_, Type_enclosing(...))` → js_top_worker `type_enclosing` RPC
421421+- Convert `X_protocol.Merlin(_, Complete_prefix(...))` → js_top_worker `complete_prefix` RPC
422422+- Convert `X_protocol.Merlin(_, All_errors(...))` → js_top_worker `query_errors` RPC
423423+- Convert responses back
424424+425425+**Step 2: Implement jtw_client.ml**
426426+427427+```ocaml
428428+(* Bridge between X_protocol and js_top_worker JSON-RPC *)
429429+type t = {
430430+ rpc : Js_top_worker_client.rpc;
431431+ mutable on_msg : X_protocol.response -> unit;
432432+}
433433+434434+let make url =
435435+ let rpc = Js_top_worker_client.start url 30000 (fun () -> ()) in
436436+ { rpc; on_msg = (fun _ -> ()) }
437437+438438+let on_message t fn = t.on_msg <- fn
439439+440440+let init t =
441441+ (* Initialize the worker *)
442442+ Lwt.async (fun () ->
443443+ let open Lwt.Syntax in
444444+ let* _result = Js_top_worker_client.W.init t.rpc
445445+ { base_url = ""; findlib_index = None } in
446446+ Lwt.return_unit)
447447+448448+let eval ~id ~line_number:_ t code =
449449+ Lwt.async (fun () ->
450450+ let open Lwt.Syntax in
451451+ let* result = Js_top_worker_client.W.exec t.rpc "" code in
452452+ match result with
453453+ | Ok exec_result ->
454454+ let outputs = (* convert exec_result to X_protocol.output list *) in
455455+ t.on_msg (Top_response (id, outputs));
456456+ Lwt.return_unit
457457+ | Error _ ->
458458+ t.on_msg (Top_response (id, []));
459459+ Lwt.return_unit)
460460+461461+let post t msg =
462462+ match msg with
463463+ | X_protocol.Eval (id, line_number, code) -> eval ~id ~line_number t code
464464+ | X_protocol.Merlin (id, action) -> handle_merlin t id action
465465+ | X_protocol.Format (id, code) -> handle_format t id code
466466+ | _ -> ()
467467+468468+(* ... handle_merlin dispatches to type_enclosing, complete_prefix, query_errors *)
469469+```
470470+471471+Note: The exact implementation will depend on the js_top_worker client API and how Lwt integrates with the x-ocaml message loop. This may need adaptation.
472472+473473+**Step 3: Update dune to add js_top_worker-client dependency**
474474+475475+**Step 4: Build and test**
476476+477477+```bash
478478+dune build x-ocaml 2>&1
479479+```
480480+481481+**Step 5: Commit**
482482+483483+```bash
484484+git add -A && git commit -m "feat(x-ocaml): implement js_top_worker backend bridge"
485485+```
486486+487487+---
488488+489489+### Task 3.2: Update scrollycode to emit x-ocaml elements
490490+491491+Replace the static HTML code blocks with `<x-ocaml>` web components.
492492+493493+**Files:**
494494+- Modify: `odoc-scrollycode-extension/src/scrollycode_extension.ml`
495495+496496+**Step 1: Update `generate_html` to emit x-ocaml tags**
497497+498498+Instead of:
499499+```html
500500+<div class="sc-code-body">
501501+ <div class="sc-line">...</div>
502502+</div>
503503+```
504504+505505+Generate:
506506+```html
507507+<script src="x-ocaml.js" src-worker="worker.js" backend="jtw"></script>
508508+<div class="sc-code-body" id="sc-code-panel">
509509+ <x-ocaml id="sc-editor" run-on="load">
510510+ (* step 1 code *)
511511+ </x-ocaml>
512512+</div>
513513+```
514514+515515+**Step 2: Update the JavaScript for step transitions**
516516+517517+When the IntersectionObserver detects a new active step, update the x-ocaml element's content:
518518+519519+```javascript
520520+// In the step observer callback:
521521+const editor = document.getElementById('sc-editor');
522522+if (editor && editor.setSource) {
523523+ editor.setSource(steps[currentStep].code);
524524+}
525525+```
526526+527527+Note: Need to check what API x-ocaml exposes for programmatic content updates.
528528+529529+**Step 3: Declare worker JS files as extension resources**
530530+531531+The extension's `extension_output` should declare the JS files as resources so odoc copies them:
532532+533533+```ocaml
534534+let resources = [
535535+ { name = "x-ocaml.js"; content = ... };
536536+ { name = "worker.js"; content = ... };
537537+]
538538+```
539539+540540+Or more likely, these will be served from a CDN or a fixed path.
541541+542542+**Step 4: Build and test in browser**
543543+544544+**Step 5: Commit**
545545+546546+```bash
547547+git add -A && git commit -m "feat(scrollycode): replace static code with x-ocaml web components"
548548+```
549549+550550+---
551551+552552+### Task 3.3: Add type-on-hover support
553553+554554+With x-ocaml + js_top_worker connected, type-on-hover should work automatically through the Merlin integration in x-ocaml's `merlin_ext.ml`. This task verifies it works and fixes any issues.
555555+556556+**Step 1: Build the full pipeline (scrollycode + x-ocaml + js_top_worker)**
557557+558558+**Step 2: Open a demo in browser, hover over identifiers**
559559+560560+Verify Merlin type tooltips appear.
561561+562562+**Step 3: Fix any integration issues**
563563+564564+Common issues:
565565+- Worker URL path incorrect
566566+- CMI files not available (findlibish index)
567567+- Position offset mismatch due to cell concatenation
568568+569569+**Step 4: Commit**
570570+571571+```bash
572572+git add -A && git commit -m "fix(scrollycode): verify and fix type-on-hover integration"
573573+```
574574+575575+---
576576+577577+## Workstream 4: Playground Overlay
578578+579579+### Task 4.1: Add "Open in Playground" button and overlay
580580+581581+**Files:**
582582+- Modify: `odoc-scrollycode-extension/src/scrollycode_extension.ml`
583583+584584+**Step 1: Add a playground button to each step**
585585+586586+In the generated HTML, each step gets a button:
587587+588588+```html
589589+<button class="sc-playground-btn" data-step="N">
590590+ ▶ Try it
591591+</button>
592592+```
593593+594594+**Step 2: Add the overlay HTML (generated once)**
595595+596596+```html
597597+<div id="sc-playground-overlay" class="sc-playground-overlay" style="display:none">
598598+ <div class="sc-playground-header">
599599+ <span class="sc-playground-title">Playground</span>
600600+ <button class="sc-playground-close">✕</button>
601601+ </div>
602602+ <div class="sc-playground-editor">
603603+ <x-ocaml id="sc-playground-editor" backend="jtw"></x-ocaml>
604604+ </div>
605605+ <div class="sc-playground-output" id="sc-playground-output"></div>
606606+</div>
607607+```
608608+609609+**Step 3: Add JavaScript to open/close and pre-load code**
610610+611611+```javascript
612612+document.querySelectorAll('.sc-playground-btn').forEach(btn => {
613613+ btn.addEventListener('click', () => {
614614+ const stepIndex = parseInt(btn.dataset.step);
615615+ // Concatenate all code up to and including this step
616616+ const code = steps.slice(0, stepIndex + 1).map(s => s.code).join('\n\n');
617617+ const overlay = document.getElementById('sc-playground-overlay');
618618+ overlay.style.display = 'flex';
619619+ const editor = document.getElementById('sc-playground-editor');
620620+ // Set editor content to accumulated code
621621+ editor.textContent = code;
622622+ // Trigger execution
623623+ });
624624+});
625625+```
626626+627627+**Step 4: Add CSS for the overlay**
628628+629629+Full-screen overlay with semi-transparent backdrop, centered editor, output panel below.
630630+631631+**Step 5: Build and test**
632632+633633+Verify: clicking "Try it" on step 3 opens overlay with code from steps 1-3, editable, with Run capability.
634634+635635+**Step 6: Commit**
636636+637637+```bash
638638+git add -A && git commit -m "feat(scrollycode): add interactive playground overlay"
639639+```
640640+641641+---
642642+643643+## Task Dependencies
644644+645645+```
646646+1.1 (merge odoc branches)
647647+ → 1.2 (extract scrollycode to standalone)
648648+ → 2.1 (diff computation)
649649+ → 2.2 (mobile layout)
650650+ → 1.3 (join x-ocaml)
651651+ → 3.1 (jtw_client bridge)
652652+ → 3.2 (emit x-ocaml elements)
653653+ → 3.3 (type-on-hover)
654654+ → 4.1 (playground overlay)
655655+ → 1.4 (sync js_top_worker)
656656+ → 3.1 (jtw_client bridge)
657657+```
658658+659659+Workstreams 2 and 3 can proceed in parallel after Task 1.2.
660660+661661+---
662662+663663+## Notes
664664+665665+- **Worker size**: js_top_worker worker.js is ~64MB uncompressed. Gzipped should be ~8-12MB. Consider showing a loading indicator while the worker initializes.
666666+- **x-ocaml API**: The exact API for programmatically updating x-ocaml content needs verification. May need to use `editor.setSource()` or recreate the element.
667667+- **Cell dependencies**: js_top_worker's cell dependency system can be used to give each step full type context from previous steps.
668668+- **Format support**: js_top_worker doesn't support code formatting — the `Format` request type in x-ocaml will be a no-op for the jtw backend.
669669+- **Testing**: Each workstream should be tested in browser with Playwright at both desktop (1280x800) and mobile (390x844) viewports.
+387
js_top_worker/docs/widget-plan.md
···11+# Widget Support Plan of Record
22+33+*Created: 2026-02-12*
44+55+## Goals
66+77+Add Jupyter-style interactive widget support to js_top_worker, enabling OCaml
88+code running in the toplevel to create interactive UI elements (sliders, buttons,
99+dropdowns, etc.) that communicate bidirectionally with the frontend.
1010+1111+## Design Principles
1212+1313+1. **Zero new dependencies in the worker** - Every dependency compiled into the
1414+ worker can conflict with libraries the user wants to load at runtime. Widget
1515+ support must not add any new OCaml dependencies to the worker.
1616+1717+2. **Broad OCaml version compatibility** - The project currently targets OCaml
1818+ >= 4.04. New features must not raise this floor unnecessarily.
1919+2020+3. **Build on the message protocol, not RPC** - The worker already uses the
2121+ message-based protocol (`message.ml` / `js_top_worker_client_msg.ml`). Widget
2222+ communication extends this protocol rather than the legacy JSON-RPC layer.
2323+2424+4. **Remove, don't accumulate** - The legacy rpclib-based communication layer
2525+ should be removed as part of this work, reducing the dependency footprint.
2626+2727+## Why Not CBOR?
2828+2929+The architecture document previously listed CBOR as a planned transport format.
3030+After investigation, we've decided against it:
3131+3232+- **Dependency risk**: Even the lightweight `cbor` opam package brings in
3333+ `ocplib-endian`. Any dependency in the worker namespace can conflict with
3434+ user-loaded libraries.
3535+- **Unnecessary complexity**: The existing message protocol uses `Js_of_ocaml`'s
3636+ native JSON handling (`Json.output` / `Json.unsafe_input`), which has zero
3737+ additional dependencies.
3838+- **Binary data via Typed Arrays**: For binary payloads (images, etc.),
3939+ `js_of_ocaml`'s `Typed_array` module provides native browser typed array
4040+ support without any extra libraries.
4141+- **JSON is the browser's native format** - No encoding/decoding overhead when
4242+ passing structured data via `postMessage`.
4343+4444+## Communication Architecture
4545+4646+### Current State (Two Parallel Layers)
4747+4848+```
4949+1. Legacy RPC (to be removed):
5050+ Client (js_top_worker_client.ml) <-> JSON-RPC <-> Server (Toplevel_api_gen)
5151+ Dependencies: rpclib, rpclib-lwt, rpclib.json, ppx_deriving_rpc
5252+5353+2. Message protocol (to be extended):
5454+ Client (js_top_worker_client_msg.ml) <-> JSON messages <-> Worker (worker.ml)
5555+ Dependencies: js_of_ocaml (already required)
5656+```
5757+5858+### Target State
5959+6060+```
6161+Client (js_top_worker_client_msg.ml) <-> JSON messages <-> Worker (worker.ml)
6262+ | |
6363+ |-- Request/Response (existing: eval, complete, errors, ...) |
6464+ |-- Push messages (existing: output_at streaming) |
6565+ |-- Widget messages (NEW: comm_open, comm_update, comm_msg, ...) |
6666+```
6767+6868+All communication uses the existing `message.ml` protocol extended with widget
6969+message types. No new serialization libraries.
7070+7171+## Widget Protocol Design
7272+7373+### Message Types (Worker -> Client)
7474+7575+```
7676+CommOpen { comm_id; target; state } -- Widget created by OCaml code
7777+CommUpdate { comm_id; state } -- Widget state changed
7878+CommClose { comm_id } -- Widget destroyed
7979+```
8080+8181+### Message Types (Client -> Worker)
8282+8383+```
8484+CommMsg { comm_id; data } -- Frontend event (click, value change)
8585+CommClose { comm_id } -- Frontend closed widget
8686+```
8787+8888+### Widget State Format
8989+9090+Widget state is a JSON object with well-known keys, following the Jupyter widget
9191+convention where practical:
9292+9393+```json
9494+{
9595+ "widget_type": "slider",
9696+ "value": 50,
9797+ "min": 0,
9898+ "max": 100,
9999+ "step": 1,
100100+ "description": "Threshold",
101101+ "disabled": false
102102+}
103103+```
104104+105105+The `widget_type` field replaces Jupyter's `_model_module` / `_model_name` /
106106+`_view_module` / `_view_name` quartet, since we don't need the npm module
107107+indirection - our widget renderers are built into the client.
108108+109109+### Alignment with Jupyter Protocol
110110+111111+We adopt the **concepts** from the Jupyter widget protocol but simplify the
112112+implementation:
113113+114114+| Jupyter Concept | Our Equivalent |
115115+|-----------------|----------------|
116116+| comm_open | CommOpen message |
117117+| comm_msg method:"update" | CommUpdate message |
118118+| comm_msg method:"custom" | CommMsg message |
119119+| comm_close | CommClose message |
120120+| _model_module + _model_name | widget_type string |
121121+| buffer_paths (binary) | Typed_array via js_of_ocaml |
122122+| Display message | CommOpen includes display flag |
123123+124124+We do **not** implement:
125125+- echo_update (single frontend, no multi-client sync needed)
126126+- request_state / request_states (state is authoritative in worker)
127127+- Version negotiation (internal protocol, not cross-system)
128128+129129+## OCaml Widget API
130130+131131+User-facing API available as an OCaml library in the toplevel:
132132+133133+```ocaml
134134+module Widget : sig
135135+ type t
136136+137137+ (** Create a widget. Returns it and displays it. *)
138138+ val slider : ?min:int -> ?max:int -> ?step:int ->
139139+ ?description:string -> int -> t
140140+141141+ val button : ?style:string -> string -> t
142142+143143+ val text : ?placeholder:string -> ?description:string -> string -> t
144144+145145+ val dropdown : ?description:string -> options:string list -> string -> t
146146+147147+ val checkbox : ?description:string -> bool -> t
148148+149149+ val html : string -> t
150150+151151+ (** Read current value *)
152152+ val get : t -> Yojson.Safe.t (* or a simpler JSON type *)
153153+154154+ (** Update widget state *)
155155+ val set : t -> string -> Yojson.Safe.t -> unit
156156+157157+ (** Register event handler *)
158158+ val on_change : t -> (Yojson.Safe.t -> unit) -> unit
159159+ val on_click : t -> (unit -> unit) -> unit
160160+161161+ (** Display / close *)
162162+ val display : t -> unit
163163+ val close : t -> unit
164164+end
165165+```
166166+167167+**Important**: This API library (`widget` or similar) runs *inside* the toplevel
168168+and must have minimal dependencies. It communicates with the frontend by pushing
169169+messages through the same channel as `Mime_printer`.
170170+171171+## Code Removal Plan
172172+173173+### Files to Remove
174174+175175+| File | Reason |
176176+|------|--------|
177177+| `idl/transport.ml`, `transport.mli` | JSON-RPC transport wrapper |
178178+| `idl/js_top_worker_client.ml`, `.mli` | RPC-based Lwt client |
179179+| `idl/js_top_worker_client_fut.ml` | RPC-based Fut client |
180180+| `idl/_old/` directory | Historical RPC reference code |
181181+182182+### Dependencies to Remove
183183+184184+| Package | Used By |
185185+|---------|---------|
186186+| `rpclib` | transport.ml, RPC clients |
187187+| `rpclib-lwt` | impl.ml (IdlM module) |
188188+| `rpclib.json` | transport.ml, RPC clients |
189189+| `ppx_deriving_rpc` | toplevel_api.ml code generation |
190190+| `xmlm` | transitive via rpclib |
191191+| `cmdliner` | transitive via rpclib |
192192+193193+### Files to Refactor
194194+195195+| File | Change |
196196+|------|--------|
197197+| `lib/impl.ml` | Remove `IdlM` / `Rpc_lwt` usage, use own types |
198198+| `idl/toplevel_api.ml` | Keep type definitions, remove RPC IDL machinery |
199199+| `idl/toplevel_api_gen.ml` | Replace with hand-written types (no ppx_deriving_rpc) |
200200+| `idl/dune` | Remove rpclib library deps and ppx rules |
201201+| `lib/dune` | Remove rpclib-lwt dep |
202202+| `test/node/*.ml` | Migrate from RPC Server/Client to message protocol |
203203+| `test/browser/client_test.ml` | Use message-based client |
204204+| `example/unix_worker.ml` | Use message protocol over Unix socket |
205205+| `example/unix_client.ml` | Use message protocol over Unix socket |
206206+207207+### Impact on `toplevel_api_gen.ml`
208208+209209+The generated file is 92k+ lines (from ppx_deriving_rpc). The types it defines
210210+are used extensively in `impl.ml` and `worker.ml`. The plan:
211211+212212+1. Extract the **type definitions** into a new lightweight module (no ppx)
213213+2. Hand-write any needed serialization for the message protocol
214214+3. Remove `ppx_deriving_rpc` dependency entirely
215215+4. Delete `toplevel_api_gen.ml`
216216+217217+## Testing Strategy
218218+219219+### Existing Test Infrastructure
220220+221221+| Backend | Location | Framework |
222222+|---------|----------|-----------|
223223+| Unit tests | `test/libtest/` | ppx_expect |
224224+| Node.js | `test/node/` | js_of_ocaml + Node |
225225+| Unix (cram) | `test/cram/` | Cram tests with unix_worker |
226226+| Browser | `test/browser/` | Playwright + Chromium |
227227+228228+### Widget Testing Approach
229229+230230+#### 1. Unit Tests (`test/libtest/`)
231231+232232+- Widget state management logic
233233+- Message serialization/deserialization for widget messages
234234+- CommManager state tracking (open/update/close lifecycle)
235235+236236+#### 2. Node.js Tests (`test/node/`)
237237+238238+- Widget creation produces correct CommOpen messages
239239+- Widget state updates produce CommUpdate messages
240240+- Event handler registration and dispatch
241241+- Widget close cleanup
242242+- Multiple simultaneous widgets
243243+- Widget interaction with regular exec output (mime_vals + widgets)
244244+245245+#### 3. Cram Tests (`test/cram/`)
246246+247247+- Unix worker handles widget messages over socket
248248+- Widget lifecycle via command-line client
249249+- Widget messages interleaved with regular eval output
250250+251251+#### 4. Browser Tests (`test/browser/`)
252252+253253+- **End-to-end widget rendering**: OCaml creates widget -> message sent ->
254254+ client renders DOM element -> user interaction -> event sent back -> OCaml
255255+ handler fires
256256+- **Widget types**: Test each widget type (slider, button, text, dropdown,
257257+ checkbox, html)
258258+- **State synchronization**: Frontend changes propagated to worker and back
259259+- **Multiple widgets**: Several widgets active simultaneously
260260+- **Widget cleanup**: Closing widgets removes DOM elements
261261+- **Integration with existing features**: Widgets alongside code completion,
262262+ error reporting, MIME output
263263+264264+### Test Utilities
265265+266266+A shared test helper module for widget testing:
267267+268268+```ocaml
269269+(* test/test_widget_helpers.ml *)
270270+val assert_comm_open : worker_msg -> comm_id:string -> widget_type:string -> unit
271271+val assert_comm_update : worker_msg -> comm_id:string -> key:string -> unit
272272+val assert_comm_close : worker_msg -> comm_id:string -> unit
273273+val simulate_event : comm_id:string -> data:string -> client_msg
274274+```
275275+276276+## Example Widgets
277277+278278+### Priority 1: Core Widgets (Implement First)
279279+280280+These are the most commonly used Jupyter widgets and cover the fundamental
281281+interaction patterns:
282282+283283+| Widget | State | Events | Jupyter Equivalent |
284284+|--------|-------|--------|--------------------|
285285+| IntSlider | value, min, max, step | on_change(int) | IntSlider |
286286+| Button | description, style | on_click | Button |
287287+| Text | value, placeholder | on_change(string) | Text |
288288+| Dropdown | value, options | on_change(string) | Dropdown |
289289+| Checkbox | value, description | on_change(bool) | Checkbox |
290290+| HTML | value (html string) | none | HTML |
291291+292292+### Priority 2: Composition Widgets
293293+294294+| Widget | Purpose | Jupyter Equivalent |
295295+|--------|---------|-------------------|
296296+| HBox / VBox | Layout containers | HBox / VBox |
297297+| Output | Capture stdout/display | Output |
298298+| FloatSlider | Decimal slider | FloatSlider |
299299+300300+### Priority 3: Domain-Specific Widgets
301301+302302+| Widget | Purpose | Inspired By |
303303+|--------|---------|-------------|
304304+| Plot | Simple 2D charts (SVG) | bqplot (simplified) |
305305+| Table | Data grid display | ipydatagrid (read-only) |
306306+| Image | Display image bytes | Image widget |
307307+308308+### Example: Interactive Slider
309309+310310+```ocaml
311311+(* User code in toplevel *)
312312+let threshold = Widget.slider ~min:0 ~max:100 ~description:"Threshold" 50;;
313313+314314+Widget.on_change threshold (fun v ->
315315+ let n = Widget.Int.of_json v in
316316+ Printf.printf "Threshold changed to: %d\n" n
317317+);;
318318+```
319319+320320+### Example: Button with Output
321321+322322+```ocaml
323323+let count = ref 0;;
324324+let label = Widget.html (Printf.sprintf "<b>Count: %d</b>" !count);;
325325+let btn = Widget.button "Increment";;
326326+327327+Widget.on_click btn (fun () ->
328328+ incr count;
329329+ Widget.set label "value"
330330+ (`String (Printf.sprintf "<b>Count: %d</b>" !count))
331331+);;
332332+```
333333+334334+### Example: Linked Widgets
335335+336336+```ocaml
337337+let slider = Widget.slider ~min:0 ~max:255 ~description:"Red" 128;;
338338+let preview = Widget.html {|<div style="width:50px;height:50px"></div>|};;
339339+340340+Widget.on_change slider (fun v ->
341341+ let r = Widget.Int.of_json v in
342342+ Widget.set preview "value"
343343+ (`String (Printf.sprintf
344344+ {|<div style="width:50px;height:50px;background:rgb(%d,0,0)"></div>|} r))
345345+);;
346346+```
347347+348348+## Implementation Phases
349349+350350+### Phase 1: RPC Removal & Type Cleanup
351351+352352+Remove the legacy RPC layer and establish clean type definitions.
353353+354354+### Phase 2: Widget Message Protocol
355355+356356+Extend `message.ml` and `worker.ml` with widget message types.
357357+358358+### Phase 3: Widget Manager (Worker Side)
359359+360360+Implement the comm manager that tracks widget state and routes events.
361361+362362+### Phase 4: OCaml Widget API
363363+364364+Create the user-facing `Widget` module available in the toplevel.
365365+366366+### Phase 5: JavaScript Widget Renderer
367367+368368+Implement widget rendering in the JavaScript client.
369369+370370+### Phase 6: Testing & Examples
371371+372372+Full test coverage across all backends, example widgets, documentation.
373373+374374+## Open Questions
375375+376376+1. **JSON representation in OCaml API**: Use `Yojson.Safe.t`? A custom minimal
377377+ JSON type? Raw `Js.Unsafe.any`? (Yojson adds a dependency; custom type is
378378+ more work; Unsafe.any is untyped.)
379379+380380+2. **Widget library loading**: Should the Widget module be preloaded in the
381381+ worker, or loaded on demand via `#require "widget"`?
382382+383383+3. **Layout model**: How much of Jupyter's CSS-based layout model to support?
384384+ Full flexbox control per-widget, or simpler HBox/VBox only?
385385+386386+4. **Persistence**: Should widget state survive cell re-execution? Jupyter
387387+ widgets are destroyed and recreated; we could do the same or preserve state.