My aggregated monorepo of OCaml code, automaintained

chore: stage pending changes before monopam sync

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

+1077 -42
+1 -3
day10/day10-web.opam
··· 5 5 maintainer: ["Maintainer Name <maintainer@example.com>"] 6 6 authors: ["Author Name <author@example.com>"] 7 7 license: "LICENSE" 8 - homepage: "https://github.com/username/reponame" 9 8 doc: "https://url/to/documentation" 10 - bug-reports: "https://github.com/username/reponame/issues" 11 9 depends: [ 12 10 "ocaml" {>= "5.3.0"} 13 11 "dune" {>= "3.17"} ··· 30 28 "@doc" {with-doc} 31 29 ] 32 30 ] 33 - dev-repo: "git+https://github.com/username/reponame.git" 31 + dev-repo: "https://tangled.org/jon.recoil.org/day10"
+1 -3
day10/day10.opam
··· 6 6 authors: ["Author Name <author@example.com>"] 7 7 license: "LICENSE" 8 8 tags: ["add topics" "to describe" "your" "project"] 9 - homepage: "https://github.com/username/reponame" 10 9 doc: "https://url/to/documentation" 11 - bug-reports: "https://github.com/username/reponame/issues" 12 10 depends: [ 13 11 "ocaml" {>= "5.3.0"} 14 12 "dune" {>= "3.17"} ··· 32 30 "@doc" {with-doc} 33 31 ] 34 32 ] 35 - dev-repo: "git+https://github.com/username/reponame.git" 33 + dev-repo: "https://tangled.org/jon.recoil.org/day10" 36 34 pin-depends: [ 37 35 ["opam-client.2.4.1" "git+https://github.com/dra27/opam#6693-2.4.1"] 38 36 ["opam-core.2.4.1" "git+https://github.com/dra27/opam#6693-2.4.1"]
+1 -1
day10/dune-project
··· 5 5 (generate_opam_files true) 6 6 7 7 (source 8 - (github username/reponame)) 8 + (uri https://tangled.org/jon.recoil.org/day10)) 9 9 10 10 (authors "Author Name <author@example.com>") 11 11
+669
docs/plans/2026-02-16-scrollycode-monorepo-integration.md
··· 1 + # Scrollycode Monorepo Integration Plan 2 + 3 + > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. 4 + 5 + **Goal:** Consolidate odoc (with extension-plugins), js_top_worker, x-ocaml, and a standalone scrollycode extension into the monorepo, then add mobile responsiveness and x-ocaml/js_top_worker integration for interactive code tutorials. 6 + 7 + **Architecture:** Four workstreams executed in sequence: (1) monorepo consolidation via monopam, (2) mobile responsive CSS with diff view, (3) x-ocaml + js_top_worker integration for type-on-hover and live execution, (4) "Open in Playground" overlay. The monorepo work unblocks everything else by getting all packages building together. 8 + 9 + **Tech Stack:** OCaml, dune, monopam (git subtree monorepo manager), odoc extension API, x-ocaml web component, js_top_worker (browser OCaml toplevel), CodeMirror 6, CSS media queries, IntersectionObserver. 10 + 11 + --- 12 + 13 + ## Workstream 1: Monorepo Consolidation 14 + 15 + ### Task 1.1: Merge extension-plugins into odoc staging 16 + 17 + The monorepo tracks odoc on `jonludlam/staging`. The `extension-plugins` branch has 6 commits staging doesn't have (custom tags + dune-site plugin system). Staging has 41 commits extension-plugins doesn't have. They diverge from `d8460cda`. 18 + 19 + **Files:** 20 + - Modify: `~/workspace/odoc/` (git merge) 21 + 22 + **Step 1: Create a working branch from staging** 23 + 24 + ```bash 25 + cd ~/workspace/odoc 26 + git fetch jonludlam staging extension-plugins 27 + git checkout -b staging-with-extensions jonludlam/staging 28 + ``` 29 + 30 + **Step 2: Merge extension-plugins into the new branch** 31 + 32 + ```bash 33 + git merge jonludlam/extension-plugins 34 + ``` 35 + 36 + Expected: Merge conflicts in ~5-10 files. Key conflict areas: 37 + - `src/parser/ast.ml` — both add Custom tag support 38 + - `src/odoc/bin/main.ml` — both add extension CLI code 39 + - `src/model/comment.ml` — tag handling logic 40 + - `src/document/comment.ml` — custom tag rendering 41 + - Build files (`dune-project`, `odoc.opam`, various `dune` files) 42 + 43 + **Step 3: Resolve conflicts** 44 + 45 + For each conflict: 46 + - Staging's extension infrastructure is more mature (41 commits of iteration) 47 + - Extension-plugins adds the dune-site dynamic loading that staging may not have 48 + - Prefer staging's versions where they cover the same functionality 49 + - Keep extension-plugins' dune-site plugin loading (`src/odoc/bin/main.ml` — `Sites.Plugins.Extensions.load_all ()`) 50 + 51 + **Step 4: Build and test** 52 + 53 + ```bash 54 + dune build @all 2>&1 | head -50 55 + dune runtest 2>&1 | tail -20 56 + ``` 57 + 58 + Expected: Clean build. If tests fail, fix incrementally. 59 + 60 + **Step 5: Cherry-pick the scrollycode-demos commit** 61 + 62 + ```bash 63 + git cherry-pick 8f6c4537f 64 + ``` 65 + 66 + This adds `src/extensions/scrollycode/` and `test/scrollycode-demos/` — the scrollycode extension (which will be extracted in Task 1.2). 67 + 68 + **Step 6: Push the merged branch** 69 + 70 + ```bash 71 + git push jonludlam staging-with-extensions 72 + ``` 73 + 74 + Then update `~/workspace/mono/sources.toml` to track this branch: 75 + 76 + ```toml 77 + [odoc] 78 + url = "git+https://github.com/jonludlam/odoc.git" 79 + upstream = "git+https://github.com/ocaml/odoc.git" 80 + branch = "staging-with-extensions" 81 + ``` 82 + 83 + **Step 7: Sync odoc in monorepo** 84 + 85 + ```bash 86 + cd ~/workspace/mono 87 + monopam sync odoc 88 + ``` 89 + 90 + **Step 8: Commit** 91 + 92 + ```bash 93 + git add -A && git commit -m "chore: sync odoc with extension-plugins merge" 94 + ``` 95 + 96 + --- 97 + 98 + ### Task 1.2: Extract scrollycode extension into standalone repo 99 + 100 + Create `odoc-scrollycode-extension` as a standalone package in the monorepo, following the pattern of `odoc-rfc-extension` and `odoc-admonition-extension`. 101 + 102 + **Files:** 103 + - Create: `mono/odoc-scrollycode-extension/dune-project` 104 + - Create: `mono/odoc-scrollycode-extension/src/dune` 105 + - Create: `mono/odoc-scrollycode-extension/src/scrollycode_extension.ml` (move from odoc) 106 + - Create: `mono/odoc-scrollycode-extension/test/` (move demo files from odoc) 107 + 108 + **Step 1: Create directory structure** 109 + 110 + ```bash 111 + cd ~/workspace/mono 112 + mkdir -p odoc-scrollycode-extension/src odoc-scrollycode-extension/test 113 + ``` 114 + 115 + **Step 2: Create dune-project** 116 + 117 + Write `odoc-scrollycode-extension/dune-project`: 118 + 119 + ``` 120 + (lang dune 3.18) 121 + (using dune_site 0.1) 122 + (name odoc-scrollycode-extension) 123 + (generate_opam_files true) 124 + 125 + (package 126 + (name odoc-scrollycode-extension) 127 + (synopsis "Scrollycode tutorial extension for odoc") 128 + (description 129 + "Provides @scrolly.warm, @scrolly.dark, and @scrolly.notebook tags 130 + for creating scroll-driven code tutorials in odoc documentation") 131 + (depends 132 + (ocaml (>= 4.14)) 133 + odoc 134 + odoc_model 135 + odoc_document)) 136 + ``` 137 + 138 + **Step 3: Create src/dune** 139 + 140 + ``` 141 + (library 142 + (public_name odoc-scrollycode-extension.impl) 143 + (name scrollycode_extension) 144 + (libraries odoc.extension_api odoc_model odoc_document)) 145 + 146 + (plugin 147 + (name odoc-scrollycode-extension) 148 + (libraries odoc-scrollycode-extension.impl) 149 + (site (odoc extensions))) 150 + ``` 151 + 152 + **Step 4: Move the extension source** 153 + 154 + ```bash 155 + cp odoc/src/extensions/scrollycode/scrollycode_extension.ml \ 156 + odoc-scrollycode-extension/src/scrollycode_extension.ml 157 + ``` 158 + 159 + **Step 5: Move demo files to test/** 160 + 161 + ```bash 162 + cp odoc/test/scrollycode-demos/*.mld odoc-scrollycode-extension/test/ 163 + cp odoc/test/scrollycode-demos/odoc_scrolly.ml odoc-scrollycode-extension/test/ 164 + cp odoc/test/scrollycode-demos/odoc_scrolly_main.ml odoc-scrollycode-extension/test/ 165 + cp odoc/test/scrollycode-demos/dune odoc-scrollycode-extension/test/ 166 + ``` 167 + 168 + Update `test/dune` to use the public library name: 169 + 170 + ``` 171 + (executable 172 + (name odoc_scrolly) 173 + (libraries 174 + cmdliner 175 + odoc_model 176 + odoc_odoc 177 + odoc_extension_api 178 + odoc-scrollycode-extension.impl)) 179 + ``` 180 + 181 + **Step 6: Remove scrollycode from odoc tree** 182 + 183 + ```bash 184 + rm -rf odoc/src/extensions/scrollycode/ 185 + rm -rf odoc/test/scrollycode-demos/ 186 + ``` 187 + 188 + **Step 7: Build to verify** 189 + 190 + ```bash 191 + dune build odoc-scrollycode-extension 2>&1 192 + ``` 193 + 194 + Expected: Clean build. 195 + 196 + **Step 8: Commit** 197 + 198 + ```bash 199 + git add -A && git commit -m "feat: extract odoc-scrollycode-extension into standalone package" 200 + ``` 201 + 202 + **Step 9: Fork into its own repo** 203 + 204 + ```bash 205 + monopam fork odoc-scrollycode-extension 206 + ``` 207 + 208 + This creates `src/odoc-scrollycode-extension/` with extracted git history and re-adds `mono/odoc-scrollycode-extension/` as a proper subtree. 209 + 210 + --- 211 + 212 + ### Task 1.3: Join x-ocaml into monorepo 213 + 214 + **Step 1: Join x-ocaml** 215 + 216 + ```bash 217 + cd ~/workspace/mono 218 + monopam join https://github.com/art-w/x-ocaml.git 219 + ``` 220 + 221 + **Step 2: Verify the subtree exists** 222 + 223 + ```bash 224 + ls mono/x-ocaml/src/ 225 + ``` 226 + 227 + Expected: `backend.ml`, `cell.ml`, `client.ml`, `editor.ml`, `jtw_client.ml`, `merlin_ext.ml`, `webcomponent.ml` 228 + 229 + **Step 3: Update root dune-project if needed** 230 + 231 + Check if `ocamlformat-lib` or other x-ocaml deps need adding to the root package's `depends` list. 232 + 233 + **Step 4: Build to verify** 234 + 235 + ```bash 236 + dune build x-ocaml 2>&1 | head -20 237 + ``` 238 + 239 + **Step 5: Commit** 240 + 241 + ```bash 242 + git add -A && git commit -m "chore: join x-ocaml into monorepo" 243 + ``` 244 + 245 + --- 246 + 247 + ### Task 1.4: Sync js_top_worker 248 + 249 + **Step 1: Sync js_top_worker (12 commits behind)** 250 + 251 + ```bash 252 + cd ~/workspace/mono 253 + monopam sync js_top_worker 254 + ``` 255 + 256 + **Step 2: Build to verify everything compiles together** 257 + 258 + ```bash 259 + dune build @all 2>&1 | tail -20 260 + ``` 261 + 262 + **Step 3: Commit** 263 + 264 + ```bash 265 + git add -A && git commit -m "chore: sync js_top_worker with upstream" 266 + ``` 267 + 268 + --- 269 + 270 + ## Workstream 2: Mobile Responsive Layout 271 + 272 + ### Task 2.1: Add diff computation to scrollycode extension 273 + 274 + Compute diffs between consecutive steps at HTML generation time. 275 + 276 + **Files:** 277 + - Modify: `odoc-scrollycode-extension/src/scrollycode_extension.ml` 278 + 279 + **Step 1: Add a simple LCS-based line diff function** 280 + 281 + After the `highlight_ocaml` function (~line 325), add: 282 + 283 + ```ocaml 284 + (** {1 Diff Computation} *) 285 + 286 + type diff_line = 287 + | Same of string 288 + | Added of string 289 + | Removed of string 290 + 291 + (** Simple LCS-based line diff between two code strings *) 292 + let diff_lines old_code new_code = 293 + let old_lines = String.split_on_char '\n' old_code in 294 + let new_lines = String.split_on_char '\n' new_code in 295 + let n = List.length old_lines in 296 + let m = List.length new_lines in 297 + let old_arr = Array.of_list old_lines in 298 + let new_arr = Array.of_list new_lines in 299 + (* LCS table *) 300 + let dp = Array.make_matrix (n + 1) (m + 1) 0 in 301 + for i = 1 to n do 302 + for j = 1 to m do 303 + if old_arr.(i-1) = new_arr.(j-1) then 304 + dp.(i).(j) <- dp.(i-1).(j-1) + 1 305 + else 306 + dp.(i).(j) <- max dp.(i-1).(j) dp.(i).(j-1) 307 + done 308 + done; 309 + (* Backtrack to produce diff *) 310 + let result = ref [] in 311 + let i = ref n and j = ref m in 312 + while !i > 0 || !j > 0 do 313 + if !i > 0 && !j > 0 && old_arr.(!i-1) = new_arr.(!j-1) then begin 314 + result := Same old_arr.(!i-1) :: !result; 315 + decr i; decr j 316 + end else if !j > 0 && (!i = 0 || dp.(!i).(!j-1) >= dp.(!i-1).(!j)) then begin 317 + result := Added new_arr.(!j-1) :: !result; 318 + decr j 319 + end else begin 320 + result := Removed old_arr.(!i-1) :: !result; 321 + decr i 322 + end 323 + done; 324 + !result 325 + ``` 326 + 327 + **Step 2: Add diff HTML generation function** 328 + 329 + ```ocaml 330 + (** Generate HTML for a diff view of a step *) 331 + let generate_diff_html ~theme ~step_index ~title ~prose ~prev_code ~curr_code = 332 + let diff = match prev_code with 333 + | None -> List.map (fun l -> Added l) (String.split_on_char '\n' curr_code) 334 + | Some prev -> diff_lines prev curr_code 335 + in 336 + (* Generate diff lines HTML with green/red backgrounds *) 337 + ... 338 + ``` 339 + 340 + **Step 3: Commit** 341 + 342 + ```bash 343 + git add -A && git commit -m "feat(scrollycode): add LCS-based line diff computation" 344 + ``` 345 + 346 + --- 347 + 348 + ### Task 2.2: Generate mobile stacked layout HTML 349 + 350 + **Files:** 351 + - Modify: `odoc-scrollycode-extension/src/scrollycode_extension.ml` 352 + 353 + **Step 1: Add a `generate_mobile_html` function** 354 + 355 + This generates the stacked layout with diff blocks per step. It's wrapped in a `<div class="sc-mobile">` that's hidden on desktop and shown on mobile via media query. 356 + 357 + **Step 2: Update `generate_html` to include both layouts** 358 + 359 + The main `generate_html` function now emits: 360 + - `<div class="sc-desktop">` — existing side-by-side layout (hidden on mobile) 361 + - `<div class="sc-mobile">` — stacked layout with diffs (hidden on desktop) 362 + 363 + **Step 3: Add responsive CSS to all three themes** 364 + 365 + Each theme's CSS gets a `@media (max-width: 700px)` block: 366 + 367 + ```css 368 + @media (max-width: 700px) { 369 + .sc-desktop { display: none !important; } 370 + .sc-mobile { display: block !important; } 371 + .sc-mobile-step { margin: 2rem 1rem; } 372 + .sc-diff-block { ... } 373 + .sc-diff-added { background: rgba(0, 180, 0, 0.12); } 374 + .sc-diff-removed { background: rgba(255, 0, 0, 0.1); text-decoration: line-through; } 375 + .sc-diff-context { opacity: 0.5; } 376 + } 377 + @media (min-width: 701px) { 378 + .sc-mobile { display: none !important; } 379 + } 380 + ``` 381 + 382 + **Step 4: Rebuild and test at mobile viewport** 383 + 384 + ```bash 385 + # Rebuild 386 + dune build odoc-scrollycode-extension/test/odoc_scrolly.exe 387 + # Run pipeline 388 + ODOC=./_build/default/odoc-scrollycode-extension/test/odoc_scrolly.exe 389 + for f in warm_parser dark_repl notebook_testing index; do 390 + $ODOC compile --package scrolly-demos -o /tmp/scrolly-build/page-${f}.odoc \ 391 + odoc-scrollycode-extension/test/${f}.mld 392 + done 393 + # ... link and html-generate ... 394 + ``` 395 + 396 + Verify in Playwright at 390x844 (iPhone 14 portrait) — should show stacked layout with diffs. 397 + 398 + **Step 5: Commit** 399 + 400 + ```bash 401 + git add -A && git commit -m "feat(scrollycode): add mobile responsive stacked layout with diff view" 402 + ``` 403 + 404 + --- 405 + 406 + ## Workstream 3: x-ocaml + js_top_worker Integration 407 + 408 + ### Task 3.1: Implement jtw_client.ml in x-ocaml 409 + 410 + Wire up the js_top_worker backend in x-ocaml. The current `src/jtw_client.ml` is a stub. 411 + 412 + **Files:** 413 + - Modify: `x-ocaml/src/jtw_client.ml` 414 + - Modify: `x-ocaml/src/dune` (add js_top_worker-client dependency) 415 + 416 + **Step 1: Understand the protocol bridge** 417 + 418 + x-ocaml uses `X_protocol` with binary Marshal. js_top_worker uses JSON-RPC. The bridge must: 419 + - Convert `X_protocol.Eval` → js_top_worker `exec` RPC 420 + - Convert `X_protocol.Merlin(_, Type_enclosing(...))` → js_top_worker `type_enclosing` RPC 421 + - Convert `X_protocol.Merlin(_, Complete_prefix(...))` → js_top_worker `complete_prefix` RPC 422 + - Convert `X_protocol.Merlin(_, All_errors(...))` → js_top_worker `query_errors` RPC 423 + - Convert responses back 424 + 425 + **Step 2: Implement jtw_client.ml** 426 + 427 + ```ocaml 428 + (* Bridge between X_protocol and js_top_worker JSON-RPC *) 429 + type t = { 430 + rpc : Js_top_worker_client.rpc; 431 + mutable on_msg : X_protocol.response -> unit; 432 + } 433 + 434 + let make url = 435 + let rpc = Js_top_worker_client.start url 30000 (fun () -> ()) in 436 + { rpc; on_msg = (fun _ -> ()) } 437 + 438 + let on_message t fn = t.on_msg <- fn 439 + 440 + let init t = 441 + (* Initialize the worker *) 442 + Lwt.async (fun () -> 443 + let open Lwt.Syntax in 444 + let* _result = Js_top_worker_client.W.init t.rpc 445 + { base_url = ""; findlib_index = None } in 446 + Lwt.return_unit) 447 + 448 + let eval ~id ~line_number:_ t code = 449 + Lwt.async (fun () -> 450 + let open Lwt.Syntax in 451 + let* result = Js_top_worker_client.W.exec t.rpc "" code in 452 + match result with 453 + | Ok exec_result -> 454 + let outputs = (* convert exec_result to X_protocol.output list *) in 455 + t.on_msg (Top_response (id, outputs)); 456 + Lwt.return_unit 457 + | Error _ -> 458 + t.on_msg (Top_response (id, [])); 459 + Lwt.return_unit) 460 + 461 + let post t msg = 462 + match msg with 463 + | X_protocol.Eval (id, line_number, code) -> eval ~id ~line_number t code 464 + | X_protocol.Merlin (id, action) -> handle_merlin t id action 465 + | X_protocol.Format (id, code) -> handle_format t id code 466 + | _ -> () 467 + 468 + (* ... handle_merlin dispatches to type_enclosing, complete_prefix, query_errors *) 469 + ``` 470 + 471 + Note: The exact implementation will depend on the js_top_worker client API and how Lwt integrates with the x-ocaml message loop. This may need adaptation. 472 + 473 + **Step 3: Update dune to add js_top_worker-client dependency** 474 + 475 + **Step 4: Build and test** 476 + 477 + ```bash 478 + dune build x-ocaml 2>&1 479 + ``` 480 + 481 + **Step 5: Commit** 482 + 483 + ```bash 484 + git add -A && git commit -m "feat(x-ocaml): implement js_top_worker backend bridge" 485 + ``` 486 + 487 + --- 488 + 489 + ### Task 3.2: Update scrollycode to emit x-ocaml elements 490 + 491 + Replace the static HTML code blocks with `<x-ocaml>` web components. 492 + 493 + **Files:** 494 + - Modify: `odoc-scrollycode-extension/src/scrollycode_extension.ml` 495 + 496 + **Step 1: Update `generate_html` to emit x-ocaml tags** 497 + 498 + Instead of: 499 + ```html 500 + <div class="sc-code-body"> 501 + <div class="sc-line">...</div> 502 + </div> 503 + ``` 504 + 505 + Generate: 506 + ```html 507 + <script src="x-ocaml.js" src-worker="worker.js" backend="jtw"></script> 508 + <div class="sc-code-body" id="sc-code-panel"> 509 + <x-ocaml id="sc-editor" run-on="load"> 510 + (* step 1 code *) 511 + </x-ocaml> 512 + </div> 513 + ``` 514 + 515 + **Step 2: Update the JavaScript for step transitions** 516 + 517 + When the IntersectionObserver detects a new active step, update the x-ocaml element's content: 518 + 519 + ```javascript 520 + // In the step observer callback: 521 + const editor = document.getElementById('sc-editor'); 522 + if (editor && editor.setSource) { 523 + editor.setSource(steps[currentStep].code); 524 + } 525 + ``` 526 + 527 + Note: Need to check what API x-ocaml exposes for programmatic content updates. 528 + 529 + **Step 3: Declare worker JS files as extension resources** 530 + 531 + The extension's `extension_output` should declare the JS files as resources so odoc copies them: 532 + 533 + ```ocaml 534 + let resources = [ 535 + { name = "x-ocaml.js"; content = ... }; 536 + { name = "worker.js"; content = ... }; 537 + ] 538 + ``` 539 + 540 + Or more likely, these will be served from a CDN or a fixed path. 541 + 542 + **Step 4: Build and test in browser** 543 + 544 + **Step 5: Commit** 545 + 546 + ```bash 547 + git add -A && git commit -m "feat(scrollycode): replace static code with x-ocaml web components" 548 + ``` 549 + 550 + --- 551 + 552 + ### Task 3.3: Add type-on-hover support 553 + 554 + With x-ocaml + js_top_worker connected, type-on-hover should work automatically through the Merlin integration in x-ocaml's `merlin_ext.ml`. This task verifies it works and fixes any issues. 555 + 556 + **Step 1: Build the full pipeline (scrollycode + x-ocaml + js_top_worker)** 557 + 558 + **Step 2: Open a demo in browser, hover over identifiers** 559 + 560 + Verify Merlin type tooltips appear. 561 + 562 + **Step 3: Fix any integration issues** 563 + 564 + Common issues: 565 + - Worker URL path incorrect 566 + - CMI files not available (findlibish index) 567 + - Position offset mismatch due to cell concatenation 568 + 569 + **Step 4: Commit** 570 + 571 + ```bash 572 + git add -A && git commit -m "fix(scrollycode): verify and fix type-on-hover integration" 573 + ``` 574 + 575 + --- 576 + 577 + ## Workstream 4: Playground Overlay 578 + 579 + ### Task 4.1: Add "Open in Playground" button and overlay 580 + 581 + **Files:** 582 + - Modify: `odoc-scrollycode-extension/src/scrollycode_extension.ml` 583 + 584 + **Step 1: Add a playground button to each step** 585 + 586 + In the generated HTML, each step gets a button: 587 + 588 + ```html 589 + <button class="sc-playground-btn" data-step="N"> 590 + ▶ Try it 591 + </button> 592 + ``` 593 + 594 + **Step 2: Add the overlay HTML (generated once)** 595 + 596 + ```html 597 + <div id="sc-playground-overlay" class="sc-playground-overlay" style="display:none"> 598 + <div class="sc-playground-header"> 599 + <span class="sc-playground-title">Playground</span> 600 + <button class="sc-playground-close">✕</button> 601 + </div> 602 + <div class="sc-playground-editor"> 603 + <x-ocaml id="sc-playground-editor" backend="jtw"></x-ocaml> 604 + </div> 605 + <div class="sc-playground-output" id="sc-playground-output"></div> 606 + </div> 607 + ``` 608 + 609 + **Step 3: Add JavaScript to open/close and pre-load code** 610 + 611 + ```javascript 612 + document.querySelectorAll('.sc-playground-btn').forEach(btn => { 613 + btn.addEventListener('click', () => { 614 + const stepIndex = parseInt(btn.dataset.step); 615 + // Concatenate all code up to and including this step 616 + const code = steps.slice(0, stepIndex + 1).map(s => s.code).join('\n\n'); 617 + const overlay = document.getElementById('sc-playground-overlay'); 618 + overlay.style.display = 'flex'; 619 + const editor = document.getElementById('sc-playground-editor'); 620 + // Set editor content to accumulated code 621 + editor.textContent = code; 622 + // Trigger execution 623 + }); 624 + }); 625 + ``` 626 + 627 + **Step 4: Add CSS for the overlay** 628 + 629 + Full-screen overlay with semi-transparent backdrop, centered editor, output panel below. 630 + 631 + **Step 5: Build and test** 632 + 633 + Verify: clicking "Try it" on step 3 opens overlay with code from steps 1-3, editable, with Run capability. 634 + 635 + **Step 6: Commit** 636 + 637 + ```bash 638 + git add -A && git commit -m "feat(scrollycode): add interactive playground overlay" 639 + ``` 640 + 641 + --- 642 + 643 + ## Task Dependencies 644 + 645 + ``` 646 + 1.1 (merge odoc branches) 647 + → 1.2 (extract scrollycode to standalone) 648 + → 2.1 (diff computation) 649 + → 2.2 (mobile layout) 650 + → 1.3 (join x-ocaml) 651 + → 3.1 (jtw_client bridge) 652 + → 3.2 (emit x-ocaml elements) 653 + → 3.3 (type-on-hover) 654 + → 4.1 (playground overlay) 655 + → 1.4 (sync js_top_worker) 656 + → 3.1 (jtw_client bridge) 657 + ``` 658 + 659 + Workstreams 2 and 3 can proceed in parallel after Task 1.2. 660 + 661 + --- 662 + 663 + ## Notes 664 + 665 + - **Worker size**: js_top_worker worker.js is ~64MB uncompressed. Gzipped should be ~8-12MB. Consider showing a loading indicator while the worker initializes. 666 + - **x-ocaml API**: The exact API for programmatically updating x-ocaml content needs verification. May need to use `editor.setSource()` or recreate the element. 667 + - **Cell dependencies**: js_top_worker's cell dependency system can be used to give each step full type context from previous steps. 668 + - **Format support**: js_top_worker doesn't support code formatting — the `Format` request type in x-ocaml will be a no-op for the jtw backend. 669 + - **Testing**: Each workstream should be tested in browser with Playwright at both desktop (1280x800) and mobile (390x844) viewports.
+387
js_top_worker/docs/widget-plan.md
··· 1 + # Widget Support Plan of Record 2 + 3 + *Created: 2026-02-12* 4 + 5 + ## Goals 6 + 7 + Add Jupyter-style interactive widget support to js_top_worker, enabling OCaml 8 + code running in the toplevel to create interactive UI elements (sliders, buttons, 9 + dropdowns, etc.) that communicate bidirectionally with the frontend. 10 + 11 + ## Design Principles 12 + 13 + 1. **Zero new dependencies in the worker** - Every dependency compiled into the 14 + worker can conflict with libraries the user wants to load at runtime. Widget 15 + support must not add any new OCaml dependencies to the worker. 16 + 17 + 2. **Broad OCaml version compatibility** - The project currently targets OCaml 18 + >= 4.04. New features must not raise this floor unnecessarily. 19 + 20 + 3. **Build on the message protocol, not RPC** - The worker already uses the 21 + message-based protocol (`message.ml` / `js_top_worker_client_msg.ml`). Widget 22 + communication extends this protocol rather than the legacy JSON-RPC layer. 23 + 24 + 4. **Remove, don't accumulate** - The legacy rpclib-based communication layer 25 + should be removed as part of this work, reducing the dependency footprint. 26 + 27 + ## Why Not CBOR? 28 + 29 + The architecture document previously listed CBOR as a planned transport format. 30 + After investigation, we've decided against it: 31 + 32 + - **Dependency risk**: Even the lightweight `cbor` opam package brings in 33 + `ocplib-endian`. Any dependency in the worker namespace can conflict with 34 + user-loaded libraries. 35 + - **Unnecessary complexity**: The existing message protocol uses `Js_of_ocaml`'s 36 + native JSON handling (`Json.output` / `Json.unsafe_input`), which has zero 37 + additional dependencies. 38 + - **Binary data via Typed Arrays**: For binary payloads (images, etc.), 39 + `js_of_ocaml`'s `Typed_array` module provides native browser typed array 40 + support without any extra libraries. 41 + - **JSON is the browser's native format** - No encoding/decoding overhead when 42 + passing structured data via `postMessage`. 43 + 44 + ## Communication Architecture 45 + 46 + ### Current State (Two Parallel Layers) 47 + 48 + ``` 49 + 1. Legacy RPC (to be removed): 50 + Client (js_top_worker_client.ml) <-> JSON-RPC <-> Server (Toplevel_api_gen) 51 + Dependencies: rpclib, rpclib-lwt, rpclib.json, ppx_deriving_rpc 52 + 53 + 2. Message protocol (to be extended): 54 + Client (js_top_worker_client_msg.ml) <-> JSON messages <-> Worker (worker.ml) 55 + Dependencies: js_of_ocaml (already required) 56 + ``` 57 + 58 + ### Target State 59 + 60 + ``` 61 + Client (js_top_worker_client_msg.ml) <-> JSON messages <-> Worker (worker.ml) 62 + | | 63 + |-- Request/Response (existing: eval, complete, errors, ...) | 64 + |-- Push messages (existing: output_at streaming) | 65 + |-- Widget messages (NEW: comm_open, comm_update, comm_msg, ...) | 66 + ``` 67 + 68 + All communication uses the existing `message.ml` protocol extended with widget 69 + message types. No new serialization libraries. 70 + 71 + ## Widget Protocol Design 72 + 73 + ### Message Types (Worker -> Client) 74 + 75 + ``` 76 + CommOpen { comm_id; target; state } -- Widget created by OCaml code 77 + CommUpdate { comm_id; state } -- Widget state changed 78 + CommClose { comm_id } -- Widget destroyed 79 + ``` 80 + 81 + ### Message Types (Client -> Worker) 82 + 83 + ``` 84 + CommMsg { comm_id; data } -- Frontend event (click, value change) 85 + CommClose { comm_id } -- Frontend closed widget 86 + ``` 87 + 88 + ### Widget State Format 89 + 90 + Widget state is a JSON object with well-known keys, following the Jupyter widget 91 + convention where practical: 92 + 93 + ```json 94 + { 95 + "widget_type": "slider", 96 + "value": 50, 97 + "min": 0, 98 + "max": 100, 99 + "step": 1, 100 + "description": "Threshold", 101 + "disabled": false 102 + } 103 + ``` 104 + 105 + The `widget_type` field replaces Jupyter's `_model_module` / `_model_name` / 106 + `_view_module` / `_view_name` quartet, since we don't need the npm module 107 + indirection - our widget renderers are built into the client. 108 + 109 + ### Alignment with Jupyter Protocol 110 + 111 + We adopt the **concepts** from the Jupyter widget protocol but simplify the 112 + implementation: 113 + 114 + | Jupyter Concept | Our Equivalent | 115 + |-----------------|----------------| 116 + | comm_open | CommOpen message | 117 + | comm_msg method:"update" | CommUpdate message | 118 + | comm_msg method:"custom" | CommMsg message | 119 + | comm_close | CommClose message | 120 + | _model_module + _model_name | widget_type string | 121 + | buffer_paths (binary) | Typed_array via js_of_ocaml | 122 + | Display message | CommOpen includes display flag | 123 + 124 + We do **not** implement: 125 + - echo_update (single frontend, no multi-client sync needed) 126 + - request_state / request_states (state is authoritative in worker) 127 + - Version negotiation (internal protocol, not cross-system) 128 + 129 + ## OCaml Widget API 130 + 131 + User-facing API available as an OCaml library in the toplevel: 132 + 133 + ```ocaml 134 + module Widget : sig 135 + type t 136 + 137 + (** Create a widget. Returns it and displays it. *) 138 + val slider : ?min:int -> ?max:int -> ?step:int -> 139 + ?description:string -> int -> t 140 + 141 + val button : ?style:string -> string -> t 142 + 143 + val text : ?placeholder:string -> ?description:string -> string -> t 144 + 145 + val dropdown : ?description:string -> options:string list -> string -> t 146 + 147 + val checkbox : ?description:string -> bool -> t 148 + 149 + val html : string -> t 150 + 151 + (** Read current value *) 152 + val get : t -> Yojson.Safe.t (* or a simpler JSON type *) 153 + 154 + (** Update widget state *) 155 + val set : t -> string -> Yojson.Safe.t -> unit 156 + 157 + (** Register event handler *) 158 + val on_change : t -> (Yojson.Safe.t -> unit) -> unit 159 + val on_click : t -> (unit -> unit) -> unit 160 + 161 + (** Display / close *) 162 + val display : t -> unit 163 + val close : t -> unit 164 + end 165 + ``` 166 + 167 + **Important**: This API library (`widget` or similar) runs *inside* the toplevel 168 + and must have minimal dependencies. It communicates with the frontend by pushing 169 + messages through the same channel as `Mime_printer`. 170 + 171 + ## Code Removal Plan 172 + 173 + ### Files to Remove 174 + 175 + | File | Reason | 176 + |------|--------| 177 + | `idl/transport.ml`, `transport.mli` | JSON-RPC transport wrapper | 178 + | `idl/js_top_worker_client.ml`, `.mli` | RPC-based Lwt client | 179 + | `idl/js_top_worker_client_fut.ml` | RPC-based Fut client | 180 + | `idl/_old/` directory | Historical RPC reference code | 181 + 182 + ### Dependencies to Remove 183 + 184 + | Package | Used By | 185 + |---------|---------| 186 + | `rpclib` | transport.ml, RPC clients | 187 + | `rpclib-lwt` | impl.ml (IdlM module) | 188 + | `rpclib.json` | transport.ml, RPC clients | 189 + | `ppx_deriving_rpc` | toplevel_api.ml code generation | 190 + | `xmlm` | transitive via rpclib | 191 + | `cmdliner` | transitive via rpclib | 192 + 193 + ### Files to Refactor 194 + 195 + | File | Change | 196 + |------|--------| 197 + | `lib/impl.ml` | Remove `IdlM` / `Rpc_lwt` usage, use own types | 198 + | `idl/toplevel_api.ml` | Keep type definitions, remove RPC IDL machinery | 199 + | `idl/toplevel_api_gen.ml` | Replace with hand-written types (no ppx_deriving_rpc) | 200 + | `idl/dune` | Remove rpclib library deps and ppx rules | 201 + | `lib/dune` | Remove rpclib-lwt dep | 202 + | `test/node/*.ml` | Migrate from RPC Server/Client to message protocol | 203 + | `test/browser/client_test.ml` | Use message-based client | 204 + | `example/unix_worker.ml` | Use message protocol over Unix socket | 205 + | `example/unix_client.ml` | Use message protocol over Unix socket | 206 + 207 + ### Impact on `toplevel_api_gen.ml` 208 + 209 + The generated file is 92k+ lines (from ppx_deriving_rpc). The types it defines 210 + are used extensively in `impl.ml` and `worker.ml`. The plan: 211 + 212 + 1. Extract the **type definitions** into a new lightweight module (no ppx) 213 + 2. Hand-write any needed serialization for the message protocol 214 + 3. Remove `ppx_deriving_rpc` dependency entirely 215 + 4. Delete `toplevel_api_gen.ml` 216 + 217 + ## Testing Strategy 218 + 219 + ### Existing Test Infrastructure 220 + 221 + | Backend | Location | Framework | 222 + |---------|----------|-----------| 223 + | Unit tests | `test/libtest/` | ppx_expect | 224 + | Node.js | `test/node/` | js_of_ocaml + Node | 225 + | Unix (cram) | `test/cram/` | Cram tests with unix_worker | 226 + | Browser | `test/browser/` | Playwright + Chromium | 227 + 228 + ### Widget Testing Approach 229 + 230 + #### 1. Unit Tests (`test/libtest/`) 231 + 232 + - Widget state management logic 233 + - Message serialization/deserialization for widget messages 234 + - CommManager state tracking (open/update/close lifecycle) 235 + 236 + #### 2. Node.js Tests (`test/node/`) 237 + 238 + - Widget creation produces correct CommOpen messages 239 + - Widget state updates produce CommUpdate messages 240 + - Event handler registration and dispatch 241 + - Widget close cleanup 242 + - Multiple simultaneous widgets 243 + - Widget interaction with regular exec output (mime_vals + widgets) 244 + 245 + #### 3. Cram Tests (`test/cram/`) 246 + 247 + - Unix worker handles widget messages over socket 248 + - Widget lifecycle via command-line client 249 + - Widget messages interleaved with regular eval output 250 + 251 + #### 4. Browser Tests (`test/browser/`) 252 + 253 + - **End-to-end widget rendering**: OCaml creates widget -> message sent -> 254 + client renders DOM element -> user interaction -> event sent back -> OCaml 255 + handler fires 256 + - **Widget types**: Test each widget type (slider, button, text, dropdown, 257 + checkbox, html) 258 + - **State synchronization**: Frontend changes propagated to worker and back 259 + - **Multiple widgets**: Several widgets active simultaneously 260 + - **Widget cleanup**: Closing widgets removes DOM elements 261 + - **Integration with existing features**: Widgets alongside code completion, 262 + error reporting, MIME output 263 + 264 + ### Test Utilities 265 + 266 + A shared test helper module for widget testing: 267 + 268 + ```ocaml 269 + (* test/test_widget_helpers.ml *) 270 + val assert_comm_open : worker_msg -> comm_id:string -> widget_type:string -> unit 271 + val assert_comm_update : worker_msg -> comm_id:string -> key:string -> unit 272 + val assert_comm_close : worker_msg -> comm_id:string -> unit 273 + val simulate_event : comm_id:string -> data:string -> client_msg 274 + ``` 275 + 276 + ## Example Widgets 277 + 278 + ### Priority 1: Core Widgets (Implement First) 279 + 280 + These are the most commonly used Jupyter widgets and cover the fundamental 281 + interaction patterns: 282 + 283 + | Widget | State | Events | Jupyter Equivalent | 284 + |--------|-------|--------|--------------------| 285 + | IntSlider | value, min, max, step | on_change(int) | IntSlider | 286 + | Button | description, style | on_click | Button | 287 + | Text | value, placeholder | on_change(string) | Text | 288 + | Dropdown | value, options | on_change(string) | Dropdown | 289 + | Checkbox | value, description | on_change(bool) | Checkbox | 290 + | HTML | value (html string) | none | HTML | 291 + 292 + ### Priority 2: Composition Widgets 293 + 294 + | Widget | Purpose | Jupyter Equivalent | 295 + |--------|---------|-------------------| 296 + | HBox / VBox | Layout containers | HBox / VBox | 297 + | Output | Capture stdout/display | Output | 298 + | FloatSlider | Decimal slider | FloatSlider | 299 + 300 + ### Priority 3: Domain-Specific Widgets 301 + 302 + | Widget | Purpose | Inspired By | 303 + |--------|---------|-------------| 304 + | Plot | Simple 2D charts (SVG) | bqplot (simplified) | 305 + | Table | Data grid display | ipydatagrid (read-only) | 306 + | Image | Display image bytes | Image widget | 307 + 308 + ### Example: Interactive Slider 309 + 310 + ```ocaml 311 + (* User code in toplevel *) 312 + let threshold = Widget.slider ~min:0 ~max:100 ~description:"Threshold" 50;; 313 + 314 + Widget.on_change threshold (fun v -> 315 + let n = Widget.Int.of_json v in 316 + Printf.printf "Threshold changed to: %d\n" n 317 + );; 318 + ``` 319 + 320 + ### Example: Button with Output 321 + 322 + ```ocaml 323 + let count = ref 0;; 324 + let label = Widget.html (Printf.sprintf "<b>Count: %d</b>" !count);; 325 + let btn = Widget.button "Increment";; 326 + 327 + Widget.on_click btn (fun () -> 328 + incr count; 329 + Widget.set label "value" 330 + (`String (Printf.sprintf "<b>Count: %d</b>" !count)) 331 + );; 332 + ``` 333 + 334 + ### Example: Linked Widgets 335 + 336 + ```ocaml 337 + let slider = Widget.slider ~min:0 ~max:255 ~description:"Red" 128;; 338 + let preview = Widget.html {|<div style="width:50px;height:50px"></div>|};; 339 + 340 + Widget.on_change slider (fun v -> 341 + let r = Widget.Int.of_json v in 342 + Widget.set preview "value" 343 + (`String (Printf.sprintf 344 + {|<div style="width:50px;height:50px;background:rgb(%d,0,0)"></div>|} r)) 345 + );; 346 + ``` 347 + 348 + ## Implementation Phases 349 + 350 + ### Phase 1: RPC Removal & Type Cleanup 351 + 352 + Remove the legacy RPC layer and establish clean type definitions. 353 + 354 + ### Phase 2: Widget Message Protocol 355 + 356 + Extend `message.ml` and `worker.ml` with widget message types. 357 + 358 + ### Phase 3: Widget Manager (Worker Side) 359 + 360 + Implement the comm manager that tracks widget state and routes events. 361 + 362 + ### Phase 4: OCaml Widget API 363 + 364 + Create the user-facing `Widget` module available in the toplevel. 365 + 366 + ### Phase 5: JavaScript Widget Renderer 367 + 368 + Implement widget rendering in the JavaScript client. 369 + 370 + ### Phase 6: Testing & Examples 371 + 372 + Full test coverage across all backends, example widgets, documentation. 373 + 374 + ## Open Questions 375 + 376 + 1. **JSON representation in OCaml API**: Use `Yojson.Safe.t`? A custom minimal 377 + JSON type? Raw `Js.Unsafe.any`? (Yojson adds a dependency; custom type is 378 + more work; Unsafe.any is untyped.) 379 + 380 + 2. **Widget library loading**: Should the Widget module be preloaded in the 381 + worker, or loaded on demand via `#require "widget"`? 382 + 383 + 3. **Layout model**: How much of Jupyter's CSS-based layout model to support? 384 + Full flexbox control per-widget, or simpler HBox/VBox only? 385 + 386 + 4. **Persistence**: Should widget state survive cell re-execution? Jupyter 387 + widgets are destroyed and recreated; we could do the same or preserve state.
+15 -35
root.opam
··· 2 2 opam-version: "2.0" 3 3 synopsis: "Monorepo root package with external dependencies" 4 4 depends: [ 5 - "ISO3166" 6 5 "alcotest" 6 + "angstrom" 7 7 "astring" 8 8 "base64" 9 9 "bigstringaf" ··· 11 11 "bos" 12 12 "brr" 13 13 "bytesrw" 14 - "ca-certs" 15 14 "camlp-streams" 15 + "cbort" 16 16 "cmarkit" 17 17 "cmdliner" 18 18 "conf-jq" 19 19 "cppo" 20 - "crowbar" 21 20 "crunch" 22 - "cstruct" 23 21 "decompress" 24 - "digestif" 25 - "domain-name" 26 22 "dune" {>= "3.20"} 27 23 "dune-site" 28 24 "eio" 29 25 "eio_main" 30 - "eqaf" 31 26 "fmt" 32 27 "fpath" 33 - "geojson" 34 - "ipaddr" 35 28 "js_of_ocaml" 36 29 "js_of_ocaml-compiler" 30 + "js_of_ocaml-lwt" 37 31 "js_of_ocaml-ppx" 38 - "jsonm" 39 - "jsont" 40 - "kdf" 32 + "js_of_ocaml-toplevel" 41 33 "logs" 42 34 "lwt" 43 - "magic-mime" 44 - "mdx" 45 35 "menhir" 46 - "mirage-crypto" 47 - "mirage-crypto-ec" 48 - "mirage-crypto-pk" 49 - "mirage-crypto-rng" 50 - "multibase" 36 + "merlin-lib" 37 + "mime_printer" 51 38 "ocaml" 52 39 "ocamlfind" 53 40 "odig" 54 - "opam-file-format" 55 41 "opam-format" 56 - "openapi" 57 - "optint" 58 - "parsexp" 59 42 "ppx_blob" 43 + "ppx_deriving" 44 + "ppx_deriving_rpc" 60 45 "ppx_expect" 61 46 "ppx_sexp_conv" 47 + "ppxlib" 62 48 "progress" 63 - "ptime" 64 - "re" 65 49 "result" 50 + "rpclib" 51 + "rpclib-lwt" 52 + "rresult" 66 53 "sexplib" 67 54 "sexplib0" 68 - "spdx_licenses" 69 - "tls" 70 - "tls-eio" 71 55 "tyxml" 72 56 "uri" 73 - "uunf" 74 - "uutf" 75 - "uuuu" 76 - "wasm_of_ocaml-compiler" 77 - "x509" 78 - "xdg" 79 - "xmlm" 80 57 "yojson" 81 58 "zarith" 82 59 "odoc" {with-doc} ··· 96 73 ] 97 74 ] 98 75 x-maintenance-intent: ["(latest)"] 76 + pin-depends: [ 77 + ["dune.3.21.0" "git+https://github.com/jonludlam/dune.git#odoc-v3-rules-3.21"] 78 + ]
+3
root.opam.template
··· 1 + pin-depends: [ 2 + ["dune.3.21.0" "git+https://github.com/jonludlam/dune.git#odoc-v3-rules-3.21"] 3 + ]