jon.recoil.org
forked from
anil.recoil.org/monopam-myspace
My aggregated monorepo of OCaml code, automaintained
1# Interactive OCaml Demos — End-to-End Setup
2
3This document explains how to build and serve the interactive OCaml demo
4pages that use the `odoc-interactive-extension` and `js_top_worker` (jtw)
5to create executable code cells inside odoc documentation.
6
7## Overview
8
9The system has three main components:
10
111. **odoc-interactive-extension** — an odoc plugin that transforms
12 `{@ocaml[...]}` code blocks into `<x-ocaml>` custom elements and
13 processes `@x-ocaml.universe` tags.
14
152. **x-ocaml** — a WebComponent (`<x-ocaml>`) that creates CodeMirror
16 editors, connects to a Web Worker for OCaml evaluation, and renders
17 output inline.
18
193. **jtw** (js_top_worker) — builds "universe" directories containing
20 compiled OCaml libraries (`.cma.js` files, `.cmi` stubs, findlib
21 metadata) that the worker loads at runtime.
22
23### Demo pages
24
25| Page | Description | Universe |
26|------|-------------|----------|
27| `demo1.html` | Basic expressions + Yojson | `./universe` (yojson from default switch) |
28| `demo2_v2.html` | Yojson 2.x API | `./universe-v2` (yojson from demo-yojson-v2 switch) |
29| `demo2_v3.html` | Yojson 3.x API | `./universe-v3` (yojson from default switch) |
30
31The v2 and v3 demos show that different library versions can coexist as
32separate universes, each page loading its own set of compiled libraries.
33
34## Prerequisites
35
36- OCaml 5.4+ with opam
37- The monorepo checked out at `/home/jons-agent/workspace/mono` (or adjust paths)
38- Two opam switches:
39 - `default` — has yojson 3.x (or latest)
40 - `demo-yojson-v2` — has yojson 2.2.2
41- Python 3 (for the dev server, or any static file server with CORS)
42
43### Creating the yojson v2 switch (one-time)
44
45```bash
46opam switch create demo-yojson-v2 ocaml-base-compiler.5.4.1
47eval $(opam env --switch demo-yojson-v2 --set-switch)
48opam install yojson.2.2.2
49```
50
51## Build Steps
52
53All commands assume you start from the monorepo root.
54
55### Step 1: Build and install the toolchain
56
57The odoc extension must be installed before generating docs, because odoc
58loads extensions as plugins at doc-generation time.
59
60```bash
61eval $(opam env --switch default --set-switch)
62dune build @install
63```
64
65This builds and installs (locally) `odoc`, `jtw`, `x-ocaml`, and the
66`odoc-interactive-extension` plugin.
67
68### Step 2: Generate odoc HTML
69
70```bash
71dune build @doc
72```
73
74This generates HTML for all packages. The demo `.mld` files
75(`odoc-interactive-extension/doc/demo1.mld`, etc.) are compiled as part
76of the `odoc-interactive-extension` package documentation.
77
78The HTML output lands in:
79```
80_build/default/_doc/_html/odoc-interactive-extension/
81├── demo1.html
82├── demo2_v2.html
83├── demo2_v3.html
84└── index.html
85```
86
87### Step 3: Copy x-ocaml assets
88
89The extension loads `x-ocaml.js` and `worker.js` from a local
90`_x-ocaml/` directory relative to the HTML pages. Copy them from the
91dune build output:
92
93```bash
94HTMLDIR=_build/default/_doc/_html/odoc-interactive-extension
95mkdir -p "$HTMLDIR/_x-ocaml"
96cp _build/default/x-ocaml/src/x_ocaml.bc.js "$HTMLDIR/_x-ocaml/x-ocaml.js"
97cp _build/default/js_top_worker/lib/.worker.eobjs/jsoo/worker.bc.js "$HTMLDIR/_x-ocaml/worker.js"
98```
99
100### Step 4: Build universe directories
101
102Each demo page declares a universe via `@x-ocaml.universe ./universe-name`
103in its `.mld` source. We use `jtw opam` to generate each universe
104directory.
105
106The `jtw` binary is at `_build/install/default/bin/jtw`.
107
108```bash
109JTW=_build/install/default/bin/jtw
110
111# Universe for demo1 (default switch, yojson latest)
112eval $(opam env --switch default --set-switch)
113$JTW opam --no-worker --output "$HTMLDIR/universe" yojson
114
115# Universe for demo2_v3 (default switch, yojson latest = 3.x)
116$JTW opam --no-worker --output "$HTMLDIR/universe-v3" yojson
117
118# Universe for demo2_v2 (demo-yojson-v2 switch, yojson 2.2.2)
119eval $(opam env --switch demo-yojson-v2 --set-switch)
120$JTW opam --no-worker --switch demo-yojson-v2 --output "$HTMLDIR/universe-v2" yojson
121
122# Switch back to default
123eval $(opam env --switch default --set-switch)
124```
125
126Key flags:
127- `--no-worker` — skip building a worker.js (we provide our own via `_x-ocaml/`)
128- `--output DIR` — where to write the universe files
129- `--switch NAME` — which opam switch to read packages from
130- `yojson` — positional argument: the library to include (plus its deps)
131
132Each universe directory will contain:
133```
134universe/
135├── findlib_index.json # lists META file locations
136└── lib/
137 ├── ocaml/
138 │ ├── META
139 │ ├── dynamic_cmis.json # lists .cmi files to load on demand
140 │ └── *.cmi # stdlib type interfaces
141 ├── yojson/
142 │ ├── META
143 │ ├── *.cmi
144 │ └── yojson.cma.js # compiled library code
145 └── seq/ # (dependency, if needed)
146 └── ...
147```
148
149### Step 5: Serve and test
150
151Start a local HTTP server from the HTML root:
152
153```bash
154python3 docs/demos/cors_server.py 8080 _build/default/_doc/_html
155```
156
157Then open in a browser:
158- http://localhost:8080/odoc-interactive-extension/demo1.html
159- http://localhost:8080/odoc-interactive-extension/demo2_v2.html
160- http://localhost:8080/odoc-interactive-extension/demo2_v3.html
161
162Each page should:
1631. Auto-evaluate all code cells on load
1642. Show output inline below each expression (e.g., `- : int = 7`)
1653. Display stdout output (e.g., `Hello, World!`)
166
167## Writing Demo Pages
168
169Demo pages are standard odoc `.mld` files with two additions:
170
171### Universe tag
172
173Declares the universe directory (relative to the HTML page):
174
175```
176@x-ocaml.universe ./universe
177```
178
179This makes the extension use the `jtw` backend (js_top_worker) instead of
180the builtin merlin-js backend. The worker loads libraries from the
181specified universe directory.
182
183### Code cells
184
185Standard odoc `{@ocaml[...]}` blocks become interactive cells:
186
187```
188{@ocaml[
1891 + 2 * 3
190]}
191```
192
193For cells that use `#require` or reference libraries:
194
195```
196{@ocaml[
197#require "yojson"
198]}
199
200{@ocaml[
201let json = `Assoc [("key", `String "value")]
202let () = print_endline (Yojson.Safe.to_string json)
203]}
204```
205
206Note: The `#require` directive must be in a separate cell before the code
207that uses the library, because each cell is evaluated independently and
208`#require` loads the library's `.cma.js` file.
209
210### Cell modes (extension syntax)
211
212For the extended `{x@ocaml ... x[...]x}` syntax, additional attributes
213control cell behaviour:
214
215```
216{x@ocaml hidden x[let helper = 42]x} (* invisible setup cell *)
217{x@ocaml exercise id=solve x[...]x} (* editable cell *)
218{x@ocaml test for=solve x[assert (...)]x} (* test cell, linked *)
219{x@ocaml interactive x[read_only_code]x} (* default: visible, read-only *)
220```
221
222## Architecture Notes
223
224### Same-origin constraint
225
226Web Workers cannot be loaded from a different origin. The extension
227therefore always loads `x-ocaml.js` and `worker.js` from the local
228`_x-ocaml/` directory (same origin as the page).
229
230Universe directories can technically be cross-origin for fetching
231`findlib_index.json` and `.cmi` files (these use `fetch()`), but
232`.cma.js` files are loaded via `importScripts()` inside the worker,
233which also requires same-origin in modern browsers. For simplicity,
234keep universe directories same-origin (relative paths in the
235`@x-ocaml.universe` tag).
236
237### How the pieces connect
238
239```
240Page Load
241 │
242 ├─ odoc extension emits:
243 │ <meta name="x-ocaml-universe" content="./universe">
244 │ <script src="./_x-ocaml/x-ocaml.js" src-worker="./_x-ocaml/worker.js" backend="jtw">
245 │ <x-ocaml mode="interactive">code...</x-ocaml>
246 │
247 ├─ x-ocaml.js (WebComponent):
248 │ 1. Reads <meta> tags for universe URL
249 │ 2. Creates Web Worker from worker.js
250 │ 3. Sends init message with findlib_index URL
251 │ 4. For each <x-ocaml> cell:
252 │ - Creates CodeMirror editor
253 │ - Sends eval message to worker
254 │ - Renders streaming output as decorations
255 │
256 └─ worker.js (js_top_worker):
257 1. Receives init with findlib_index URL
258 2. Fetches findlib_index.json → learns library locations
259 3. Fetches dynamic_cmis.json → learns available .cmi files
260 4. On eval: compiles OCaml code using toplevel
261 5. On #require: loads .cma.js via importScripts
262 6. Sends output_at (per-phrase) and output (final) messages
263```
264
265### jtw_client.ml — the bridge
266
267`x-ocaml/src/jtw_client.ml` bridges x-ocaml's `X_protocol` with
268js_top_worker's message protocol. Key design decisions:
269
270- Uses `Jtw.eval_stream` (not `Jtw.eval`) to capture per-phrase
271 `output_at` messages, which contain the actual evaluation results
272 (e.g., `val x : int = 3`). The `Jtw.eval` function only returns the
273 final `output` message, which often has an empty `caml_ppf`.
274
275- Derives `stdlib_dcs` (the URL for `dynamic_cmis.json`) from the
276 `findlib_index` URL, so the worker can locate stdlib CMIs regardless
277 of whether the universe is at a relative or absolute path.
278
279## Troubleshooting
280
281**No output after clicking Run / cells don't auto-evaluate**
282- Check browser console for worker errors
283- Verify `_x-ocaml/worker.js` exists and is accessible
284- Verify the universe's `findlib_index.json` is accessible
285
286**"Toplevel not initialized" error**
287- The worker.js must be built with `js_of_ocaml-toplevel` and the flags
288 `--toplevel +toplevel.js +dynlink.js`
289
290**"Unbound module Yojson" after #require**
291- Check that `#require "yojson"` is in a separate earlier cell
292- Verify the universe directory contains `lib/yojson/yojson.cma.js`
293
294**Cross-origin errors**
295- Keep universe directories same-origin (use relative paths in
296 `@x-ocaml.universe`)
297- The `_x-ocaml/` directory must be on the same origin as the HTML page
298
299**Console shows 404 for .cmi files**
300- Normal if the code doesn't reference those modules; the worker
301 fetches CMIs on demand and some 404s are expected for unused modules