jon.recoil.org
forked from
anil.recoil.org/monopam-myspace
My aggregated monorepo of OCaml code, automaintained
1# js_top_worker Admin Guide
2
3This guide covers how to generate and host JTW (js_top_worker) artifacts --
4the compiled OCaml libraries and toplevel worker that power in-browser REPLs.
5
6There are two tools:
7
8- **`jtw opam`** -- standalone tool for generating artifacts from an opam
9 switch. Good for self-hosting a fixed set of packages.
10- **`day10 batch --with-jtw`** -- the universe builder pipeline. Solves
11 dependencies, builds packages in containers, generates documentation and
12 JTW artifacts at scale. Used for ocaml.org.
13
14## jtw opam: standalone artifact generation
15
16### Prerequisites
17
18An opam switch with the desired packages installed:
19
20```bash
21opam switch create myswitch ocaml-base-compiler.5.4.0
22opam install fmt cmdliner str
23```
24
25The `jtw` binary, built from the js_top_worker repo:
26
27```bash
28git clone https://tangled.org/jon.recoil.org/js_top_worker
29cd js_top_worker
30opam install . --deps-only
31dune build
32```
33
34### Generating artifacts for a set of packages
35
36```bash
37dune exec -- jtw opam -o output fmt cmdliner str
38```
39
40This produces:
41
42```
43output/
44 worker.js
45 findlib_index.json
46 lib/
47 fmt/
48 META, *.cmi, fmt.cma.js, dynamic_cmis.json
49 cmdliner/
50 META, *.cmi, cmdliner.cma.js, dynamic_cmis.json
51 str/
52 META, *.cmi, str.cma.js, dynamic_cmis.json
53 ocaml/
54 META, *.cmi, stdlib.cma.js, dynamic_cmis.json
55```
56
57The tool:
581. Resolves transitive dependencies via `ocamlfind`
592. Copies `.cmi` files for each library (used by the type checker)
603. Compiles `.cma` archives to `.cma.js` via `js_of_ocaml`
614. Generates `dynamic_cmis.json` metadata per library directory
625. Writes `findlib_index.json` listing all META file paths
636. Compiles `worker.js` (the OCaml toplevel as a web worker)
64
65### Flags
66
67| Flag | Default | Description |
68|------|---------|-------------|
69| `-o DIR` | `html` | Output directory |
70| `-v` | off | Verbose logging |
71| `--switch SWITCH` | current | Opam switch to use |
72| `--no-worker` | off | Skip worker.js generation |
73| `--path PATH` | none | Write output under a subdirectory (for per-package builds) |
74| `--deps-file FILE` | none | File listing dependency paths (one per line) |
75
76### Generating per-package universes
77
78To generate separate artifact directories per package (each with its own
79dependency closure):
80
81```bash
82dune exec -- jtw opam-all -o output --all
83```
84
85This produces a directory per installed findlib package, each containing its
86own `findlib_index.json` and `lib/` tree, plus a root-level
87`findlib_index.json` covering everything.
88
89The `--all` flag builds every package returned by `ocamlfind list`. Without
90it, pass specific package names as positional arguments.
91
92### Serving the output
93
94Serve the output directory over HTTP. Any static file server works:
95
96```bash
97cd output
98python3 -m http.server 8080
99```
100
101If loading from a different origin, configure CORS headers on the server.
102
103The `findlib_index.json` URL is the single entry point clients need. See the
104User's Guide for how to connect to it from JavaScript.
105
106## day10: universe builder pipeline
107
108day10 is the batch pipeline that builds, tests, documents, and generates JTW
109artifacts for opam packages at scale. It runs builds inside OCI containers
110using `runc` with overlay filesystems.
111
112### Prerequisites
113
114- Linux (uses `runc`, overlay mounts, user namespaces)
115- An opam-repository checkout
116- Root access (for container operations)
117- The js_top_worker repo accessible via HTTPS (for container builds)
118
119### Building day10
120
121```bash
122cd monopam # or wherever the monorepo lives
123dune build day10/
124```
125
126### Running a batch with JTW
127
128```bash
129dune exec -- day10 batch \
130 --cache-dir /var/cache/day10 \
131 --opam-repository /var/cache/opam-repository \
132 --ocaml-version ocaml-base-compiler.5.4.0 \
133 --with-jtw \
134 --jtw-output /var/www/jtw \
135 --html-output /var/www/docs \
136 --with-doc \
137 @packages.json
138```
139
140Where `packages.json` is:
141
142```json
143{"packages": ["fmt.0.9.0", "cmdliner.1.3.0", "lwt.5.9.0"]}
144```
145
146### JTW-specific flags
147
148| Flag | Default | Description |
149|------|---------|-------------|
150| `--with-jtw` | false | Enable JTW artifact generation |
151| `--jtw-output DIR` | none | Output directory for assembled JTW artifacts |
152| `--jtw-tools-repo URL` | `https://tangled.org/jon.recoil.org/js_top_worker` | Git repo for js_top_worker |
153| `--jtw-tools-branch BRANCH` | `main` | Git branch for js_top_worker |
154
155### How it works
156
157The JTW pipeline has three phases:
158
159#### Phase 1: jtw-tools layer
160
161A one-time (per OCaml version + repo + branch) container build that installs
162the JTW toolchain:
163
1641. Installs `ocaml-base-compiler.<version>` in a fresh container
1652. Pins all js_top_worker packages from the configured git repo/branch
1663. Installs `js_of_ocaml`, `js_top_worker-bin`, `js_top_worker-web`
1674. Runs `jtw opam -o /home/opam/jtw-tools-output stdlib` to produce
168 `worker.js` and stdlib artifacts
169
170The result is cached at `<cache>/<os-key>/jtw-tools-<hash>/`. The hash
171depends on the OCaml version, repo URL, and branch name. Changing any of
172these invalidates the cache.
173
174#### Phase 2: per-package JTW generation
175
176For each package in the solution, a container runs:
177
178```bash
179jtw opam --path <pkg-name> --no-worker -o /home/opam/jtw-output <findlib-names>
180```
181
182This produces `.cmi`, `.cma.js`, `META`, and `dynamic_cmis.json` for the
183package's findlib libraries. The container has the package's build layer and
184all dependency build layers mounted, so `ocamlfind` can resolve everything.
185
186Results are cached per package in `<cache>/<os-key>/jtw-<hash>/lib/`.
187
188Packages with no findlib META files (e.g. `ocaml-base-compiler`,
189`base-threads`) produce no JTW artifacts and are marked as `"status":"success"`
190with an empty layer.
191
192#### Phase 3: assembly
193
194`assemble_jtw_output` combines per-package layers into the final
195content-hashed directory structure:
196
197```
198<jtw-output>/
199 compiler/<version>/<compiler-hash>/
200 worker.js
201 lib/ocaml/*.cmi, stdlib.cma.js, dynamic_cmis.json
202
203 p/<package>/<version>/<content-hash>/
204 lib/<findlib-name>/
205 META, *.cmi, *.cma.js, dynamic_cmis.json
206
207 u/<universe-hash>/
208 findlib_index.json
209```
210
211Content hashes are computed from the payload files (`.cmi`, `.cma.js`, `META`).
212Identical content always produces the same hash, enabling deduplication across
213universe builds.
214
215The `dynamic_cmis.json` files have their `dcs_url` field rewritten to use
216relative paths from the compiler directory (where `worker.js` loads them).
217
218### Caching and layer structure
219
220day10 uses a layered caching system. JTW-related layers:
221
222| Layer | Path pattern | Contents |
223|-------|-------------|----------|
224| jtw-tools | `jtw-tools-<hash>/` | js_of_ocaml + jtw binaries, worker.js, stdlib |
225| per-package | `jtw-<hash>/` | Package's `.cmi`, `.cma.js`, `META`, `dynamic_cmis.json` |
226
227Each layer has a `layer.json` with metadata:
228
229```json
230{
231 "package": "fmt.0.9.0",
232 "build_hash": "build-abc123",
233 "jtw": {"status": "success"}
234}
235```
236
237Possible status values: `"success"`, `"failure"` (with `"error"` field),
238`"skipped"` (no findlib packages or jtw-tools unavailable).
239
240### Inspecting layers
241
242```bash
243# List all jtw layers
244ls /var/cache/day10/ubuntu-25.04-x86_64/jtw-*/
245
246# Check a layer's status
247cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-abc123/layer.json
248
249# View the build log
250cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-abc123/jtw.log
251
252# Check jtw-tools layer
253cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-tools-*/layer.json
254cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-tools-*/build.log
255```
256
257### Invalidating caches
258
259To force a rebuild of the jtw-tools layer (e.g. after updating
260js_top_worker):
261
262```bash
263sudo rm -rf /var/cache/day10/ubuntu-25.04-x86_64/jtw-tools-*/
264```
265
266To force a rebuild of per-package JTW artifacts:
267
268```bash
269sudo rm -rf /var/cache/day10/ubuntu-25.04-x86_64/jtw-[0-9a-f]*/
270```
271
272Root is required because the container filesystem layers are owned by the
273container's uid.
274
275### Using a custom js_top_worker
276
277To test changes to js_top_worker before merging:
278
2791. Push your branch to a git-accessible URL
2802. Pass it to day10:
281
282```bash
283dune exec -- day10 batch \
284 --with-jtw \
285 --jtw-tools-repo https://tangled.org/jon.recoil.org/js_top_worker \
286 --jtw-tools-branch my-feature-branch \
287 ...
288```
289
290The jtw-tools layer hash will change (it includes the branch name), so a
291fresh toolchain build will occur.
292
293### Troubleshooting
294
295**jtw-tools layer fails with "Unknown archive type"**
296
297The repo URL needs to be accessible as a git repository. Opam uses the
298`git+https://` scheme internally. If your URL serves HTML instead of git,
299the pin will fail. Verify with:
300
301```bash
302git ls-remote https://your-repo-url
303```
304
305**All jtw layers show status "skipped"**
306
307This means `has_jsoo` returned false -- the jtw-tools layer doesn't contain
308`js_of_ocaml`. Check the jtw-tools build log for installation failures.
309
310**jtw layer shows status "failure" with exit code 125**
311
312This usually means the package has no installable findlib libraries, or
313`jtw opam` couldn't find any `.cma` files to compile. This is normal for
314packages like `ocaml-compiler` that don't install findlib packages.
315
316**Per-package artifacts exist in cache but jtw-output is empty**
317
318Assembly only runs in `batch` mode, not `health-check`. Use:
319
320```bash
321dune exec -- day10 batch --with-jtw --jtw-output /path/to/output ...
322```
323
324**Content hashes change unexpectedly**
325
326The content hash is computed from `.cmi`, `.cma.js`, and `META` file contents.
327If the OCaml compiler version or any dependency changes, the compiled artifacts
328will differ and produce a new hash. This is by design.