js_top_worker Admin Guide#

This guide covers how to generate and host JTW (js_top_worker) artifacts -- the compiled OCaml libraries and toplevel worker that power in-browser REPLs.

There are two tools:

• jtw opam -- standalone tool for generating artifacts from an opam switch. Good for self-hosting a fixed set of packages.
• day10 batch --with-jtw -- the universe builder pipeline. Solves dependencies, builds packages in containers, generates documentation and JTW artifacts at scale. Used for ocaml.org.

jtw opam: standalone artifact generation#

Prerequisites#

An opam switch with the desired packages installed:

opam switch create myswitch ocaml-base-compiler.5.4.0
opam install fmt cmdliner str

The jtw binary, built from the js_top_worker repo:

git clone https://tangled.org/jon.recoil.org/js_top_worker
cd js_top_worker
opam install . --deps-only
dune build

Generating artifacts for a set of packages#

dune exec -- jtw opam -o output fmt cmdliner str

This produces:

output/
  worker.js
  findlib_index.json
  lib/
    fmt/
      META, *.cmi, fmt.cma.js, dynamic_cmis.json
    cmdliner/
      META, *.cmi, cmdliner.cma.js, dynamic_cmis.json
    str/
      META, *.cmi, str.cma.js, dynamic_cmis.json
    ocaml/
      META, *.cmi, stdlib.cma.js, dynamic_cmis.json

The tool:

1. Resolves transitive dependencies via ocamlfind
2. Copies .cmi files for each library (used by the type checker)
3. Compiles .cma archives to .cma.js via js_of_ocaml
4. Generates dynamic_cmis.json metadata per library directory
5. Writes findlib_index.json listing all META file paths
6. Compiles worker.js (the OCaml toplevel as a web worker)

Flags#

Generating per-package universes#

To generate separate artifact directories per package (each with its own dependency closure):

dune exec -- jtw opam-all -o output --all

This produces a directory per installed findlib package, each containing its own findlib_index.json and lib/ tree, plus a root-level findlib_index.json covering everything.

The --all flag builds every package returned by ocamlfind list. Without it, pass specific package names as positional arguments.

Serving the output#

Serve the output directory over HTTP. Any static file server works:

cd output
python3 -m http.server 8080

If loading from a different origin, configure CORS headers on the server.

The findlib_index.json URL is the single entry point clients need. See the User's Guide for how to connect to it from JavaScript.

day10: universe builder pipeline#

day10 is the batch pipeline that builds, tests, documents, and generates JTW artifacts for opam packages at scale. It runs builds inside OCI containers using runc with overlay filesystems.

Prerequisites#

• Linux (uses runc, overlay mounts, user namespaces)
• An opam-repository checkout
• Root access (for container operations)
• The js_top_worker repo accessible via HTTPS (for container builds)

Building day10#

cd monopam    # or wherever the monorepo lives
dune build day10/

Running a batch with JTW#

dune exec -- day10 batch \
  --cache-dir /var/cache/day10 \
  --opam-repository /var/cache/opam-repository \
  --ocaml-version ocaml-base-compiler.5.4.0 \
  --with-jtw \
  --jtw-output /var/www/jtw \
  --html-output /var/www/docs \
  --with-doc \
  @packages.json

Where packages.json is:

{"packages": ["fmt.0.9.0", "cmdliner.1.3.0", "lwt.5.9.0"]}

JTW-specific flags#

How it works#

The JTW pipeline has three phases:

Phase 1: jtw-tools layer#

A one-time (per OCaml version + repo + branch) container build that installs the JTW toolchain:

1. Installs ocaml-base-compiler.<version> in a fresh container
2. Pins all js_top_worker packages from the configured git repo/branch
3. Installs js_of_ocaml, js_top_worker-bin, js_top_worker-web
4. Runs jtw opam -o /home/opam/jtw-tools-output stdlib to produce worker.js and stdlib artifacts

The result is cached at <cache>/<os-key>/jtw-tools-<hash>/. The hash depends on the OCaml version, repo URL, and branch name. Changing any of these invalidates the cache.

Phase 2: per-package JTW generation#

For each package in the solution, a container runs:

jtw opam --path <pkg-name> --no-worker -o /home/opam/jtw-output <findlib-names>

This produces .cmi, .cma.js, META, and dynamic_cmis.json for the package's findlib libraries. The container has the package's build layer and all dependency build layers mounted, so ocamlfind can resolve everything.

Results are cached per package in <cache>/<os-key>/jtw-<hash>/lib/.

Packages with no findlib META files (e.g. ocaml-base-compiler, base-threads) produce no JTW artifacts and are marked as "status":"success" with an empty layer.

Phase 3: assembly#

assemble_jtw_output combines per-package layers into the final content-hashed directory structure:

<jtw-output>/
  compiler/<version>/<compiler-hash>/
    worker.js
    lib/ocaml/*.cmi, stdlib.cma.js, dynamic_cmis.json

  p/<package>/<version>/<content-hash>/
    lib/<findlib-name>/
      META, *.cmi, *.cma.js, dynamic_cmis.json

  u/<universe-hash>/
    findlib_index.json

Content hashes are computed from the payload files (.cmi, .cma.js, META). Identical content always produces the same hash, enabling deduplication across universe builds.

The dynamic_cmis.json files have their dcs_url field rewritten to use relative paths from the compiler directory (where worker.js loads them).

Caching and layer structure#

day10 uses a layered caching system. JTW-related layers:

Each layer has a layer.json with metadata:

{
  "package": "fmt.0.9.0",
  "build_hash": "build-abc123",
  "jtw": {"status": "success"}
}

Possible status values: "success", "failure" (with "error" field), "skipped" (no findlib packages or jtw-tools unavailable).

Inspecting layers#

# List all jtw layers
ls /var/cache/day10/ubuntu-25.04-x86_64/jtw-*/

# Check a layer's status
cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-abc123/layer.json

# View the build log
cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-abc123/jtw.log

# Check jtw-tools layer
cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-tools-*/layer.json
cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-tools-*/build.log

Invalidating caches#

To force a rebuild of the jtw-tools layer (e.g. after updating js_top_worker):

sudo rm -rf /var/cache/day10/ubuntu-25.04-x86_64/jtw-tools-*/

To force a rebuild of per-package JTW artifacts:

sudo rm -rf /var/cache/day10/ubuntu-25.04-x86_64/jtw-[0-9a-f]*/

Root is required because the container filesystem layers are owned by the container's uid.

Using a custom js_top_worker#

To test changes to js_top_worker before merging:

1. Push your branch to a git-accessible URL
2. Pass it to day10:

dune exec -- day10 batch \
  --with-jtw \
  --jtw-tools-repo https://tangled.org/jon.recoil.org/js_top_worker \
  --jtw-tools-branch my-feature-branch \
  ...

The jtw-tools layer hash will change (it includes the branch name), so a fresh toolchain build will occur.

Troubleshooting#

jtw-tools layer fails with "Unknown archive type"

The repo URL needs to be accessible as a git repository. Opam uses the git+https:// scheme internally. If your URL serves HTML instead of git, the pin will fail. Verify with:

git ls-remote https://your-repo-url

All jtw layers show status "skipped"

This means has_jsoo returned false -- the jtw-tools layer doesn't contain js_of_ocaml. Check the jtw-tools build log for installation failures.

jtw layer shows status "failure" with exit code 125

This usually means the package has no installable findlib libraries, or jtw opam couldn't find any .cma files to compile. This is normal for packages like ocaml-compiler that don't install findlib packages.

Per-package artifacts exist in cache but jtw-output is empty

Assembly only runs in batch mode, not health-check. Use:

dune exec -- day10 batch --with-jtw --jtw-output /path/to/output ...

Content hashes change unexpectedly

The content hash is computed from .cmi, .cma.js, and META file contents. If the OCaml compiler version or any dependency changes, the compiled artifacts will differ and produce a new hash. This is by design.