# js_top_worker Admin Guide

This guide covers how to generate and host JTW (js_top_worker) artifacts --
the compiled OCaml libraries and toplevel worker that power in-browser REPLs.

There are two tools:

- **`jtw opam`** -- standalone tool for generating artifacts from an opam
  switch. Good for self-hosting a fixed set of packages.
- **`day10 batch --with-jtw`** -- the universe builder pipeline. Solves
  dependencies, builds packages in containers, generates documentation and
  JTW artifacts at scale. Used for ocaml.org.

## jtw opam: standalone artifact generation

### Prerequisites

An opam switch with the desired packages installed:

```bash
opam switch create myswitch ocaml-base-compiler.5.4.0
opam install fmt cmdliner str
```

The `jtw` binary, built from the js_top_worker repo:

```bash
git clone https://tangled.org/jon.recoil.org/js_top_worker
cd js_top_worker
opam install . --deps-only
dune build
```

### Generating artifacts for a set of packages

```bash
dune exec -- jtw opam -o output fmt cmdliner str
```

This produces:

```
output/
  worker.js
  findlib_index.json
  lib/
    fmt/
      META, *.cmi, fmt.cma.js, dynamic_cmis.json
    cmdliner/
      META, *.cmi, cmdliner.cma.js, dynamic_cmis.json
    str/
      META, *.cmi, str.cma.js, dynamic_cmis.json
    ocaml/
      META, *.cmi, stdlib.cma.js, dynamic_cmis.json
```

The tool:
1. Resolves transitive dependencies via `ocamlfind`
2. Copies `.cmi` files for each library (used by the type checker)
3. Compiles `.cma` archives to `.cma.js` via `js_of_ocaml`
4. Generates `dynamic_cmis.json` metadata per library directory
5. Writes `findlib_index.json` listing all META file paths
6. Compiles `worker.js` (the OCaml toplevel as a web worker)

### Flags

| Flag | Default | Description |
|------|---------|-------------|
| `-o DIR` | `html` | Output directory |
| `-v` | off | Verbose logging |
| `--switch SWITCH` | current | Opam switch to use |
| `--no-worker` | off | Skip worker.js generation |
| `--path PATH` | none | Write output under a subdirectory (for per-package builds) |
| `--deps-file FILE` | none | File listing dependency paths (one per line) |

### Generating per-package universes

To generate separate artifact directories per package (each with its own
dependency closure):

```bash
dune exec -- jtw opam-all -o output --all
```

This produces a directory per installed findlib package, each containing its
own `findlib_index.json` and `lib/` tree, plus a root-level
`findlib_index.json` covering everything.

The `--all` flag builds every package returned by `ocamlfind list`. Without
it, pass specific package names as positional arguments.

### Serving the output

Serve the output directory over HTTP. Any static file server works:

```bash
cd output
python3 -m http.server 8080
```

If loading from a different origin, configure CORS headers on the server.

The `findlib_index.json` URL is the single entry point clients need. See the
User's Guide for how to connect to it from JavaScript.

## day10: universe builder pipeline

day10 is the batch pipeline that builds, tests, documents, and generates JTW
artifacts for opam packages at scale. It runs builds inside OCI containers
using `runc` with overlay filesystems.

### Prerequisites

- Linux (uses `runc`, overlay mounts, user namespaces)
- An opam-repository checkout
- Root access (for container operations)
- The js_top_worker repo accessible via HTTPS (for container builds)

### Building day10

```bash
cd monopam    # or wherever the monorepo lives
dune build day10/
```

### Running a batch with JTW

```bash
dune exec -- day10 batch \
  --cache-dir /var/cache/day10 \
  --opam-repository /var/cache/opam-repository \
  --ocaml-version ocaml-base-compiler.5.4.0 \
  --with-jtw \
  --jtw-output /var/www/jtw \
  --html-output /var/www/docs \
  --with-doc \
  @packages.json
```

Where `packages.json` is:

```json
{"packages": ["fmt.0.9.0", "cmdliner.1.3.0", "lwt.5.9.0"]}
```

### JTW-specific flags

| Flag | Default | Description |
|------|---------|-------------|
| `--with-jtw` | false | Enable JTW artifact generation |
| `--jtw-output DIR` | none | Output directory for assembled JTW artifacts |
| `--jtw-tools-repo URL` | `https://tangled.org/jon.recoil.org/js_top_worker` | Git repo for js_top_worker |
| `--jtw-tools-branch BRANCH` | `main` | Git branch for js_top_worker |

### How it works

The JTW pipeline has three phases:

#### Phase 1: jtw-tools layer

A one-time (per OCaml version + repo + branch) container build that installs
the JTW toolchain:

1. Installs `ocaml-base-compiler.<version>` in a fresh container
2. Pins all js_top_worker packages from the configured git repo/branch
3. Installs `js_of_ocaml`, `js_top_worker-bin`, `js_top_worker-web`
4. Runs `jtw opam -o /home/opam/jtw-tools-output stdlib` to produce
   `worker.js` and stdlib artifacts

The result is cached at `<cache>/<os-key>/jtw-tools-<hash>/`. The hash
depends on the OCaml version, repo URL, and branch name. Changing any of
these invalidates the cache.

#### Phase 2: per-package JTW generation

For each package in the solution, a container runs:

```bash
jtw opam --path <pkg-name> --no-worker -o /home/opam/jtw-output <findlib-names>
```

This produces `.cmi`, `.cma.js`, `META`, and `dynamic_cmis.json` for the
package's findlib libraries. The container has the package's build layer and
all dependency build layers mounted, so `ocamlfind` can resolve everything.

Results are cached per package in `<cache>/<os-key>/jtw-<hash>/lib/`.

Packages with no findlib META files (e.g. `ocaml-base-compiler`,
`base-threads`) produce no JTW artifacts and are marked as `"status":"success"`
with an empty layer.

#### Phase 3: assembly

`assemble_jtw_output` combines per-package layers into the final
content-hashed directory structure:

```
<jtw-output>/
  compiler/<version>/<compiler-hash>/
    worker.js
    lib/ocaml/*.cmi, stdlib.cma.js, dynamic_cmis.json

  p/<package>/<version>/<content-hash>/
    lib/<findlib-name>/
      META, *.cmi, *.cma.js, dynamic_cmis.json

  u/<universe-hash>/
    findlib_index.json
```

Content hashes are computed from the payload files (`.cmi`, `.cma.js`, `META`).
Identical content always produces the same hash, enabling deduplication across
universe builds.

The `dynamic_cmis.json` files have their `dcs_url` field rewritten to use
relative paths from the compiler directory (where `worker.js` loads them).

### Caching and layer structure

day10 uses a layered caching system. JTW-related layers:

| Layer | Path pattern | Contents |
|-------|-------------|----------|
| jtw-tools | `jtw-tools-<hash>/` | js_of_ocaml + jtw binaries, worker.js, stdlib |
| per-package | `jtw-<hash>/` | Package's `.cmi`, `.cma.js`, `META`, `dynamic_cmis.json` |

Each layer has a `layer.json` with metadata:

```json
{
  "package": "fmt.0.9.0",
  "build_hash": "build-abc123",
  "jtw": {"status": "success"}
}
```

Possible status values: `"success"`, `"failure"` (with `"error"` field),
`"skipped"` (no findlib packages or jtw-tools unavailable).

### Inspecting layers

```bash
# List all jtw layers
ls /var/cache/day10/ubuntu-25.04-x86_64/jtw-*/

# Check a layer's status
cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-abc123/layer.json

# View the build log
cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-abc123/jtw.log

# Check jtw-tools layer
cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-tools-*/layer.json
cat /var/cache/day10/ubuntu-25.04-x86_64/jtw-tools-*/build.log
```

### Invalidating caches

To force a rebuild of the jtw-tools layer (e.g. after updating
js_top_worker):

```bash
sudo rm -rf /var/cache/day10/ubuntu-25.04-x86_64/jtw-tools-*/
```

To force a rebuild of per-package JTW artifacts:

```bash
sudo rm -rf /var/cache/day10/ubuntu-25.04-x86_64/jtw-[0-9a-f]*/
```

Root is required because the container filesystem layers are owned by the
container's uid.

### Using a custom js_top_worker

To test changes to js_top_worker before merging:

1. Push your branch to a git-accessible URL
2. Pass it to day10:

```bash
dune exec -- day10 batch \
  --with-jtw \
  --jtw-tools-repo https://tangled.org/jon.recoil.org/js_top_worker \
  --jtw-tools-branch my-feature-branch \
  ...
```

The jtw-tools layer hash will change (it includes the branch name), so a
fresh toolchain build will occur.

### Troubleshooting

**jtw-tools layer fails with "Unknown archive type"**

The repo URL needs to be accessible as a git repository. Opam uses the
`git+https://` scheme internally. If your URL serves HTML instead of git,
the pin will fail. Verify with:

```bash
git ls-remote https://your-repo-url
```

**All jtw layers show status "skipped"**

This means `has_jsoo` returned false -- the jtw-tools layer doesn't contain
`js_of_ocaml`. Check the jtw-tools build log for installation failures.

**jtw layer shows status "failure" with exit code 125**

This usually means the package has no installable findlib libraries, or
`jtw opam` couldn't find any `.cma` files to compile. This is normal for
packages like `ocaml-compiler` that don't install findlib packages.

**Per-package artifacts exist in cache but jtw-output is empty**

Assembly only runs in `batch` mode, not `health-check`. Use:

```bash
dune exec -- day10 batch --with-jtw --jtw-output /path/to/output ...
```

**Content hashes change unexpectedly**

The content hash is computed from `.cmi`, `.cma.js`, and `META` file contents.
If the OCaml compiler version or any dependency changes, the compiled artifacts
will differ and produce a new hash. This is by design.