jon.recoil.org
this repo has no description
1# js_top_worker Architecture
2
3This document describes the current architecture of js_top_worker and the planned changes.
4
5## Overview
6
7js_top_worker is an OCaml toplevel (REPL) designed to run in a Web Worker or remote process. It enables interactive OCaml execution in browsers for:
8
9- Jupyter-style notebooks
10- Interactive documentation
11- Educational tools (lecture slides, tutorials)
12- Library documentation with live examples
13
14## System Architecture
15
16```
17┌─────────────────────────────────────────────────────────────────┐
18│ Browser │
19│ ┌──────────────────┐ ┌──────────────────────────────┐ │
20│ │ Frontend │ │ Web Worker │ │
21│ │ │ │ │ │
22│ │ ┌────────────┐ │ RPC │ ┌────────────────────────┐ │ │
23│ │ │ Client │◄─┼────────►│ │ Server │ │ │
24│ │ │ (Lwt/Fut) │ │ JSON │ │ (worker.ml) │ │ │
25│ │ └────────────┘ │ │ └──────────┬─────────────┘ │ │
26│ │ │ │ │ │ │
27│ │ │ │ ┌──────────▼─────────────┐ │ │
28│ │ │ │ │ Implementation │ │ │
29│ │ │ │ │ (impl.ml) │ │ │
30│ │ │ │ │ - Execute phrases │ │ │
31│ │ │ │ │ - Type checking │ │ │
32│ │ │ │ │ - Code completion │ │ │
33│ │ │ │ └──────────┬─────────────┘ │ │
34│ │ │ │ │ │ │
35│ │ │ │ ┌──────────▼─────────────┐ │ │
36│ │ │ │ │ js_of_ocaml-toplevel │ │ │
37│ │ │ │ │ + Merlin │ │ │
38│ │ │ │ └────────────────────────┘ │ │
39│ └──────────────────┘ └──────────────────────────────┘ │
40└─────────────────────────────────────────────────────────────────┘
41```
42
43## Package Structure
44
45| Package | Purpose | Key Files |
46|---------|---------|-----------|
47| `js_top_worker` | Core toplevel implementation | `lib/impl.ml`, `lib/ocamltop.ml` |
48| `js_top_worker-web` | Web Worker implementation | `lib/worker.ml`, `lib/findlibish.ml` |
49| `js_top_worker-rpc` | RPC type definitions | `idl/toplevel_api.ml` |
50| `js_top_worker-client` | Lwt-based client | `idl/js_top_worker_client.ml` |
51| `js_top_worker-client_fut` | Fut-based client | `idl/js_top_worker_client_fut.ml` |
52| `js_top_worker-unix` | Unix socket backend (testing) | - |
53| `js_top_worker-bin` | CLI tools (`jtw`) | `bin/jtw.ml` |
54
55## Current Communication Layer
56
57### RPC Protocol
58
59Uses [ocaml-rpc](https://github.com/mirage/ocaml-rpc) with JSON-RPC 2.0:
60
61```
62Client Server (Worker)
63 │ │
64 │ ──── JSON-RPC request ────────► │
65 │ {method: "exec", │
66 │ params: ["let x = 1"], │
67 │ id: 1} │
68 │ │
69 │ ◄─── JSON-RPC response ──────── │
70 │ {result: {...}, │
71 │ id: 1} │
72 │ │
73```
74
75### RPC Operations
76
77| Method | Parameters | Returns | Description |
78|--------|------------|---------|-------------|
79| `init` | `init_config` | `unit` | Initialize toplevel |
80| `setup` | `unit` | `exec_result` | Start toplevel |
81| `exec` | `string` | `exec_result` | Execute OCaml phrase |
82| `typecheck` | `string` | `exec_result` | Type check without execution |
83| `complete_prefix` | `id, deps, source, position` | `completions` | Autocomplete |
84| `query_errors` | `id, deps, source` | `error list` | Get compilation errors |
85| `type_enclosing` | `id, deps, source, position` | `typed_enclosings` | Type at position |
86
87### Type Definitions
88
89Key types from `idl/toplevel_api.ml`:
90
91```ocaml
92type exec_result = {
93 stdout : string option;
94 stderr : string option;
95 sharp_ppf : string option; (* # directive output *)
96 caml_ppf : string option; (* Regular output *)
97 highlight : highlight option; (* Error location *)
98 mime_vals : mime_val list; (* Rich output *)
99}
100
101type mime_val = {
102 mime_type : string; (* e.g., "text/html" *)
103 encoding : encoding; (* Noencoding | Base64 *)
104 data : string;
105}
106
107type init_config = {
108 findlib_requires : string list; (* Packages to preload *)
109 stdlib_dcs : string option; (* Dynamic CMIs URL *)
110 execute : bool; (* Allow execution? *)
111}
112```
113
114## Core Implementation
115
116### Module Structure (`lib/impl.ml`)
117
118```ocaml
119module type S = sig
120 type findlib_t
121 val capture : (unit -> 'a) -> unit -> captured * 'a
122 val sync_get : string -> string option
123 val async_get : string -> (string, [`Msg of string]) result Lwt.t
124 val import_scripts : string list -> unit
125 val get_stdlib_dcs : string -> dynamic_cmis list
126 val findlib_init : string -> findlib_t Lwt.t
127 val require : bool -> findlib_t -> string list -> dynamic_cmis list
128 val path : string
129end
130
131module Make (S : S) : sig
132 val init : init_config -> unit Lwt.t
133 val setup : unit -> exec_result Lwt.t
134 val exec : string -> exec_result Lwt.t
135 val typecheck : string -> exec_result Lwt.t
136 (* ... *)
137end
138```
139
140### Execution Flow
141
142```
143exec(phrase)
144 │
145 ▼
146capture stdout/stderr
147 │
148 ▼
149parse phrase (Ocamltop.parse_toplevel)
150 │
151 ▼
152execute (Toploop.execute_phrase)
153 │
154 ▼
155collect MIME outputs
156 │
157 ▼
158return exec_result
159```
160
161### Cell Dependency System
162
163Cells can depend on previous cells via module wrapping:
164
165```ocaml
166(* Cell "c1" defines: *)
167let x = 1
168
169(* Internally becomes module Cell__c1 *)
170
171(* Cell "c2" with deps=["c1"]: *)
172let y = x + 1
173
174(* Prepended with: open Cell__c1 *)
175```
176
177The `mangle_toplevel` function handles this transformation.
178
179## Library Loading
180
181### findlibish.ml
182
183Custom findlib-like implementation for WebWorker context:
184
185```
186 ┌─────────────────┐
187 │ findlib_index.json │ (list of META URLs)
188 └────────┬────────┘
189 │
190 ┌──────────────┼──────────────┐
191 ▼ ▼ ▼
192 ┌─────────┐ ┌─────────┐ ┌─────────┐
193 │ META │ │ META │ │ META │
194 │ (pkg A) │ │ (pkg B) │ │ (pkg C) │
195 └────┬────┘ └────┬────┘ └────┬────┘
196 │ │ │
197 ▼ ▼ ▼
198 ┌─────────────────────────────────────┐
199 │ Dependency Resolution │
200 └─────────────────┬───────────────────┘
201 │
202 ┌────────────┼────────────┐
203 ▼ ▼ ▼
204 ┌──────────┐ ┌──────────┐ ┌──────────┐
205 │ .cma.js │ │ .cma.js │ │ .cma.js │
206 │ (import) │ │ (import) │ │ (import) │
207 └──────────┘ └──────────┘ └──────────┘
208```
209
210### Package Loading Process
211
2121. Fetch `findlib_index.json` (list of META file URLs)
2132. Parse each META file with `Fl_metascanner`
2143. Build dependency graph
2154. On `#require`:
216 - Resolve dependencies
217 - Fetch `dynamic_cmis.json` for each package
218 - Load `.cma.js` via `import_scripts`
219
220### Preloaded Packages
221
222These are compiled into the worker and not loaded dynamically:
223
224- `compiler-libs.common`, `compiler-libs.toplevel`
225- `merlin-lib.*`
226- `js_of_ocaml-compiler`, `js_of_ocaml-toplevel`
227- `findlib`, `findlib.top`
228
229## Merlin Integration
230
231Code intelligence features use Merlin:
232
233| Feature | Merlin Query | Implementation |
234|---------|--------------|----------------|
235| Completion | `Query_protocol.Complete_prefix` | `complete_prefix` |
236| Type info | `Query_protocol.Type_enclosing` | `type_enclosing` |
237| Errors | `Query_protocol.Errors` | `query_errors` |
238
239Queries run through `Mpipeline` with source "mangled" to include cell dependencies.
240
241## Planned Architecture Changes
242
243### Phase 1: Communication Redesign
244
245Replace JSON-RPC with CBOR-based bidirectional channel:
246
247```
248Current: Planned:
249┌─────────┐ JSON-RPC ┌─────────┐ CBOR
250│ Client │◄──────────────► │ Client │◄──────────────►
251│ │ request/response │ │ bidirectional
252└─────────┘ └─────────┘
253
254 Message types:
255 - Request/Response (like RPC)
256 - Push (server → client)
257 - Widget events (bidirectional)
258```
259
260### Phase 2: Environment Isolation
261
262Multiple isolated execution contexts:
263
264```
265┌──────────────────────────────────────────┐
266│ Web Worker │
267│ │
268│ ┌─────────────┐ ┌─────────────┐ │
269│ │ Env "a" │ │ Env "b" │ │
270│ │ │ │ │ │
271│ │ Cell 1 │ │ Cell 1 │ │
272│ │ Cell 2 │ │ Cell 2 │ │
273│ │ (isolated) │ │ (isolated) │ │
274│ └─────────────┘ └─────────────┘ │
275│ │
276│ Shared: stdlib, preloaded packages │
277└──────────────────────────────────────────┘
278```
279
280### Phase 3: Rich Output & Widgets
281
282MIME-typed output with bidirectional widget communication:
283
284```ocaml
285(* User code *)
286let chart = Chart.bar [1; 2; 3; 4] in
287Display.show chart
288
289(* Generates *)
290{
291 mime_type = "application/vnd.widget+json";
292 data = {widget_id = "w1"; state = ...}
293}
294
295(* Frontend renders widget, sends events back *)
296Widget_event {widget_id = "w1"; event = Click {x; y}}
297```
298
299## File Reference
300
301### Core Files
302
303| File | Lines | Purpose |
304|------|-------|---------|
305| `lib/impl.ml` | 985 | Main implementation (execute, typecheck, etc.) |
306| `lib/worker.ml` | 100 | WebWorker server setup |
307| `lib/findlibish.ml` | 221 | Package loading |
308| `idl/toplevel_api.ml` | 315 | RPC type definitions |
309| `idl/js_top_worker_client.ml` | 126 | Lwt client |
310
311### Build Outputs
312
313| File | Description |
314|------|-------------|
315| `worker.bc.js` | Compiled Web Worker |
316| `*.cma.js` | JavaScript-compiled OCaml libraries |
317| `dynamic_cmis.json` | CMI metadata for each package |
318
319## Dependencies
320
321### Runtime
322
323- `js_of_ocaml` >= 3.11.0
324- `js_of_ocaml-toplevel`
325- `js_of_ocaml-compiler`
326- `rpclib`, `rpclib-lwt`
327- `merlin-lib`
328- `compiler-libs`
329- `brr` >= 0.0.4
330
331### Planned Additions
332
333- `cbort` - CBOR codec (tangled.org)
334- `zarith_stubs_js` - JS stubs for zarith
335- `bytesrw` - Streaming I/O
336
337---
338
339*Last updated: 2026-01-20*