jon.recoil.org
A fork of mtelver's day10 project
1# day10 Administrator's Guide
2
3This guide covers how to set up and run day10 as a documentation generation system for OCaml packages, intended as a replacement for ocaml-docs-ci.
4
5## Overview
6
7day10 builds OCaml packages and generates documentation using odoc. Key features:
8
9- **Fresh solving**: Always solves against current opam-repository (no stale cross-references)
10- **Graceful degradation**: Failed rebuilds preserve existing docs
11- **Layer caching**: Fast rebuilds via overlay filesystem caching
12- **Parallel processing**: Fork-based parallelism for batch runs
13
14## Prerequisites
15
16### System Requirements
17
18- Linux (Debian/Ubuntu recommended)
19- Root access (for runc containers)
20- At least 50GB disk space for cache
21- 8GB+ RAM recommended
22
23### Dependencies
24
25```bash
26# System packages
27sudo apt-get update
28sudo apt-get install -y \
29 build-essential \
30 git \
31 curl \
32 runc \
33 opam
34
35# Initialize opam
36opam init -y
37eval $(opam env)
38
39# Install OCaml and day10 dependencies
40opam switch create 5.2.0
41opam install -y dune opam-0install yojson cmdliner dockerfile ppx_deriving_yojson
42```
43
44### Clone opam-repository
45
46```bash
47git clone https://github.com/ocaml/opam-repository /data/opam-repository
48```
49
50## Installation
51
52### Build day10
53
54```bash
55git clone https://github.com/mtelvers/ohc day10
56cd day10
57opam install . --deps-only
58dune build
59dune install
60```
61
62Verify installation:
63```bash
64day10 --version
65day10 --help
66```
67
68## Directory Structure
69
70Recommended production layout:
71
72```
73/data/
74├── opam-repository/ # Clone of ocaml/opam-repository
75├── cache/ # Layer cache (can grow large)
76│ ├── debian-12-x86_64/
77│ │ ├── base/ # Base image layer
78│ │ ├── solutions/ # Cached solver results
79│ │ ├── build-*/ # Build layers
80│ │ └── doc-*/ # Doc layers
81│ └── logs/
82│ ├── runs/ # Per-run logs and summaries
83│ └── latest # Symlink to most recent run
84├── html/ # Generated documentation
85│ ├── p/ # Blessed package docs
86│ │ └── {pkg}/{ver}/
87│ └── u/ # Universe docs (dependencies)
88│ └── {hash}/{pkg}/{ver}/
89└── packages.json # Package list for batch runs
90```
91
92## Basic Usage
93
94### Single Package
95
96Build and generate docs for one package:
97
98```bash
99day10 health-check \
100 --cache-dir /data/cache \
101 --opam-repository /data/opam-repository \
102 --html-output /data/html \
103 base.0.16.0
104```
105
106### Multiple Packages
107
108Create a JSON file listing packages:
109
110```bash
111# packages.json
112{"packages": ["base.0.16.0", "core.0.16.0", "async.0.16.0"]}
113```
114
115Run batch mode:
116
117```bash
118day10 batch \
119 --cache-dir /data/cache \
120 --opam-repository /data/opam-repository \
121 --html-output /data/html \
122 --fork 8 \
123 @packages.json
124```
125
126### All Packages
127
128Generate a list of all packages in opam-repository:
129
130```bash
131day10 list \
132 --opam-repository /data/opam-repository \
133 --all-versions \
134 --json /data/all-packages.json
135```
136
137Run on everything (this takes hours/days):
138
139```bash
140day10 batch \
141 --cache-dir /data/cache \
142 --opam-repository /data/opam-repository \
143 --html-output /data/html \
144 --fork 16 \
145 @/data/all-packages.json
146```
147
148## Command Reference
149
150### day10 batch
151
152Main command for production use.
153
154```
155day10 batch [OPTIONS] PACKAGE
156
157PACKAGE: Single package (e.g., "base.0.16.0") or @filename for JSON list
158
159Required:
160 --cache-dir DIR Layer cache directory
161 --opam-repository DIR Path to opam-repository (can specify multiple)
162
163Recommended:
164 --html-output DIR Where to write documentation
165 --fork N Parallel workers (default: 1)
166
167Optional:
168 --ocaml-version VER Pin OCaml version (default: solver picks)
169 --dry-run Check what would be built without building
170 --log Print build logs to stdout
171 --json DIR Write per-package JSON results
172 --md DIR Write per-package markdown results
173```
174
175### day10 health-check
176
177Run on single package or small set (simpler than batch for testing):
178
179```
180day10 health-check [OPTIONS] PACKAGE
181```
182
183### day10 list
184
185List packages in opam-repository:
186
187```
188day10 list --opam-repository DIR [--all-versions] [--json FILE]
189```
190
191## Production Setup
192
193### Systemd Service
194
195Create `/etc/systemd/system/day10.service`:
196
197```ini
198[Unit]
199Description=day10 documentation generator
200After=network.target
201
202[Service]
203Type=oneshot
204User=root
205WorkingDirectory=/data
206ExecStart=/usr/local/bin/day10 batch \
207 --cache-dir /data/cache \
208 --opam-repository /data/opam-repository \
209 --html-output /data/html \
210 --fork 8 \
211 @/data/packages.json
212StandardOutput=journal
213StandardError=journal
214
215[Install]
216WantedBy=multi-user.target
217```
218
219### Cron Job
220
221For periodic rebuilds (e.g., daily at 2 AM):
222
223```bash
224# /etc/cron.d/day10
2250 2 * * * root flock -n /var/run/day10.lock /usr/local/bin/day10 batch --cache-dir /data/cache --opam-repository /data/opam-repository --html-output /data/html --fork 8 @/data/packages.json >> /var/log/day10-cron.log 2>&1
226```
227
228### Webhook Trigger
229
230To rebuild on opam-repository updates, set up a webhook endpoint that:
231
2321. Pulls latest opam-repository
2332. Triggers day10 batch run
234
235Example script `/usr/local/bin/day10-trigger.sh`:
236
237```bash
238#!/bin/bash
239set -e
240
241cd /data/opam-repository
242git fetch origin
243git reset --hard origin/master
244
245flock -n /var/run/day10.lock \
246 day10 batch \
247 --cache-dir /data/cache \
248 --opam-repository /data/opam-repository \
249 --html-output /data/html \
250 --fork 8 \
251 @/data/packages.json
252```
253
254### Serving Documentation
255
256Use nginx to serve the HTML output:
257
258```nginx
259server {
260 listen 80;
261 server_name docs.example.com;
262 root /data/html;
263
264 location / {
265 autoindex on;
266 try_files $uri $uri/ =404;
267 }
268}
269```
270
271### Status Dashboard (day10-web)
272
273day10-web provides a web interface for monitoring package build status:
274
275```bash
276# Install day10-web
277opam install day10-web
278
279# Run the dashboard
280day10-web --cache-dir /data/cache --html-dir /data/html --port 8080
281```
282
283#### Systemd Service for day10-web
284
285Create `/etc/systemd/system/day10-web.service`:
286
287```ini
288[Unit]
289Description=day10 status dashboard
290After=network.target
291
292[Service]
293Type=simple
294User=www-data
295ExecStart=/usr/local/bin/day10-web \
296 --cache-dir /data/cache \
297 --html-dir /data/html \
298 --host 0.0.0.0 \
299 --port 8080
300Restart=always
301
302[Install]
303WantedBy=multi-user.target
304```
305
306Enable and start:
307
308```bash
309sudo systemctl enable day10-web
310sudo systemctl start day10-web
311```
312
313#### Combined nginx Configuration
314
315Serve both the dashboard and documentation:
316
317```nginx
318server {
319 listen 80;
320 server_name docs.example.com;
321
322 # Status dashboard
323 location / {
324 proxy_pass http://127.0.0.1:8080;
325 proxy_set_header Host $host;
326 proxy_set_header X-Real-IP $remote_addr;
327 }
328
329 # Generated documentation
330 location /docs/ {
331 alias /data/html/;
332 autoindex on;
333 try_files $uri $uri/ =404;
334 }
335}
336```
337
338#### Dashboard Features
339
340- **Dashboard** (`/`): Overview with build/doc success rates, latest run summary
341- **Packages** (`/packages`): Searchable list of all packages with docs
342- **Package Detail** (`/packages/{name}/{version}`): Version list and doc links
343- **Runs** (`/runs`): History of all batch runs
344- **Run Detail** (`/runs/{id}`): Statistics, failures, and log links
345- **Logs** (`/runs/{id}/build/{pkg}`, `/runs/{id}/docs/{pkg}`): View build and doc logs
346
347## Monitoring
348
349### Run Logs
350
351Each batch run creates a timestamped directory:
352
353```
354/data/cache/logs/runs/2026-02-04-120000/
355├── summary.json # Run statistics
356├── build/ # Build logs by package
357│ ├── base.0.16.0.log
358│ └── core.0.16.0.log
359└── docs/ # Doc generation logs
360 ├── base.0.16.0.log
361 └── core.0.16.0.log
362```
363
364The `latest` symlink always points to the most recent run:
365
366```bash
367cat /data/cache/logs/latest/summary.json
368```
369
370### summary.json Format
371
372```json
373{
374 "run_id": "2026-02-04-120000",
375 "start_time": "2026-02-04T12:00:00",
376 "end_time": "2026-02-04T14:30:00",
377 "duration_seconds": 9000,
378 "targets_requested": 100,
379 "solutions_found": 95,
380 "build_success": 90,
381 "build_failed": 5,
382 "doc_success": 85,
383 "doc_failed": 3,
384 "doc_skipped": 2,
385 "failures": [
386 {"package": "broken-pkg.1.0.0", "error": "build exit code 2"},
387 {"package": "bad-docs.2.0.0", "error": "doc: odoc error"}
388 ]
389}
390```
391
392### Checking Status
393
394```bash
395# Quick status
396jq '.build_success, .build_failed, .doc_success, .doc_failed' \
397 /data/cache/logs/latest/summary.json
398
399# List failures
400jq -r '.failures[] | "\(.package): \(.error)"' \
401 /data/cache/logs/latest/summary.json
402
403# Duration
404jq '.duration_seconds / 60 | floor | "\(.)m"' \
405 /data/cache/logs/latest/summary.json
406```
407
408### Disk Usage
409
410Monitor cache growth:
411
412```bash
413du -sh /data/cache/debian-12-x86_64/
414du -sh /data/html/
415```
416
417## Maintenance
418
419### Cache Management
420
421The cache grows over time. After each batch run, garbage collection automatically:
422
4231. **Layer GC**: Deletes build/doc layers not referenced by current solutions
4242. **Universe GC**: Deletes universe directories not referenced by any blessed package
425
426GC runs automatically at the end of each batch. Special layers are preserved:
427- `base` - Base OS image
428- `solutions` - Solver cache
429- `doc-driver-*` - Shared odoc driver
430- `doc-odoc-*` - Per-OCaml-version odoc
431
432### Manual Cache Cleanup
433
434To force a complete rebuild:
435
436```bash
437# Remove all layers (keeps base)
438rm -rf /data/cache/debian-12-x86_64/build-*
439rm -rf /data/cache/debian-12-x86_64/doc-*
440
441# Remove solution cache (forces re-solving)
442rm -rf /data/cache/debian-12-x86_64/solutions/
443```
444
445### Updating opam-repository
446
447```bash
448cd /data/opam-repository
449git fetch origin
450git reset --hard origin/master
451```
452
453Solutions are cached by opam-repository commit hash, so updating automatically invalidates old solutions.
454
455### Epoch Transitions
456
457For major changes (new odoc version, URL scheme change), you may want a clean rebuild:
458
4591. Create new html directory: `/data/html-new/`
4602. Run full batch with `--html-output /data/html-new/`
4613. Once complete, atomically swap: `mv /data/html /data/html-old && mv /data/html-new /data/html`
4624. Remove old: `rm -rf /data/html-old`
463
464## Troubleshooting
465
466### Build Failures
467
468Check the build log:
469
470```bash
471cat /data/cache/logs/latest/build/failing-pkg.1.0.0.log
472```
473
474Or check the layer directly:
475
476```bash
477cat /data/cache/debian-12-x86_64/build-*/build.log
478```
479
480### Doc Generation Failures
481
482```bash
483cat /data/cache/logs/latest/docs/failing-pkg.1.0.0.log
484```
485
486Common issues:
487- Missing `.cmti` files (package doesn't install them)
488- odoc bugs with certain code patterns
489- Memory exhaustion on large packages
490
491### Stale .new/.old Directories
492
493If a run was interrupted, stale staging directories may exist:
494
495```bash
496find /data/html -name "*.new" -o -name "*.old"
497```
498
499These are automatically cleaned up at the start of each batch run.
500
501### Permission Issues
502
503day10 uses runc containers which require root. If you see permission errors:
504
505```bash
506# Check runc works
507sudo runc --version
508
509# Ensure cache directory is accessible
510sudo chown -R root:root /data/cache
511```
512
513### Memory Issues
514
515For large package sets, you may need to limit parallelism:
516
517```bash
518# Reduce fork count
519day10 batch --fork 4 ...
520```
521
522Or increase system memory/swap.
523
524## Architecture Notes
525
526### How Layers Work
527
528Each package build creates a layer using overlay filesystem:
529
530```
531build-{hash}/
532├── fs/ # Filesystem overlay (installed files)
533├── build.log # Build output
534└── layer.json # Metadata (package, deps, status)
535```
536
537The hash is computed from the package and its dependencies, so unchanged packages reuse existing layers.
538
539### Blessing
540
541In batch mode, day10 computes "blessings" - which package version is canonical for each package name. Blessed packages go to `/html/p/`, non-blessed go to `/html/u/{universe}/`.
542
543### Graceful Degradation
544
545When doc generation fails:
5461. New docs are written to a staging directory
5472. On success: atomically swap staging → final
5483. On failure: staging is discarded, old docs remain
549
550This ensures the live site never shows broken docs.
551
552## Getting Help
553
554- Check logs in `/data/cache/logs/latest/`
555- Review `summary.json` for failure details
556- File issues at: https://github.com/mtelvers/ohc/issues