day10 Administrator's Guide#

This guide covers how to set up and run day10 as a documentation generation system for OCaml packages, intended as a replacement for ocaml-docs-ci.

Overview#

day10 builds OCaml packages and generates documentation using odoc. Key features:

Fresh solving: Always solves against current opam-repository (no stale cross-references)
Graceful degradation: Failed rebuilds preserve existing docs
Layer caching: Fast rebuilds via overlay filesystem caching
Parallel processing: Fork-based parallelism for batch runs

Prerequisites#

System Requirements#

Linux (Debian/Ubuntu recommended)
Root access (for runc containers)
At least 50GB disk space for cache
8GB+ RAM recommended

Dependencies#

# System packages
sudo apt-get update
sudo apt-get install -y \
  build-essential \
  git \
  curl \
  runc \
  opam

# Initialize opam
opam init -y
eval $(opam env)

# Install OCaml and day10 dependencies
opam switch create 5.2.0
opam install -y dune opam-0install yojson cmdliner dockerfile ppx_deriving_yojson

Clone opam-repository#

git clone https://github.com/ocaml/opam-repository /data/opam-repository

Installation#

Build day10#

git clone https://github.com/mtelvers/ohc day10
cd day10
opam install . --deps-only
dune build
dune install

Verify installation:

day10 --version
day10 --help

Directory Structure#

Recommended production layout:

/data/
├── opam-repository/     # Clone of ocaml/opam-repository
├── cache/               # Layer cache (can grow large)
│   ├── debian-12-x86_64/
│   │   ├── base/        # Base image layer
│   │   ├── solutions/   # Cached solver results
│   │   ├── build-*/     # Build layers
│   │   └── doc-*/       # Doc layers
│   └── logs/
│       ├── runs/        # Per-run logs and summaries
│       └── latest       # Symlink to most recent run
├── html/                # Generated documentation
│   ├── p/               # Blessed package docs
│   │   └── {pkg}/{ver}/
│   └── u/               # Universe docs (dependencies)
│       └── {hash}/{pkg}/{ver}/
└── packages.json        # Package list for batch runs

Basic Usage#

Single Package#

Build and generate docs for one package:

day10 health-check \
  --cache-dir /data/cache \
  --opam-repository /data/opam-repository \
  --html-output /data/html \
  base.0.16.0

Multiple Packages#

Create a JSON file listing packages:

# packages.json
{"packages": ["base.0.16.0", "core.0.16.0", "async.0.16.0"]}

Run batch mode:

day10 batch \
  --cache-dir /data/cache \
  --opam-repository /data/opam-repository \
  --html-output /data/html \
  --fork 8 \
  @packages.json

All Packages#

Generate a list of all packages in opam-repository:

day10 list \
  --opam-repository /data/opam-repository \
  --all-versions \
  --json /data/all-packages.json

Run on everything (this takes hours/days):

day10 batch \
  --cache-dir /data/cache \
  --opam-repository /data/opam-repository \
  --html-output /data/html \
  --fork 16 \
  @/data/all-packages.json

Command Reference#

day10 batch#

Main command for production use.

day10 batch [OPTIONS] PACKAGE

PACKAGE: Single package (e.g., "base.0.16.0") or @filename for JSON list

Required:
  --cache-dir DIR         Layer cache directory
  --opam-repository DIR   Path to opam-repository (can specify multiple)

Recommended:
  --html-output DIR       Where to write documentation
  --fork N                Parallel workers (default: 1)

Optional:
  --ocaml-version VER     Pin OCaml version (default: solver picks)
  --dry-run               Check what would be built without building
  --log                   Print build logs to stdout
  --json DIR              Write per-package JSON results
  --md DIR                Write per-package markdown results

day10 health-check#

Run on single package or small set (simpler than batch for testing):

day10 health-check [OPTIONS] PACKAGE

day10 list#

List packages in opam-repository:

day10 list --opam-repository DIR [--all-versions] [--json FILE]

Production Setup#

Systemd Service#

Create /etc/systemd/system/day10.service:

[Unit]
Description=day10 documentation generator
After=network.target

[Service]
Type=oneshot
User=root
WorkingDirectory=/data
ExecStart=/usr/local/bin/day10 batch \
  --cache-dir /data/cache \
  --opam-repository /data/opam-repository \
  --html-output /data/html \
  --fork 8 \
  @/data/packages.json
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Cron Job#

For periodic rebuilds (e.g., daily at 2 AM):

# /etc/cron.d/day10
0 2 * * * root flock -n /var/run/day10.lock /usr/local/bin/day10 batch --cache-dir /data/cache --opam-repository /data/opam-repository --html-output /data/html --fork 8 @/data/packages.json >> /var/log/day10-cron.log 2>&1

Webhook Trigger#

To rebuild on opam-repository updates, set up a webhook endpoint that:

Pulls latest opam-repository
Triggers day10 batch run

Example script /usr/local/bin/day10-trigger.sh:

#!/bin/bash
set -e

cd /data/opam-repository
git fetch origin
git reset --hard origin/master

flock -n /var/run/day10.lock \
  day10 batch \
    --cache-dir /data/cache \
    --opam-repository /data/opam-repository \
    --html-output /data/html \
    --fork 8 \
    @/data/packages.json

Serving Documentation#

Use nginx to serve the HTML output:

server {
    listen 80;
    server_name docs.example.com;
    root /data/html;

    location / {
        autoindex on;
        try_files $uri $uri/ =404;
    }
}

Status Dashboard (day10-web)#

day10-web provides a web interface for monitoring package build status:

# Install day10-web
opam install day10-web

# Run the dashboard
day10-web --cache-dir /data/cache --html-dir /data/html --port 8080

Systemd Service for day10-web#

Create /etc/systemd/system/day10-web.service:

[Unit]
Description=day10 status dashboard
After=network.target

[Service]
Type=simple
User=www-data
ExecStart=/usr/local/bin/day10-web \
  --cache-dir /data/cache \
  --html-dir /data/html \
  --host 0.0.0.0 \
  --port 8080
Restart=always

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl enable day10-web
sudo systemctl start day10-web

Combined nginx Configuration#

Serve both the dashboard and documentation:

server {
    listen 80;
    server_name docs.example.com;

    # Status dashboard
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # Generated documentation
    location /docs/ {
        alias /data/html/;
        autoindex on;
        try_files $uri $uri/ =404;
    }
}

Dashboard Features#

Dashboard (/): Overview with build/doc success rates, latest run summary
Packages (/packages): Searchable list of all packages with docs
Package Detail (/packages/{name}/{version}): Version list and doc links
Runs (/runs): History of all batch runs
Run Detail (/runs/{id}): Statistics, failures, and log links
Logs (/runs/{id}/build/{pkg}, /runs/{id}/docs/{pkg}): View build and doc logs

Monitoring#

Run Logs#

Each batch run creates a timestamped directory:

/data/cache/logs/runs/2026-02-04-120000/
├── summary.json      # Run statistics
├── build/            # Build logs by package
│   ├── base.0.16.0.log
│   └── core.0.16.0.log
└── docs/             # Doc generation logs
    ├── base.0.16.0.log
    └── core.0.16.0.log

The latest symlink always points to the most recent run:

cat /data/cache/logs/latest/summary.json

summary.json Format#

{
  "run_id": "2026-02-04-120000",
  "start_time": "2026-02-04T12:00:00",
  "end_time": "2026-02-04T14:30:00",
  "duration_seconds": 9000,
  "targets_requested": 100,
  "solutions_found": 95,
  "build_success": 90,
  "build_failed": 5,
  "doc_success": 85,
  "doc_failed": 3,
  "doc_skipped": 2,
  "failures": [
    {"package": "broken-pkg.1.0.0", "error": "build exit code 2"},
    {"package": "bad-docs.2.0.0", "error": "doc: odoc error"}
  ]
}

Checking Status#

# Quick status
jq '.build_success, .build_failed, .doc_success, .doc_failed' \
  /data/cache/logs/latest/summary.json

# List failures
jq -r '.failures[] | "\(.package): \(.error)"' \
  /data/cache/logs/latest/summary.json

# Duration
jq '.duration_seconds / 60 | floor | "\(.)m"' \
  /data/cache/logs/latest/summary.json

Disk Usage#

Monitor cache growth:

du -sh /data/cache/debian-12-x86_64/
du -sh /data/html/

Maintenance#

Cache Management#

The cache grows over time. After each batch run, garbage collection automatically:

Layer GC: Deletes build/doc layers not referenced by current solutions
Universe GC: Deletes universe directories not referenced by any blessed package

GC runs automatically at the end of each batch. Special layers are preserved:

base - Base OS image
solutions - Solver cache
doc-driver-* - Shared odoc driver
doc-odoc-* - Per-OCaml-version odoc

Manual Cache Cleanup#

To force a complete rebuild:

# Remove all layers (keeps base)
rm -rf /data/cache/debian-12-x86_64/build-*
rm -rf /data/cache/debian-12-x86_64/doc-*

# Remove solution cache (forces re-solving)
rm -rf /data/cache/debian-12-x86_64/solutions/

Updating opam-repository#

cd /data/opam-repository
git fetch origin
git reset --hard origin/master

Solutions are cached by opam-repository commit hash, so updating automatically invalidates old solutions.

Epoch Transitions#

For major changes (new odoc version, URL scheme change), you may want a clean rebuild:

Create new html directory: /data/html-new/
Run full batch with --html-output /data/html-new/
Once complete, atomically swap: mv /data/html /data/html-old && mv /data/html-new /data/html
Remove old: rm -rf /data/html-old

Troubleshooting#

Build Failures#

Check the build log:

cat /data/cache/logs/latest/build/failing-pkg.1.0.0.log

Or check the layer directly:

cat /data/cache/debian-12-x86_64/build-*/build.log

Doc Generation Failures#

cat /data/cache/logs/latest/docs/failing-pkg.1.0.0.log

Common issues:

Missing .cmti files (package doesn't install them)
odoc bugs with certain code patterns
Memory exhaustion on large packages

Stale .new/.old Directories#

If a run was interrupted, stale staging directories may exist:

find /data/html -name "*.new" -o -name "*.old"

These are automatically cleaned up at the start of each batch run.

Permission Issues#

day10 uses runc containers which require root. If you see permission errors:

# Check runc works
sudo runc --version

# Ensure cache directory is accessible
sudo chown -R root:root /data/cache

Memory Issues#

For large package sets, you may need to limit parallelism:

# Reduce fork count
day10 batch --fork 4 ...

Or increase system memory/swap.

Architecture Notes#

How Layers Work#

Each package build creates a layer using overlay filesystem:

build-{hash}/
├── fs/           # Filesystem overlay (installed files)
├── build.log     # Build output
└── layer.json    # Metadata (package, deps, status)

The hash is computed from the package and its dependencies, so unchanged packages reuse existing layers.

Blessing#

In batch mode, day10 computes "blessings" - which package version is canonical for each package name. Blessed packages go to /html/p/, non-blessed go to /html/u/{universe}/.

Graceful Degradation#

When doc generation fails:

New docs are written to a staging directory
On success: atomically swap staging → final
On failure: staging is discarded, old docs remain

This ensures the live site never shows broken docs.

Getting Help#

Check logs in /data/cache/logs/latest/
Review summary.json for failure details
File issues at: https://github.com/mtelvers/ohc/issues