# day10 Administrator's Guide

This guide covers how to set up and run day10 as a documentation generation system for OCaml packages, intended as a replacement for ocaml-docs-ci.

## Overview

day10 builds OCaml packages and generates documentation using odoc. Key features:

- **Fresh solving**: Always solves against current opam-repository (no stale cross-references)
- **Graceful degradation**: Failed rebuilds preserve existing docs
- **Layer caching**: Fast rebuilds via overlay filesystem caching
- **Parallel processing**: Fork-based parallelism for batch runs

## Prerequisites

### System Requirements

- Linux (Debian/Ubuntu recommended)
- Root access (for runc containers)
- At least 50GB disk space for cache
- 8GB+ RAM recommended

### Dependencies

```bash
# System packages
sudo apt-get update
sudo apt-get install -y \
  build-essential \
  git \
  curl \
  runc \
  opam

# Initialize opam
opam init -y
eval $(opam env)

# Install OCaml and day10 dependencies
opam switch create 5.2.0
opam install -y dune opam-0install yojson cmdliner dockerfile ppx_deriving_yojson
```

### Clone opam-repository

```bash
git clone https://github.com/ocaml/opam-repository /data/opam-repository
```

## Installation

### Build day10

```bash
git clone https://github.com/mtelvers/ohc day10
cd day10
opam install . --deps-only
dune build
dune install
```

Verify installation:
```bash
day10 --version
day10 --help
```

## Directory Structure

Recommended production layout:

```
/data/
├── opam-repository/     # Clone of ocaml/opam-repository
├── cache/               # Layer cache (can grow large)
│   ├── debian-12-x86_64/
│   │   ├── base/        # Base image layer
│   │   ├── solutions/   # Cached solver results
│   │   ├── build-*/     # Build layers
│   │   └── doc-*/       # Doc layers
│   └── logs/
│       ├── runs/        # Per-run logs and summaries
│       └── latest       # Symlink to most recent run
├── html/                # Generated documentation
│   ├── p/               # Blessed package docs
│   │   └── {pkg}/{ver}/
│   └── u/               # Universe docs (dependencies)
│       └── {hash}/{pkg}/{ver}/
└── packages.json        # Package list for batch runs
```

## Basic Usage

### Single Package

Build and generate docs for one package:

```bash
day10 health-check \
  --cache-dir /data/cache \
  --opam-repository /data/opam-repository \
  --html-output /data/html \
  base.0.16.0
```

### Multiple Packages

Create a JSON file listing packages:

```bash
# packages.json
{"packages": ["base.0.16.0", "core.0.16.0", "async.0.16.0"]}
```

Run batch mode:

```bash
day10 batch \
  --cache-dir /data/cache \
  --opam-repository /data/opam-repository \
  --html-output /data/html \
  --fork 8 \
  @packages.json
```

### All Packages

Generate a list of all packages in opam-repository:

```bash
day10 list \
  --opam-repository /data/opam-repository \
  --all-versions \
  --json /data/all-packages.json
```

Run on everything (this takes hours/days):

```bash
day10 batch \
  --cache-dir /data/cache \
  --opam-repository /data/opam-repository \
  --html-output /data/html \
  --fork 16 \
  @/data/all-packages.json
```

## Command Reference

### day10 batch

Main command for production use.

```
day10 batch [OPTIONS] PACKAGE

PACKAGE: Single package (e.g., "base.0.16.0") or @filename for JSON list

Required:
  --cache-dir DIR         Layer cache directory
  --opam-repository DIR   Path to opam-repository (can specify multiple)

Recommended:
  --html-output DIR       Where to write documentation
  --fork N                Parallel workers (default: 1)

Optional:
  --ocaml-version VER     Pin OCaml version (default: solver picks)
  --dry-run               Check what would be built without building
  --log                   Print build logs to stdout
  --json DIR              Write per-package JSON results
  --md DIR                Write per-package markdown results
```

### day10 health-check

Run on single package or small set (simpler than batch for testing):

```
day10 health-check [OPTIONS] PACKAGE
```

### day10 list

List packages in opam-repository:

```
day10 list --opam-repository DIR [--all-versions] [--json FILE]
```

## Production Setup

### Systemd Service

Create `/etc/systemd/system/day10.service`:

```ini
[Unit]
Description=day10 documentation generator
After=network.target

[Service]
Type=oneshot
User=root
WorkingDirectory=/data
ExecStart=/usr/local/bin/day10 batch \
  --cache-dir /data/cache \
  --opam-repository /data/opam-repository \
  --html-output /data/html \
  --fork 8 \
  @/data/packages.json
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

### Cron Job

For periodic rebuilds (e.g., daily at 2 AM):

```bash
# /etc/cron.d/day10
0 2 * * * root flock -n /var/run/day10.lock /usr/local/bin/day10 batch --cache-dir /data/cache --opam-repository /data/opam-repository --html-output /data/html --fork 8 @/data/packages.json >> /var/log/day10-cron.log 2>&1
```

### Webhook Trigger

To rebuild on opam-repository updates, set up a webhook endpoint that:

1. Pulls latest opam-repository
2. Triggers day10 batch run

Example script `/usr/local/bin/day10-trigger.sh`:

```bash
#!/bin/bash
set -e

cd /data/opam-repository
git fetch origin
git reset --hard origin/master

flock -n /var/run/day10.lock \
  day10 batch \
    --cache-dir /data/cache \
    --opam-repository /data/opam-repository \
    --html-output /data/html \
    --fork 8 \
    @/data/packages.json
```

### Serving Documentation

Use nginx to serve the HTML output:

```nginx
server {
    listen 80;
    server_name docs.example.com;
    root /data/html;

    location / {
        autoindex on;
        try_files $uri $uri/ =404;
    }
}
```

### Status Dashboard (day10-web)

day10-web provides a web interface for monitoring package build status:

```bash
# Install day10-web
opam install day10-web

# Run the dashboard
day10-web --cache-dir /data/cache --html-dir /data/html --port 8080
```

#### Systemd Service for day10-web

Create `/etc/systemd/system/day10-web.service`:

```ini
[Unit]
Description=day10 status dashboard
After=network.target

[Service]
Type=simple
User=www-data
ExecStart=/usr/local/bin/day10-web \
  --cache-dir /data/cache \
  --html-dir /data/html \
  --host 0.0.0.0 \
  --port 8080
Restart=always

[Install]
WantedBy=multi-user.target
```

Enable and start:

```bash
sudo systemctl enable day10-web
sudo systemctl start day10-web
```

#### Combined nginx Configuration

Serve both the dashboard and documentation:

```nginx
server {
    listen 80;
    server_name docs.example.com;

    # Status dashboard
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # Generated documentation
    location /docs/ {
        alias /data/html/;
        autoindex on;
        try_files $uri $uri/ =404;
    }
}
```

#### Dashboard Features

- **Dashboard** (`/`): Overview with build/doc success rates, latest run summary
- **Packages** (`/packages`): Searchable list of all packages with docs
- **Package Detail** (`/packages/{name}/{version}`): Version list and doc links
- **Runs** (`/runs`): History of all batch runs
- **Run Detail** (`/runs/{id}`): Statistics, failures, and log links
- **Logs** (`/runs/{id}/build/{pkg}`, `/runs/{id}/docs/{pkg}`): View build and doc logs

## Monitoring

### Run Logs

Each batch run creates a timestamped directory:

```
/data/cache/logs/runs/2026-02-04-120000/
├── summary.json      # Run statistics
├── build/            # Build logs by package
│   ├── base.0.16.0.log
│   └── core.0.16.0.log
└── docs/             # Doc generation logs
    ├── base.0.16.0.log
    └── core.0.16.0.log
```

The `latest` symlink always points to the most recent run:

```bash
cat /data/cache/logs/latest/summary.json
```

### summary.json Format

```json
{
  "run_id": "2026-02-04-120000",
  "start_time": "2026-02-04T12:00:00",
  "end_time": "2026-02-04T14:30:00",
  "duration_seconds": 9000,
  "targets_requested": 100,
  "solutions_found": 95,
  "build_success": 90,
  "build_failed": 5,
  "doc_success": 85,
  "doc_failed": 3,
  "doc_skipped": 2,
  "failures": [
    {"package": "broken-pkg.1.0.0", "error": "build exit code 2"},
    {"package": "bad-docs.2.0.0", "error": "doc: odoc error"}
  ]
}
```

### Checking Status

```bash
# Quick status
jq '.build_success, .build_failed, .doc_success, .doc_failed' \
  /data/cache/logs/latest/summary.json

# List failures
jq -r '.failures[] | "\(.package): \(.error)"' \
  /data/cache/logs/latest/summary.json

# Duration
jq '.duration_seconds / 60 | floor | "\(.)m"' \
  /data/cache/logs/latest/summary.json
```

### Disk Usage

Monitor cache growth:

```bash
du -sh /data/cache/debian-12-x86_64/
du -sh /data/html/
```

## Maintenance

### Cache Management

The cache grows over time. After each batch run, garbage collection automatically:

1. **Layer GC**: Deletes build/doc layers not referenced by current solutions
2. **Universe GC**: Deletes universe directories not referenced by any blessed package

GC runs automatically at the end of each batch. Special layers are preserved:
- `base` - Base OS image
- `solutions` - Solver cache
- `doc-driver-*` - Shared odoc driver
- `doc-odoc-*` - Per-OCaml-version odoc

### Manual Cache Cleanup

To force a complete rebuild:

```bash
# Remove all layers (keeps base)
rm -rf /data/cache/debian-12-x86_64/build-*
rm -rf /data/cache/debian-12-x86_64/doc-*

# Remove solution cache (forces re-solving)
rm -rf /data/cache/debian-12-x86_64/solutions/
```

### Updating opam-repository

```bash
cd /data/opam-repository
git fetch origin
git reset --hard origin/master
```

Solutions are cached by opam-repository commit hash, so updating automatically invalidates old solutions.

### Epoch Transitions

For major changes (new odoc version, URL scheme change), you may want a clean rebuild:

1. Create new html directory: `/data/html-new/`
2. Run full batch with `--html-output /data/html-new/`
3. Once complete, atomically swap: `mv /data/html /data/html-old && mv /data/html-new /data/html`
4. Remove old: `rm -rf /data/html-old`

## Troubleshooting

### Build Failures

Check the build log:

```bash
cat /data/cache/logs/latest/build/failing-pkg.1.0.0.log
```

Or check the layer directly:

```bash
cat /data/cache/debian-12-x86_64/build-*/build.log
```

### Doc Generation Failures

```bash
cat /data/cache/logs/latest/docs/failing-pkg.1.0.0.log
```

Common issues:
- Missing `.cmti` files (package doesn't install them)
- odoc bugs with certain code patterns
- Memory exhaustion on large packages

### Stale .new/.old Directories

If a run was interrupted, stale staging directories may exist:

```bash
find /data/html -name "*.new" -o -name "*.old"
```

These are automatically cleaned up at the start of each batch run.

### Permission Issues

day10 uses runc containers which require root. If you see permission errors:

```bash
# Check runc works
sudo runc --version

# Ensure cache directory is accessible
sudo chown -R root:root /data/cache
```

### Memory Issues

For large package sets, you may need to limit parallelism:

```bash
# Reduce fork count
day10 batch --fork 4 ...
```

Or increase system memory/swap.

## Architecture Notes

### How Layers Work

Each package build creates a layer using overlay filesystem:

```
build-{hash}/
├── fs/           # Filesystem overlay (installed files)
├── build.log     # Build output
└── layer.json    # Metadata (package, deps, status)
```

The hash is computed from the package and its dependencies, so unchanged packages reuse existing layers.

### Blessing

In batch mode, day10 computes "blessings" - which package version is canonical for each package name. Blessed packages go to `/html/p/`, non-blessed go to `/html/u/{universe}/`.

### Graceful Degradation

When doc generation fails:
1. New docs are written to a staging directory
2. On success: atomically swap staging → final
3. On failure: staging is discarded, old docs remain

This ensures the live site never shows broken docs.

## Getting Help

- Check logs in `/data/cache/logs/latest/`
- Review `summary.json` for failure details
- File issues at: https://github.com/mtelvers/ohc/issues