A write-ahead log (WAL) provides durable, crash-safe storage by writing all changes to a sequential log file before acknowledging them. This implementation is inspired by LevelDB's log format.

Features#

Append-only writes - Records are always appended, never modified
CRC32C checksums - Every record is protected by a checksum
Large record support - Records larger than 32KB are automatically split
Block alignment - Efficient 32KB block-based I/O
Eio-native - Built on Eio for modern OCaml concurrency

Installation#

opam install wal

Usage#

Writing records#

Eio_main.run @@ fun env ->
let fs = Eio.Stdenv.fs env in
Eio.Switch.run @@ fun sw ->
let log = Wal.create ~sw fs (fs / "mylog.wal") in
Wal.append log "key1=value1";
Wal.append log "key2=value2";
Wal.sync log;  (* Durably persist *)
Wal.close log

Reading records#

let records = Wal.read_all fs (fs / "mylog.wal") in
List.iter (fun data -> process data) records

Folding over records#

let sum = Wal.fold path ~init:0 ~f:(fun acc data ->
  acc + int_of_string data
) in

Record Format#

WAL entries are stored as records with the following format:

+----------+----------+----------+--- ... ---+
| CRC (4B) | Len (2B) | Type (1B)| Data      |
+----------+----------+----------+--- ... ---+

CRC: CRC32C checksum of Type + Data
Len: Length of Data in bytes (max 65535)
Type: Record type (Full, First, Middle, Last)
Data: Payload bytes

Large records that exceed the block size (32KB) are split across multiple physical records using First/Middle/Last types.

API#

Wal.create - Create or open a WAL file
Wal.append - Append a record to the log
Wal.sync - Flush to disk (fsync)
Wal.close - Close the log
Wal.read_all - Read all records from a log
Wal.fold - Fold over records efficiently

LevelDB Log Format - The format this implementation is based on
RocksDB WAL - Similar WAL in RocksDB
irmin-pack - Irmin's pack file store uses similar concepts

Licence#

MIT License. See LICENSE.md for details.

Clone this repository

wal#

Overview#