IMAP Library Improvement Plan#

This document outlines opportunities for improving the ocaml-imapd library based on:

RFC 9051 (IMAP4rev2) and extension RFC compliance review
Available libraries in the monorepo
Modern OCaml patterns with Eio

Current State#

The library implements a functional IMAP4rev2 server with:

Core protocol (RFC 9051): CAPABILITY, LOGIN, SELECT, FETCH, STORE, SEARCH, etc.
Extensions: IDLE (RFC 2177), NAMESPACE (RFC 2342), ID (RFC 2971), UIDPLUS (RFC 4315), ENABLE (RFC 5161), MOVE (RFC 6851), LITERAL+ (RFC 7888)
TLS: STARTTLS and implicit TLS (RFC 8314)
Storage: Memory and Maildir backends
Auth: PAM authentication

Phase 1: Core Infrastructure Improvements#

1.1 Replace Faraday/Menhir with Eio Buf_read/Buf_write#

Priority: High Effort: Medium Dependencies: None

The current parser uses Menhir (lexer/parser generator) and Faraday (serializer). Port to pure Eio buffered I/O for consistency and fewer dependencies.

(* Current: Menhir + Faraday *)
let parse_command input =
  let lexbuf = Lexing.from_string input in
  Imap_grammar.command Imap_lexer.token lexbuf

(* Proposed: Eio.Buf_read combinators *)
module Parser = struct
  open Eio.Buf_read.Syntax

  let atom = take_while1 (fun c -> c > ' ' && not (String.contains "(){}\"\\]" c))
  let quoted_string = char '"' *> take_while ((<>) '"') <* char '"'
  let literal = char '{' *> uint <* string "}\r\n" >>= fun n -> take n
  let astring = atom <|> quoted_string

  let command =
    let* tag = atom <* char ' ' in
    let* cmd = atom in
    (* ... *)
end

Files to modify:

lib/imap_parser/imap_parser.ml - Replace with Buf_read combinators
lib/imap_parser/dune - Remove menhir, faraday dependencies
bin/dune - Update dependencies

1.2 Integrate conpool for Client Connection Pooling#

Priority: High Effort: Low Dependencies: Phase 1.1 (for client library)

The monorepo has conpool - a production-ready connection pool. Use it for:

IMAP client library (for connecting to upstream servers)
Potential proxy/relay functionality

(* Client with connection pooling *)
module Imap_client = struct
  type t = {
    pool : Conpool.t;
    endpoint : Conpool.Endpoint.t;
  }

  let create ~sw ~net ~clock ~host ~port ~tls =
    let endpoint = Conpool.Endpoint.make ~host ~port in
    let pool = Conpool.create_basic ~sw ~net ~clock ~tls () in
    { pool; endpoint }

  let with_connection t f =
    Conpool.with_connection t.pool t.endpoint f
end

New files:

lib/imap_client/imap_client.ml - Client library using conpool
lib/imap_client/imap_client.mli - Client interface

1.3 Add bytesrw-eio Integration for Streaming#

Priority: Medium Effort: Medium Dependencies: None

Use bytesrw-eio for efficient streaming of large message bodies:

(* Streaming FETCH for large messages *)
let fetch_body_streaming ~flow ~message ~section =
  let reader = Bytesrw_eio.bytes_reader_of_flow flow in
  (* Stream body parts without loading entire message into memory *)

Benefits:

Memory-efficient handling of large attachments
Consistent with other monorepo libraries (yamlrw, etc.)

Phase 2: Protocol Compliance & Features#

2.1 Complete SEARCH Implementation#

Priority: High Effort: Medium Dependencies: None

Current SEARCH is incomplete (many criteria return true). Implement full RFC 9051 Section 6.4.4:

(* Currently stubbed *)
| Search_body _ -> true  (* TODO *)
| Search_text _ -> true  (* TODO *)
| Search_header _ -> true  (* TODO *)
| Search_before _ -> true  (* TODO *)

(* Need to implement *)
| Search_body text ->
    Option.exists (fun body -> string_contains_ci body text) m.raw_body
| Search_header (name, value) ->
    parse_headers m.raw_headers
    |> List.exists (fun (n, v) ->
         String.equal_ci n name && string_contains_ci v value)
| Search_before date ->
    compare_dates m.internal_date date < 0

Search criteria to implement:

BODY, TEXT - Full-text search in body/headers
HEADER field value - Header field matching
BEFORE, ON, SINCE, SENTBEFORE, SENTON, SENTSINCE - Date comparisons
BCC, CC, FROM, SUBJECT, TO - Address/subject matching

2.2 Implement CONDSTORE/QRESYNC (RFC 7162)#

Priority: Medium Effort: High Dependencies: Storage backend changes

CONDSTORE enables efficient synchronization for disconnected clients:

type message = {
  (* existing fields *)
  modseq : int64;  (* Add modification sequence *)
}

type mailbox_state = {
  (* existing fields *)
  highestmodseq : int64;  (* Add highest modseq *)
}

(* New commands *)
| Select_condstore of mailbox_name
| Fetch_changedsince of { sequence : sequence_set; modseq : int64; items : fetch_item list }
| Store_unchangedsince of { sequence : sequence_set; modseq : int64; action : store_action; flags : flag list }

RFC 7162 features:

MODSEQ message attribute
HIGHESTMODSEQ mailbox attribute
CHANGEDSINCE FETCH modifier
UNCHANGEDSINCE STORE modifier
VANISHED response for QRESYNC

2.3 Add QUOTA Support (RFC 9208)#

Priority: Low Effort: Medium Dependencies: Storage backend changes

type quota_resource =
  | Storage  (* STORAGE - total size in KB *)
  | Message  (* MESSAGE - number of messages *)
  | Mailbox  (* MAILBOX - number of mailboxes *)

type quota = {
  root : string;
  resources : (quota_resource * int64 * int64) list;  (* resource, usage, limit *)
}

(* New commands *)
| Getquota of string
| Getquotaroot of mailbox_name
| Setquota of string * (quota_resource * int64) list

2.4 Implement NOTIFY (RFC 5465)#

Priority: Low Effort: High Dependencies: IDLE improvements

NOTIFY is an advanced version of IDLE that allows clients to specify exactly what events they want:

type notify_event =
  | Event_messagenew
  | Event_messageexpunge
  | Event_flagchange
  | Event_annotationchange
  | Event_mailboxname
  | Event_subscriptionchange

type notify_mailbox =
  | Notify_selected
  | Notify_inboxes
  | Notify_personal
  | Notify_subscribed
  | Notify_subtree of mailbox_name list
  | Notify_mailboxes of mailbox_name list

Phase 3: Integration with Monorepo Libraries#

3.1 Configuration with tomlt/yamlrw#

Priority: Medium Effort: Low Dependencies: None

Replace cmdliner-only config with file-based configuration:

# /etc/imapd/config.toml
[server]
hostname = "mail.example.com"
port = 993
tls = true

[storage]
backend = "maildir"
path = "/var/mail"

[auth]
method = "pam"
service = "imapd"

[limits]
max_connections = 1000
autologout_timeout = 1800

(* Use tomlt for config parsing *)
let config_codec =
  let open Tomlt in
  Obj.obj (fun hostname port -> { hostname; port; (* ... *) })
  |> Obj.mem "hostname" Jsont.string ~enc:(fun c -> c.hostname)
  |> Obj.mem "port" Jsont.int ~enc:(fun c -> c.port)
  |> Obj.finish

3.2 Add XDG Directory Support with xdge#

Priority: Medium Effort: Low Dependencies: None

Use xdge for standard directory handling:

(* Server state and config directories *)
let get_config_path ~env =
  let xdg = Xdge.create ~env () in
  Xdge.config_file xdg "imapd" "config.toml"

let get_state_path ~env =
  let xdg = Xdge.create ~env () in
  Xdge.state_dir xdg "imapd"

(* Client credential storage *)
let get_credentials_path ~env ~host =
  let xdg = Xdge.create ~env () in
  Xdge.config_file xdg "imap-client" (host ^ ".credentials")

Benefits:

Standard XDG Base Directory compliance
Works well with systemd and container deployments
Cmdliner integration for CLI overrides

Phase 4: Testing & Robustness#

4.1 Expand Test Coverage#

Priority: High Effort: Medium Dependencies: None

Current tests cover basic functionality. Add:

(* Fuzz testing for parser *)
let fuzz_command_parser =
  Crowbar.(map [bytes] (fun input ->
    try
      ignore (Imap_parser.parse_command input);
      true
    with _ -> true  (* Parser should never crash *)
  ))

(* Property-based testing *)
let test_uid_persistence =
  QCheck.Test.make ~name:"UIDs persist across sessions"
    (QCheck.list QCheck.string)
    (fun messages ->
      let storage = Memory_storage.create () in
      let uids1 = List.map (fun m -> append storage m) messages in
      let uids2 = List.map (fun m -> fetch_uid storage m) messages in
      uids1 = uids2)

Test categories to add:

Protocol edge cases (malformed commands, literals, etc.)
Concurrent access (multiple clients, same mailbox)
Large message handling (>100MB)
Connection lifecycle (timeouts, reconnects)

4.2 Add Eio Mock Testing#

Priority: Medium Effort: Low Dependencies: None

Use Eio_mock for deterministic testing:

let test_login () =
  Eio_mock.Backend.run @@ fun () ->
  let flow = Eio_mock.Flow.make "client" in
  Eio_mock.Flow.on_read flow [
    `Return "A001 LOGIN user pass\r\n"
  ];

  let server = create_test_server () in
  handle_connection server flow ();

  let output = Eio_mock.Flow.get_written flow in
  Alcotest.(check string) "greeting" "* OK " (String.sub output 0 5)

4.3 Stress Testing with conpool Patterns#

Priority: Medium Effort: Medium Dependencies: Phase 1.2

Adapt conpool's stress test patterns:

let stress_test_concurrent_fetch () =
  Eio_main.run @@ fun env ->
  let server = start_test_server env in
  let n_clients = 100 in
  let n_fetches = 1000 in

  Eio.Fiber.all (List.init n_clients (fun i ->
    fun () ->
      let client = connect_client env server in
      for _ = 1 to n_fetches do
        fetch_message client ~uid:(Random.int 10000)
      done
  ))

Phase 5: Documentation & Developer Experience#

5.1 Add .mld Tutorial Documentation#

Priority: Medium Effort: Low Dependencies: None

Create tutorial documentation with executable examples:

doc/
├── tutorial.mld          # Getting started
├── server_setup.mld      # Running the server
├── client_usage.mld      # Using the client
├── storage_backends.mld  # Storage customization
└── extending.mld         # Adding new commands

5.2 Add Logging with Logs#

Priority: Medium Effort: Low Dependencies: None

Consistent logging following conpool patterns:

let src = Logs.Src.create "imapd" ~doc:"IMAP4rev2 server"
module Log = (val Logs.src_log src : Logs.LOG)

let handle_command conn cmd =
  Log.debug (fun m -> m "Received command: %s" (command_to_string cmd));
  (* ... *)
  Log.info (fun m -> m "Command completed: %s -> %s" tag status)

Implementation Priority#

Phase	Item	Priority	Effort	Impact
1.1	Eio Buf_read parser	High	Medium	Reduces dependencies
1.2	conpool integration	High	Low	Enables client pooling
2.1	Complete SEARCH	High	Medium	RFC compliance
4.1	Expand tests	High	Medium	Reliability
3.1	Config files (tomlt)	Medium	Low	Usability
3.2	XDG directories (xdge)	Medium	Low	Standards compliance
1.3	bytesrw streaming	Medium	Medium	Performance
2.2	CONDSTORE/QRESYNC	Medium	High	Sync efficiency
4.2	Eio mock tests	Medium	Low	Test quality
5.1	Documentation	Medium	Low	Developer experience
5.2	Logging	Medium	Low	Observability
2.3	QUOTA support	Low	Medium	Feature completeness
2.4	NOTIFY	Low	High	Advanced feature

File Structure After Improvements#

ocaml-imapd/
├── lib/
│   ├── imap_types/          # Core types (unchanged)
│   ├── imap_parser/         # Eio Buf_read/Buf_write parser
│   ├── imap_storage/        # Storage backends
│   │   ├── memory.ml        # In-memory (testing)
│   │   └── maildir.ml       # Maildir format
│   ├── imap_auth/           # Authentication
│   ├── imap_server/         # Server implementation
│   ├── imap_client/         # NEW: Client library with conpool
│   └── imap_config/         # NEW: Configuration with tomlt/xdge
├── bin/
│   ├── main.ml              # Server binary
│   └── imap_client.ml       # Client binary
├── test/
│   ├── test_*.ml            # Unit tests
│   ├── fuzz_*.ml            # NEW: Fuzz tests
│   └── stress_*.ml          # NEW: Stress tests
├── doc/
│   └── *.mld                # NEW: Tutorial docs
├── spec/
│   └── rfc*.txt             # RFC specifications
└── PLAN.md                  # This file

Implemented:

RFC 9051 - IMAP4rev2 (core protocol)
RFC 8314 - Use of TLS for Email
RFC 2177 - IDLE
RFC 2342 - NAMESPACE
RFC 2971 - ID
RFC 4315 - UIDPLUS
RFC 5161 - ENABLE
RFC 6851 - MOVE
RFC 7888 - LITERAL+

To implement (in this plan):

RFC 7162 - CONDSTORE/QRESYNC (synchronization)
RFC 9208 - QUOTA (resource limits)
RFC 5465 - NOTIFY (advanced IDLE)

Future consideration:

RFC 6855 - UTF8=ACCEPT (internationalization)
RFC 5258 - LIST-EXTENDED (already partially supported)
RFC 4978 - COMPRESS=DEFLATE (compression)

IMAP Library Improvement Plan#

Current State#

Phase 1: Core Infrastructure Improvements#

1.1 Replace Faraday/Menhir with Eio Buf_read/Buf_write#

1.2 Integrate conpool for Client Connection Pooling#

1.3 Add bytesrw-eio Integration for Streaming#

Phase 2: Protocol Compliance & Features#

2.1 Complete SEARCH Implementation#

2.2 Implement CONDSTORE/QRESYNC (RFC 7162)#

2.3 Add QUOTA Support (RFC 9208)#

2.4 Implement NOTIFY (RFC 5465)#

Phase 3: Integration with Monorepo Libraries#

3.1 Configuration with tomlt/yamlrw#

3.2 Add XDG Directory Support with xdge#

Phase 4: Testing & Robustness#

4.1 Expand Test Coverage#

4.2 Add Eio Mock Testing#

4.3 Stress Testing with conpool Patterns#

Phase 5: Documentation & Developer Experience#

5.1 Add .mld Tutorial Documentation#

5.2 Add Logging with Logs#

Implementation Priority#

File Structure After Improvements#

Related RFCs#