# Comprehensive IMAP Implementation Plan

This document consolidates all RFC implementation plans from `spec/` and `lib/imap/PLAN.md` into a single, prioritized implementation roadmap. Per the design goals, we favor OCaml variants over strings and do not require backwards compatibility.

## Executive Summary

The ocaml-imap library implements IMAP4rev2 (RFC 9051) with several extensions. This plan covers:
- **P0**: Critical fixes and core infrastructure
- **P1**: Core protocol compliance
- **P2**: Extension support (SORT/THREAD, QUOTA, etc.)
- **P3**: Advanced features (UTF-8, CONDSTORE/QRESYNC)
- **P4**: Polish (documentation, unified flag library)

---

## Phase 0: Critical Fixes (P0)

These are blocking issues that need immediate attention.

### 0.1 Fix SEARCH Response Parsing (Client Library)

**Source**: `lib/imap/PLAN.md` - P0 Broken Functionality

**Problem**: `search` function always returns empty list - response is never parsed.

**Files**:
- `lib/imap/read.ml` - Add SEARCH response parsing
- `lib/imap/client.ml:536-544` - Fix to read response

**Implementation**:
```ocaml
(* In read.ml - add case for SEARCH response *)
| "SEARCH" ->
    let rec parse_numbers acc =
      match R.peek_char r with
      | Some ' ' -> sp r; parse_numbers (number r :: acc)
      | Some c when c >= '0' && c <= '9' -> parse_numbers (number r :: acc)
      | _ -> List.rev acc
    in
    let nums = parse_numbers [] in
    crlf r;
    Response.Search nums
```

**Tests** (`test/test_read.ml`):
```ocaml
let test_search_response () =
  let resp = parse "* SEARCH 2 4 7 11\r\n" in
  Alcotest.(check (list int)) "search" [2; 4; 7; 11]
    (match resp with Response.Search nums -> nums | _ -> [])

let test_search_empty () =
  let resp = parse "* SEARCH\r\n" in
  Alcotest.(check (list int)) "empty search" []
    (match resp with Response.Search nums -> nums | _ -> [-1])
```

### 0.2 Parse BODY/BODYSTRUCTURE Responses

**Source**: `lib/imap/PLAN.md` - P1 Incomplete Core Features

**Problem**: FETCH responses with BODY/BODYSTRUCTURE fall back to empty flags.

**Files**:
- `lib/imap/read.ml:284-302` - Add BODY/BODYSTRUCTURE parsing
- `lib/imap/body.ml` - Body structure types (may need new file)

**Implementation**: Parse nested multipart MIME structures recursively.

**Tests**:
```ocaml
let test_body_structure () =
  let resp = parse {|* 1 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("CHARSET" "UTF-8") NIL NIL "7BIT" 1234 56))|} in
  (* verify body structure parsed correctly *)
```

### 0.3 Parse BODY[section] Literal Responses

**Source**: `lib/imap/PLAN.md` - P1

**Problem**: Cannot read actual message content from FETCH.

**Implementation**: Parse section specifiers and literal data:
```ocaml
(* Patterns: BODY[HEADER], BODY[TEXT], BODY[1.2.MIME], BODY[section]<origin> *)
```

---

## Phase 1: Core Protocol Compliance (P1)

### 1.1 Complete ESEARCH Support (RFC 4731)

**Source**: `spec/PLAN-rfc4731.md`

**Current State**: Response type exists, parsing not implemented.

**Tasks**:
1. Add ESEARCH response parsing to `lib/imap/read.ml`
2. Add search return options to Command type
3. Add serialization for `RETURN (MIN MAX COUNT ALL)`
4. Add client API functions

**Types** (use variants, no strings):
```ocaml
type search_return_opt =
  | Return_min
  | Return_max
  | Return_all
  | Return_count

type esearch_result =
  | Esearch_min of int
  | Esearch_max of int
  | Esearch_count of int
  | Esearch_all of Seq.t
```

**Tests**:
```ocaml
let test_esearch_parsing () =
  let resp = parse "* ESEARCH (TAG \"A282\") MIN 2 COUNT 3\r\n" in
  assert (resp = Response.Esearch {
    tag = Some "A282";
    uid = false;
    results = [Esearch_min 2; Esearch_count 3]
  })
```

### 1.2 Parse APPENDUID/COPYUID Response Codes

**Source**: `lib/imap/PLAN.md` - P1

**Files**: `lib/imap/read.ml:169-228`

**Implementation**:
```ocaml
(* Add to response_code parsing *)
| "APPENDUID" ->
    sp r;
    let uidvalidity = number32 r in
    sp r;
    let uid = number32 r in
    Code.Appenduid (uidvalidity, uid)
| "COPYUID" ->
    sp r;
    let uidvalidity = number32 r in
    sp r;
    let source_uids = parse_uid_set r in
    sp r;
    let dest_uids = parse_uid_set r in
    Code.Copyuid (uidvalidity, source_uids, dest_uids)
```

### 1.3 UNSELECT Capability Advertisement

**Source**: `spec/PLAN-rfc3691.md`

**Status**: Fully implemented except capability not advertised.

**Fix** (`lib/imapd/server.ml`):
```ocaml
let base_capabilities_pre_tls = [
  (* existing *)
  "UNSELECT";  (* RFC 3691 - already implemented *)
]
```

### 1.4 SPECIAL-USE Support (RFC 6154)

**Source**: `spec/PLAN-rfc6154.md`

**Current**: Types exist, capability not advertised, flags not returned.

**Tasks**:
1. Add `SPECIAL-USE` to capabilities
2. Return special-use flags in LIST responses
3. Map standard mailbox names to attributes

**Types** (already exist, ensure completeness):
```ocaml
type special_use =
  | All | Archive | Drafts | Flagged | Important
  | Junk | Sent | Trash
  | Snoozed | Scheduled | Memos  (* draft-ietf-mailmaint *)
```

**Tests**:
```ocaml
let test_list_special_use () =
  (* LIST "" "*" should return \Drafts on Drafts mailbox *)
```

---

## Phase 2: Extension Support (P2)

### 2.1 SORT/THREAD Extension (RFC 5256)

**Source**: `spec/PLAN-rfc5256.md`

**Scope**: Large feature - server-side sorting and threading.

#### 2.1.1 Thread Module Types

**New file**: `lib/imap/thread.ml`

```ocaml
type algorithm =
  | Orderedsubject  (** Group by subject, sort by date *)
  | References      (** Full JWZ threading algorithm *)
  | Extension of string

type 'a node =
  | Message of 'a * 'a node list
  | Dummy of 'a node list

type 'a t = 'a node list
```

#### 2.1.2 Base Subject Extraction

**New file**: `lib/imap/subject.ml`

Implements RFC 5256 Section 2.1 algorithm:
1. Decode RFC 2047 encoded-words
2. Remove `Re:`, `Fw:`, `Fwd:` prefixes
3. Remove `[blob]` prefixes
4. Remove `(fwd)` trailers
5. Unwrap `[fwd: ...]` wrappers

```ocaml
val base_subject : string -> string
val is_reply_or_forward : string -> bool
```

#### 2.1.3 Sent Date Handling

**New file**: `lib/imap/date.ml`

```ocaml
type t

val of_header : string -> t option
val of_internaldate : string -> t
val sent_date : date_header:string option -> internaldate:string -> t
val compare : t -> t -> int
```

#### 2.1.4 Server-Side SORT Handler

**File**: `lib/imapd/server.ml`

1. Implement sort key extraction
2. Implement comparison by criteria
3. Return SORT response

#### 2.1.5 Threading Algorithms

**New file**: `lib/imapd/thread.ml`

1. `orderedsubject` - simple subject-based grouping
2. `references` - full JWZ algorithm (6 steps)

**Tests**:
```ocaml
let test_base_subject () =
  assert (Subject.base_subject "Re: test" = "test");
  assert (Subject.base_subject "Re: Re: test" = "test");
  assert (Subject.base_subject "[PATCH] Re: [ocaml] test" = "test");
  assert (Subject.base_subject "[fwd: wrapped]" = "wrapped")

let test_orderedsubject () =
  (* Test grouping by subject *)

let test_references_threading () =
  (* Test parent/child relationships *)
```

### 2.2 QUOTA Extension (RFC 9208)

**Source**: `spec/PLAN-rfc9208.md`

#### 2.2.1 Protocol Types

**File**: `lib/imapd/protocol.ml`

```ocaml
type quota_resource =
  | Quota_storage           (** KB of storage *)
  | Quota_message           (** Number of messages *)
  | Quota_mailbox           (** Number of mailboxes *)
  | Quota_annotation_storage

type quota_resource_info = {
  resource : quota_resource;
  usage : int64;
  limit : int64;
}

(* Commands *)
| Getquota of string
| Getquotaroot of mailbox_name
| Setquota of { root : string; limits : (quota_resource * int64) list }

(* Responses *)
| Quota_response of { root : string; resources : quota_resource_info list }
| Quotaroot_response of { mailbox : mailbox_name; roots : string list }
```

#### 2.2.2 Storage Backend Interface

**File**: `lib/imapd/storage.mli`

```ocaml
val get_quota_roots : t -> username:string -> mailbox_name -> string list
val get_quota : t -> username:string -> string -> (quota_resource_info list, error) result
val set_quota : t -> username:string -> string -> (quota_resource * int64) list -> (quota_resource_info list, error) result
val check_quota : t -> username:string -> mailbox_name -> additional_size:int64 -> bool
```

#### 2.2.3 Server Handlers

Implement `handle_getquota`, `handle_getquotaroot`, `handle_setquota`.

Add quota checks to APPEND/COPY/MOVE:
```ocaml
if not (Storage.check_quota ...) then
  send_response flow (No { code = Some Code_overquota; ... })
```

**Tests**:
```ocaml
let test_getquotaroot () =
  (* GETQUOTAROOT INBOX returns quota info *)

let test_quota_exceeded () =
  (* APPEND fails with OVERQUOTA when over limit *)
```

### 2.3 LIST-EXTENDED (RFC 5258)

**Source**: `spec/PLAN-rfc5258.md`

**Types**:
```ocaml
type list_select_option =
  | List_select_subscribed
  | List_select_remote
  | List_select_recursivematch
  | List_select_special_use  (* RFC 6154 *)

type list_return_option =
  | List_return_subscribed
  | List_return_children
  | List_return_special_use

type list_extended_item =
  | Childinfo of string list

type list_command =
  | List_basic of { reference : string; pattern : string }
  | List_extended of {
      selection : list_select_option list;
      reference : string;
      patterns : string list;
      return_opts : list_return_option list;
    }
```

**Tasks**:
1. Update grammar for extended LIST syntax
2. Add `\NonExistent` and `\Remote` attributes
3. Implement subscription tracking in storage
4. Handle RECURSIVEMATCH with CHILDINFO
5. Add `LIST-EXTENDED` capability

---

## Phase 3: Advanced Features (P3)

### 3.1 UTF-8 Support (RFC 6855)

**Source**: `spec/PLAN-rfc6855.md`

#### 3.1.1 Session State Tracking

```ocaml
type session_state = {
  utf8_enabled : bool;
  (* ... *)
}
```

#### 3.1.2 UTF-8 Validation

**New file**: `lib/imapd/utf8.ml`

```ocaml
val is_valid_utf8 : string -> bool
val has_non_ascii : string -> bool
val is_valid_utf8_mailbox_name : string -> bool
```

#### 3.1.3 ENABLE Handler Update

Track UTF8=ACCEPT state, reject SEARCH with CHARSET after enable.

#### 3.1.4 UTF8 APPEND Extension

Parse `UTF8 (literal)` syntax for 8-bit headers.

**Tests**:
```ocaml
let test_utf8_validation () =
  assert (Utf8.is_valid_utf8 "Hello");
  assert (Utf8.is_valid_utf8 "\xe4\xb8\xad\xe6\x96\x87");
  assert (not (Utf8.is_valid_utf8 "\xff\xfe"))
```

### 3.2 CONDSTORE/QRESYNC (RFC 7162)

**Source**: `lib/imap/PLAN.md` - P2, `PLAN.md` - Phase 2.2

#### 3.2.1 CONDSTORE Types

```ocaml
(* Fetch items *)
| Modseq
| Item_modseq of int64

(* Response codes *)
| Highestmodseq of int64
| Nomodseq
| Modified of Seq.t

(* Command modifiers *)
type fetch_modifier = { changedsince : int64 option }
type store_modifier = { unchangedsince : int64 option }
```

#### 3.2.2 Storage Backend

Add `modseq` to message type and mailbox state:
```ocaml
type message = {
  (* existing *)
  modseq : int64;
}

type mailbox_state = {
  (* existing *)
  highestmodseq : int64;
}
```

#### 3.2.3 QRESYNC

```ocaml
type qresync_params = {
  uidvalidity : int32;
  modseq : int64;
  known_uids : Seq.t option;
  seq_match : (Seq.t * Seq.t) option;
}

(* Response *)
| Vanished of { earlier : bool; uids : Seq.t }
```

---

## Phase 4: Polish and Infrastructure (P4)

### 4.1 RFC 5530 Response Code Documentation

**Source**: `spec/PLAN-rfc5530.md`

All 16 response codes already implemented. Add OCamldoc citations.

### 4.2 Unified Mail Flag Library

**Source**: `spec/PLAN-unified-mail-flag.md`

Create shared `mail-flag` library for IMAP/JMAP:

```
mail-flag/
├── keyword.ml      # Message keywords (typed variants)
├── system_flag.ml  # IMAP \Seen, \Deleted, etc.
├── mailbox_attr.ml # Mailbox attributes/roles
├── flag_color.ml   # Apple Mail flag colors
├── imap_wire.ml    # IMAP serialization
└── jmap_wire.ml    # JMAP serialization
```

### 4.3 Infrastructure Improvements

**Source**: `PLAN.md` - Phase 1

1. **Replace Menhir with Eio.Buf_read** - Pure functional parser
2. **Integrate conpool** - Connection pooling for client
3. **Add bytesrw streaming** - Large message handling
4. **Fuzz testing** - Parser robustness with Crowbar
5. **Eio mock testing** - Deterministic tests

---

## Testing Strategy

### Unit Tests

Each module should have corresponding tests in `test/`:

| Module | Test File | Coverage |
|--------|-----------|----------|
| `lib/imap/read.ml` | `test/test_read.ml` | Response parsing |
| `lib/imap/write.ml` | `test/test_write.ml` | Command serialization |
| `lib/imap/subject.ml` | `test/test_subject.ml` | Base subject extraction |
| `lib/imap/thread.ml` | `test/test_thread.ml` | Threading algorithms |
| `lib/imapd/server.ml` | `test/test_server.ml` | Command handlers |
| `lib/imapd/storage.ml` | `test/test_storage.ml` | Storage backends |

### Integration Tests

**File**: `test/integration/`

- Protocol compliance testing against real servers
- ImapTest compatibility suite
- Dovecot interoperability

### Fuzz Tests

**File**: `test/fuzz_parser.ml`

```ocaml
let fuzz_command_parser =
  Crowbar.(map [bytes] (fun input ->
    try
      ignore (Imap_parser.parse_command input);
      true
    with _ -> true  (* Parser should never crash *)
  ))
```

---

## Implementation Order

### Sprint 1: P0 Critical Fixes
1. [ ] Fix SEARCH response parsing
2. [ ] Parse BODY/BODYSTRUCTURE responses
3. [ ] Parse BODY[section] literals

### Sprint 2: P1 Core Compliance
4. [ ] Complete ESEARCH support
5. [ ] Parse APPENDUID/COPYUID response codes
6. [ ] Add UNSELECT to capabilities
7. [ ] Complete SPECIAL-USE support

### Sprint 3: P2 SORT/THREAD
8. [ ] Thread module types
9. [ ] Base subject extraction
10. [ ] Sent date handling
11. [ ] ORDEREDSUBJECT algorithm
12. [ ] REFERENCES algorithm
13. [ ] Server SORT/THREAD handlers

### Sprint 4: P2 QUOTA
14. [ ] Quota protocol types
15. [ ] Storage backend interface
16. [ ] Memory storage quota
17. [ ] Maildir storage quota
18. [ ] Server handlers

### Sprint 5: P2 LIST-EXTENDED
19. [ ] Extended LIST grammar
20. [ ] New attributes
21. [ ] Subscription tracking
22. [ ] RECURSIVEMATCH support

### Sprint 6: P3 UTF-8 & CONDSTORE
23. [ ] UTF-8 session state
24. [ ] UTF-8 validation
25. [ ] UTF8 APPEND extension
26. [ ] CONDSTORE types
27. [ ] CONDSTORE handlers
28. [ ] QRESYNC support

### Sprint 7: P4 Polish
29. [ ] Response code documentation
30. [ ] Unified mail flag library
31. [ ] Infrastructure improvements
32. [ ] Comprehensive test suite

---

## File Modification Summary

### New Files

| File | Purpose |
|------|---------|
| `lib/imap/thread.ml` | Thread types and parsing |
| `lib/imap/subject.ml` | Base subject extraction |
| `lib/imap/date.ml` | Sent date handling |
| `lib/imap/collation.ml` | Unicode collation |
| `lib/imap/mime.ml` | RFC 2047 decoding |
| `lib/imapd/thread.ml` | Threading algorithms |
| `lib/imapd/utf8.ml` | UTF-8 validation |
| `test/test_subject.ml` | Subject tests |
| `test/test_thread.ml` | Threading tests |
| `test/test_quota.ml` | Quota tests |
| `test/fuzz_parser.ml` | Fuzz tests |

### Modified Files

| File | Changes |
|------|---------|
| `lib/imap/read.ml` | SEARCH, ESEARCH, BODY parsing |
| `lib/imap/write.ml` | ESEARCH, THREAD serialization |
| `lib/imap/command.ml` | Return options, THREAD command |
| `lib/imap/response.ml` | ESEARCH, THREAD responses |
| `lib/imap/client.ml` | Fix search, add esearch/thread |
| `lib/imap/code.ml` | OCamldoc citations |
| `lib/imap/list_attr.ml` | Add NonExistent, Remote |
| `lib/imapd/protocol.ml` | Quota types, LIST-EXTENDED |
| `lib/imapd/server.ml` | Handlers, capabilities |
| `lib/imapd/storage.ml` | Quota ops, subscription tracking |
| `lib/imapd/grammar.mly` | Extended LIST, QUOTA, UTF8 |
| `lib/imapd/lexer.mll` | New tokens |
| `lib/imapd/parser.ml` | Response serialization |

---

## Design Principles

1. **Favor OCaml variants** - Use typed variants over strings where possible
2. **No backwards compatibility** - Clean API without legacy shims
3. **RFC citations** - OCamldoc links to RFC sections
4. **Incremental** - Each task is independently useful
5. **Test-driven** - Tests accompany each feature
6. **Eio-native** - Use Eio patterns throughout

---

## References

### Implemented RFCs
- RFC 9051 - IMAP4rev2 (core)
- RFC 8314 - Implicit TLS
- RFC 2177 - IDLE
- RFC 2342 - NAMESPACE
- RFC 2971 - ID
- RFC 4315 - UIDPLUS
- RFC 5161 - ENABLE
- RFC 6851 - MOVE
- RFC 7888 - LITERAL+

### RFCs in This Plan
- RFC 3691 - UNSELECT (partially complete)
- RFC 4731 - ESEARCH
- RFC 5256 - SORT/THREAD
- RFC 5258 - LIST-EXTENDED
- RFC 5530 - Response Codes (types complete)
- RFC 6154 - SPECIAL-USE (partially complete)
- RFC 6855 - UTF-8 Support
- RFC 7162 - CONDSTORE/QRESYNC
- RFC 9208 - QUOTA