anil.recoil.org
Matrix protocol in OCaml, Eio specialised
1# ocaml-matrix
2
3**Status: SUBSTANTIALLY COMPLETE**
4
5OCaml Matrix SDK providing protocol types and a full HTTP client for the Matrix chat protocol (spec v1.17). Uses `requests` for HTTP and `jsont` for JSON encoding/decoding. Provides an Eio-idiomatic wrapper for structured concurrency.
6
7## Specification
8
9The Matrix spec is vendored at `spec/matrix-spec` as a git submodule (currently tracking v1.17).
10
11See: https://github.com/matrix-org/matrix-spec
12
13## Client-Server API Feature Coverage
14
15The Matrix spec defines 41 feature modules for clients. Here is the implementation status for each:
16
17### Authentication & Session Management
18
19| Feature | Status | Module | Notes |
20|---------|--------|--------|-------|
21| Login (password) | ✅ Complete | `Auth` | Username/password with device settings |
22| Login (token/SSO) | ✅ Complete | `Auth` | Token-based login for SSO flows |
23| Token refresh | ✅ Complete | `Auth` | Refresh expired access tokens |
24| Logout | ✅ Complete | `Auth` | Single session or all sessions |
25| Registration | ✅ Complete | `Auth` | Guest or user account creation |
26| Whoami | ✅ Complete | `Auth` | Query current user/device |
27| UIAA | ✅ Complete | `Uiaa` | Password, recaptcha, email, dummy flows |
28| Soft logout | ⚠️ Partial | `Auth` | Token expiry detected, re-auth supported |
29| Account locking | ❌ Not implemented | - | Server-side feature |
30| OAuth 2.0 | ❌ Not implemented | - | Industry-standard auth (new in spec) |
31
32### Synchronization
33
34| Feature | Status | Module | Notes |
35|---------|--------|--------|-------|
36| `/sync` endpoint | ✅ Complete | `Sync` | Full parameter support |
37| Long-polling | ✅ Complete | `Sync` | Configurable timeout |
38| Incremental sync | ✅ Complete | `Sync` | next_batch token tracking |
39| Sync filters | ✅ Complete | `Sync` | Create, retrieve, apply filters |
40| Event filters | ✅ Complete | `Sync` | Type/sender filtering with limits |
41| Room event filters | ✅ Complete | `Sync` | Lazy load members, contains_url |
42| Sync loop | ✅ Complete | `Sync` | Callback-based continuous polling |
43| Sliding sync (MSC3575) | ⚠️ Stub | `Sliding_sync` | Data structures only |
44
45### Rooms & Membership
46
47| Feature | Status | Module | Notes |
48|---------|--------|--------|-------|
49| Room creation | ✅ Complete | `Rooms` | Name, topic, visibility, preset, type |
50| Join room (by ID) | ✅ Complete | `Rooms` | With server routing |
51| Join room (by alias) | ✅ Complete | `Rooms` | Alias resolution included |
52| Leave room | ✅ Complete | `Rooms` | Standard leave |
53| Forget room | ✅ Complete | `Rooms` | Remove from history |
54| Invite user | ✅ Complete | `Rooms` | By user ID |
55| Kick user | ✅ Complete | `Rooms` | With optional reason |
56| Ban/unban user | ✅ Complete | `Rooms` | With optional reason |
57| Get joined rooms | ✅ Complete | `Rooms` | List all joined rooms |
58| Get room members | ✅ Complete | `Rooms` | With membership state filtering |
59| Public room list | ✅ Complete | `Rooms` | Browse with pagination |
60| Power levels | ✅ Complete | `Rooms` | Get/set room and user permissions |
61| Knocking | ✅ Complete | `Event` | Join rule and membership type defined |
62| Restricted join rules | ✅ Complete | `Event` | Room-based access control |
63| Room upgrades | ⚠️ Partial | `Event` | Tombstone event type, no upgrade API |
64
65### Messaging
66
67| Feature | Status | Module | Notes |
68|---------|--------|--------|-------|
69| Text messages | ✅ Complete | `Messages` | Plain text with msgtype |
70| Emotes | ✅ Complete | `Messages` | m.emote support |
71| Notices | ✅ Complete | `Messages` | m.notice support |
72| HTML formatting | ✅ Complete | `Messages`, `Event` | format + formatted_body |
73| Image messages | ✅ Complete | `Messages`, `Event` | With media info |
74| File messages | ✅ Complete | `Messages`, `Event` | With media info |
75| Audio messages | ✅ Complete | `Messages`, `Event` | With media info |
76| Video messages | ✅ Complete | `Messages`, `Event` | With media info |
77| Location messages | ✅ Complete | `Event` | geo_uri with location info |
78| Stickers | ✅ Complete | `Event` | Sticker content type |
79| Redaction | ✅ Complete | `Messages` | With optional reason |
80| Get messages | ✅ Complete | `Messages` | Pagination forward/backward |
81| Get event context | ✅ Complete | `Messages` | Events before/after |
82| Rich replies | ✅ Complete | `Relations` | m.in_reply_to structure |
83| Threads | ✅ Complete | `Relations` | m.thread relation type |
84| Reactions | ✅ Complete | `Relations` | m.annotation with emoji key |
85| Edits | ✅ Complete | `Relations` | m.replace relation type |
86| Polls | ✅ Complete | `Event` | Start, response, end content types |
87
88### Room State
89
90| Feature | Status | Module | Notes |
91|---------|--------|--------|-------|
92| Get state events | ✅ Complete | `State` | By type and state key |
93| Set state events | ✅ Complete | `State` | Returns event ID |
94| Room name | ✅ Complete | `State`, `Event` | Get/set with content type |
95| Room topic | ✅ Complete | `State`, `Event` | Get/set with content type |
96| Room avatar | ✅ Complete | `State`, `Event` | Get/set with image info |
97| Room canonical alias | ✅ Complete | `Event` | Primary alias with alternates |
98| Join rules | ✅ Complete | `Event` | Public/invite/knock/restricted/private |
99| History visibility | ✅ Complete | `Event` | Invited/joined/shared/world_readable |
100| Guest access | ✅ Complete | `Event` | Can_join/forbidden |
101| Room encryption | ✅ Complete | `Event` | Algorithm and rotation settings |
102| Pinned events | ✅ Complete | `Event` | List of pinned event IDs |
103| Server ACLs | ✅ Complete | `Event` | Allow/deny rules |
104| Room creation event | ✅ Complete | `Event` | Creator, version, predecessor |
105
106### User Profile & Presence
107
108| Feature | Status | Module | Notes |
109|---------|--------|--------|-------|
110| Get profile | ✅ Complete | `Profile` | Display name and avatar |
111| Set display name | ✅ Complete | `Profile` | Update own name |
112| Set avatar | ✅ Complete | `Profile` | Update own avatar URL |
113| Get presence | ✅ Complete | `Presence` | Online/offline/unavailable |
114| Set presence | ✅ Complete | `Presence` | With status message |
115| Last active time | ✅ Complete | `Presence` | Activity timestamp |
116
117### Ephemeral Events
118
119| Feature | Status | Module | Notes |
120|---------|--------|--------|-------|
121| Typing notifications | ✅ Complete | `Typing` | Set typing state with timeout |
122| Read receipts | ✅ Complete | `Receipts` | Send with receipt type |
123| Read markers | ✅ Complete | `Receipts` | Set fully_read position |
124
125### End-to-End Encryption
126
127| Feature | Status | Module | Notes |
128|---------|--------|--------|-------|
129| Ed25519 key generation | ✅ Complete | `Keys` | Signing keys |
130| Curve25519 key generation | ✅ Complete | `Keys` | Key exchange |
131| One-time key generation | ✅ Complete | `Keys` | Batch generation with signing |
132| Fallback keys | ⚠️ Partial | `Keys` | Structure defined, upload untested |
133| Device key upload | ✅ Complete | `Keys` | With signatures |
134| Device key query | ✅ Complete | `Keys` | Fetch keys for users |
135| One-time key claim | ✅ Complete | `Keys` | Obtain keys for Olm setup |
136| Key change tracking | ✅ Complete | `Keys` | Between syncs |
137| Olm sessions | ✅ Complete | `Olm` | Create, store, retrieve |
138| X3DH key exchange | ✅ Complete | `Olm` | Triple Diffie-Hellman |
139| Double Ratchet | ✅ Complete | `Olm` | Per-session encryption |
140| Megolm sessions | ✅ Complete | `Olm` | Outbound/inbound group keys |
141| Message encryption | ✅ Complete | `Olm` | With authentication |
142| Message decryption | ✅ Complete | `Olm` | With verification |
143| Session pickling | ⚠️ Partial | `Olm` | Simplified JSON format |
144| Encrypted event type | ✅ Complete | `Event` | Olm and Megolm algorithms |
145
146### Device & Key Verification
147
148| Feature | Status | Module | Notes |
149|---------|--------|--------|-------|
150| List devices | ✅ Complete | `Devices` | All devices for user |
151| Get device info | ✅ Complete | `Devices` | IP and timestamp |
152| Update device | ✅ Complete | `Devices` | Display name |
153| Delete device | ✅ Complete | `Devices` | Single or batch |
154| Cross-signing keys | ✅ Complete | `Verification` | Master, self, user signing |
155| SAS verification | ⚠️ Partial | `Verification` | 20/64 emoji, no to-device |
156| QR verification | ⚠️ Partial | `Verification` | Structures only, no encode/decode |
157| Device trust | ✅ Complete | `Verification` | Trust state tracking |
158| Signature verification | ⚠️ Partial | `Verification` | Existence check only |
159| Verification events | ✅ Complete | `Event` | Ready, start, accept, key, mac, cancel, done |
160
161### Key Backup
162
163| Feature | Status | Module | Notes |
164|---------|--------|--------|-------|
165| Backup encryption | ✅ Complete | `Backup` | m.megolm_backup.v1.curve25519-aes-sha2 |
166| Backup decryption | ✅ Complete | `Backup` | AES-CTR with HKDF |
167| Recovery key generation | ⚠️ Partial | `Backup` | Uses base64, not base58 |
168| Recovery key parsing | ❌ Stub | `Backup` | Needs implementation |
169| Backup upload | ❌ Not implemented | - | No server API integration |
170| Backup download | ❌ Not implemented | - | No server API integration |
171| Backup version management | ❌ Not implemented | - | No server API integration |
172
173### Media
174
175| Feature | Status | Module | Notes |
176|---------|--------|--------|-------|
177| Upload media | ✅ Complete | `Media` | Binary with content type |
178| Download media | ✅ Complete | `Media` | By server/media ID |
179| Thumbnails | ✅ Complete | `Media` | Scaled/cropped versions |
180| MXC URI parsing | ✅ Complete | `Media` | Parse mxc:// URIs |
181| MXC to HTTP URL | ✅ Complete | `Media` | With thumbnail options |
182| Upload size limit | ✅ Complete | `Media` | Query configuration |
183| Encrypted media | ✅ Complete | `Event` | Encrypted file structure |
184
185### Account Management
186
187| Feature | Status | Module | Notes |
188|---------|--------|--------|-------|
189| Global account data | ✅ Complete | `Account` | Get/set arbitrary data |
190| Room account data | ✅ Complete | `Account` | Per-room data |
191| Third-party IDs | ✅ Complete | `Account` | Query, add, delete 3PIDs |
192| Email verification | ✅ Complete | `Account` | Request tokens |
193| Phone verification | ✅ Complete | `Account` | Request tokens |
194| Password change | ✅ Complete | `Account` | With device logout option |
195| Account deactivation | ✅ Complete | `Account` | Irreversible with erasure |
196| Ignored users | ✅ Complete | `Account` | Maintain ignore list |
197
198### Room Directory
199
200| Feature | Status | Module | Notes |
201|---------|--------|--------|-------|
202| Alias resolution | ✅ Complete | `Directory` | Alias to room ID |
203| Create alias | ✅ Complete | `Directory` | New room alias |
204| Delete alias | ✅ Complete | `Directory` | Remove alias |
205| Room visibility | ✅ Complete | `Directory` | Get/set directory visibility |
206| Public room search | ✅ Complete | `Directory` | With term and type filtering |
207
208### Spaces
209
210| Feature | Status | Module | Notes |
211|---------|--------|--------|-------|
212| Space child events | ✅ Complete | `Event` | Via servers, order, suggested |
213| Space parent events | ✅ Complete | `Event` | Via servers, canonical |
214| Space hierarchy | ✅ Complete | `Spaces` | Navigate structure |
215| Space management | ✅ Complete | `Spaces` | Parent/child relationships |
216
217### Voice/Video Calling
218
219| Feature | Status | Module | Notes |
220|---------|--------|--------|-------|
221| Call invite | ✅ Complete | `Event`, `Calls` | SDP offer, lifetime |
222| Call answer | ✅ Complete | `Event`, `Calls` | SDP answer |
223| Call hangup | ✅ Complete | `Event`, `Calls` | With reason codes |
224| ICE candidates | ✅ Complete | `Event`, `Calls` | Candidate exchange |
225| Call member | ✅ Complete | `Event` | LiveKit support, foci_active |
226
227### Push Notifications
228
229| Feature | Status | Module | Notes |
230|---------|--------|--------|-------|
231| Get push rules | ✅ Complete | `Push` | User notification rules |
232| Set push rules | ✅ Complete | `Push` | Update rules |
233| Enable/disable rules | ✅ Complete | `Push` | Toggle specific rules |
234| Delete rules | ✅ Complete | `Push` | Remove rules |
235| Push gateway | ❌ Not implemented | - | Separate API |
236
237### Additional Features
238
239| Feature | Status | Module | Notes |
240|---------|--------|--------|-------|
241| Room preview | ✅ Complete | `Room_preview` | Info before joining |
242| Room tagging | ⚠️ Partial | `Account` | Via account data |
243| Server notices | ❌ Not implemented | - | Server-side feature |
244| OpenID | ❌ Not implemented | - | Token provisioning |
245| Third-party invites | ❌ Not implemented | - | Email/phone invites |
246| Third-party networks | ❌ Not implemented | - | Bridge support |
247| Server-side search | ❌ Not implemented | - | Full-text search |
248| Reporting content | ❌ Not implemented | - | Event flagging |
249| Moderation policy lists | ✅ Complete | `Event` | Policy rule content type |
250| Server ACLs | ✅ Complete | `Event` | Allow/deny in state |
251| Secret storage | ❌ Not implemented | - | m.secret_storage.v1 |
252| Recently used emoji | ❌ Not implemented | - | Client-side typically |
253
254## Protocol Types Coverage
255
256### Identifiers (Matrix_id)
257
258| Type | Status | Notes |
259|------|--------|-------|
260| Server_name | ✅ Complete | Hostname validation |
261| User_id | ✅ Complete | @localpart:server format |
262| Room_id | ✅ Complete | !opaque_id:server format |
263| Event_id | ✅ Complete | v1-3 and v4 formats |
264| Room_alias | ✅ Complete | #alias:server format |
265| Device_id | ✅ Complete | Opaque string |
266| Session_id | ✅ Complete | Megolm session IDs |
267| Transaction_id | ✅ Complete | With generate() |
268
269### Event Content Types (Matrix_event)
270
271| Category | Types | Status |
272|----------|-------|--------|
273| Room state | 16 types | ✅ Complete |
274| Space events | 2 types | ✅ Complete |
275| Call events | 5 types | ✅ Complete |
276| Verification events | 7 types | ✅ Complete |
277| Message events | 8 types | ✅ Complete |
278| Encrypted content | 1 type | ✅ Complete |
279| Relations | 3 types | ✅ Complete |
280| Location/Beacons | 2 types | ✅ Complete |
281| Policy rules | 1 type | ✅ Complete |
282
283### Sync Response (Matrix_sync)
284
285| Component | Status | Notes |
286|-----------|--------|-------|
287| Timeline | ✅ Complete | Events, limited, pagination |
288| Ephemeral | ✅ Complete | Typing, receipts |
289| Account data | ✅ Complete | Per-room and global |
290| Room state | ✅ Complete | State events |
291| Joined rooms | ✅ Complete | Full room state |
292| Invited rooms | ✅ Complete | Invite state |
293| Left rooms | ✅ Complete | Leave state |
294| Knocked rooms | ✅ Complete | Knock state |
295| Device lists | ✅ Complete | Changed/left tracking |
296| To-device | ✅ Complete | Direct messages |
297| Presence | ✅ Complete | User presence |
298
299## Client Architecture
300
301### Libraries
302
303| Library | Purpose | Status |
304|---------|---------|--------|
305| `matrix_proto` | Protocol types with jsont codecs | ✅ Complete |
306| `matrix_client` | HTTP client (Result-based) | ✅ Complete |
307| `matrix_eio` | Eio wrapper (exception-based) | ✅ Complete |
308
309### Client Modules
310
311All 23 modules have Eio wrappers:
312
313| Module | Lines | Status | Notes |
314|--------|-------|--------|-------|
315| Auth | 304 | ✅ Complete | All login flows |
316| Sync | 246 | ✅ Complete | Sync loop with callbacks |
317| Rooms | 461 | ✅ Complete | Full room management |
318| Messages | 232 | ✅ Complete | All message types |
319| State | 139 | ✅ Complete | State get/set |
320| Keys | 447 | ✅ Complete | E2EE key management |
321| Olm | 895 | ✅ Complete | Session management |
322| Verification | 535 | ⚠️ Partial | SAS incomplete |
323| Backup | 466 | ⚠️ Partial | No server API |
324| Media | 118 | ✅ Complete | Upload/download |
325| Profile | 108 | ✅ Complete | Name/avatar |
326| Presence | 72 | ✅ Complete | Status management |
327| Typing | 31 | ✅ Complete | Typing indicators |
328| Receipts | 39 | ✅ Complete | Read receipts |
329| Devices | 89 | ✅ Complete | Device management |
330| Account | 329 | ✅ Complete | Account data, 3PIDs |
331| Directory | 182 | ✅ Complete | Aliases, search |
332| Relations | 330 | ✅ Complete | Reactions, edits, threads |
333| Uiaa | 435 | ✅ Complete | Multi-stage auth |
334| Timeline | 423 | ⚠️ Partial | TODO: trimming |
335| Store | 1001 | ⚠️ Partial | In-memory only |
336| Send_queue | 466 | ⚠️ Partial | No HTTP integration |
337| Push | 368 | ✅ Complete | Push rules |
338| Spaces | 212 | ✅ Complete | Hierarchy |
339| Calls | 193 | ✅ Complete | VoIP events |
340| Room_preview | 257 | ✅ Complete | Preview info |
341| Sliding_sync | 500 | ❌ Stub | MSC3575 structures only |
342
343## Known Limitations
344
345### Verification Module
346- SAS emoji table has 20 entries (spec requires 64)
347- `verify_cross_signing_signature` checks existence, not cryptographic validity
348- QR code verification has structures but no encoding/decoding
349- No to-device message sending for verification flow
350
351### Backup Module
352- Recovery key uses base64 encoding instead of base58
353- `parse_recovered_key` is a stub
354- No server API calls for backup upload/download/version management
355
356### Send_queue Module
357- Provides queue management but no HTTP integration
358- Caller must implement `send_fn` for actual sending
359- No automatic persistence
360
361### Store Module
362- All stores (STATE_STORE, CRYPTO_STORE, EVENT_CACHE_STORE) are in-memory only
363- Pluggable architecture defined but only memory backends exist
364- No SQLite or file-based implementations
365
366### Timeline Module
367- LinkedChunk `push_back` has TODO for trimming when over `max_events`
368- LRU eviction removes oldest room without access-time preservation
369
370### Olm Module
371- Session pickle/unpickle uses simplified JSON format
372- Needs real-world validation
373- No session recovery from corrupted state
374
375## Test Coverage
376
377| Area | Tests | Status |
378|------|-------|--------|
379| Protocol types | 39 roundtrip tests | ✅ Complete |
380| Sync response | Fixture-based | ✅ Complete |
381| Integration tests | None | ❌ Needed |
382
383## Dependencies
384
385- `jsont` / `jsont.bytesrw` - JSON encoding/decoding
386- `requests` - HTTP client
387- `eio` - Async I/O with structured concurrency
388- `uri` - URI handling
389- `ptime` - Timestamps
390- `base64` - Base64 encoding
391- `mirage-crypto` - AES-CBC encryption
392- `mirage-crypto-ec` - Ed25519/X25519 cryptography
393- `mirage-crypto-rng` - Secure random number generation
394- `digestif` - SHA-256 hashing
395- `kdf.hkdf` - HKDF key derivation
396
397## Room Versions
398
399The implementation supports room version events but does not enforce version-specific behavior:
400
401| Version | Key Feature | Event Support |
402|---------|-------------|---------------|
403| v1-v6 | Core messaging | ✅ |
404| v7 | Knocking | ✅ |
405| v8 | Restricted joins | ✅ |
406| v9 | Redaction fixes | ✅ |
407| v10 | Integer power levels | ✅ |
408| v11 | Redaction clarification | ✅ |
409| v12 | Creator infinite power | ✅ |
410
411## Summary
412
413**Overall completion: ~85%**
414
415- **Core features**: 95% complete (auth, sync, rooms, messages, state, presence)
416- **E2EE primitives**: 90% complete (keys, Olm, Megolm - verification partial)
417- **Advanced features**: 70% complete (backup, sliding sync, secret storage incomplete)
418- **Infrastructure**: 80% complete (storage in-memory only, no integration tests)
419
420The SDK is suitable for building Matrix clients with basic E2EE support. Full production use requires completing verification flows, backup server integration, and persistent storage backends.