anil.recoil.org
My aggregated monorepo of OCaml code, automaintained
1# HTTP/2 Implementation Plan for ocaml-requests
2
3## Overview
4
5This document outlines the plan for adding native HTTP/2 support to the ocaml-requests library, implementing RFC 9113 (HTTP/2), RFC 7541 (HPACK header compression), and integrating seamlessly with the existing Eio-based architecture.
6
7### Design Goals
8
91. **Eio-Native**: Full integration with Eio's structured concurrency primitives
102. **Zero-Copy Where Possible**: Use `Cstruct` and `Bigstringaf` for efficient buffer management
113. **Protocol Transparency**: Users should be able to use the same API for HTTP/1.1 and HTTP/2
124. **Connection Multiplexing**: True stream multiplexing within a single TCP connection
135. **Shared Types**: Maximize type sharing between HTTP/1.1 and HTTP/2 implementations
146. **Backwards Compatibility Not Required**: This is a fresh implementation
15
16## License Attribution
17
18When deriving code or patterns from the [ocaml-h2](https://github.com/anmonteiro/ocaml-h2) library, files MUST include the following license header:
19
20```ocaml
21(*---------------------------------------------------------------------------
22 Copyright (c) 2019 António Nuno Monteiro.
23 Portions Copyright (c) 2017 Inhabited Type LLC.
24 Copyright (c) 2025 Anil Madhavapeddy <anil@recoil.org>.
25
26 All rights reserved.
27
28 Redistribution and use in source and binary forms, with or without
29 modification, are permitted provided that the following conditions are met:
30
31 1. Redistributions of source code must retain the above copyright notice,
32 this list of conditions and the following disclaimer.
33
34 2. Redistributions in binary form must reproduce the above copyright notice,
35 this list of conditions and the following disclaimer in the documentation
36 and/or other materials provided with the distribution.
37
38 3. Neither the name of the copyright holder nor the names of its contributors
39 may be used to endorse or promote products derived from this software
40 without specific prior written permission.
41
42 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
43 AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
44 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
45 ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
46 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
47 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
48 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
49 INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
50 CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
51 ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
52 POSSIBILITY OF SUCH DAMAGE.
53 SPDX-License-Identifier: BSD-3-Clause
54 ---------------------------------------------------------------------------*)
55```
56
57Files NOT derived from h2 continue to use the ISC license header.
58
59## Specification References
60
61| RFC | Title | Status |
62|-----|-------|--------|
63| [RFC 9113](spec/rfc9113.txt) | HTTP/2 | Primary spec (obsoletes 7540) |
64| [RFC 7541](spec/rfc7541.txt) | HPACK: Header Compression for HTTP/2 | Required |
65| [RFC 9110](spec/rfc9110.txt) | HTTP Semantics | Shared with HTTP/1.1 |
66
67## Shared Types (HTTP/1.1 and HTTP/2)
68
69The requests library already provides protocol-agnostic types that will be reused for HTTP/2:
70
71| Module | Type | Description | Shared? |
72|--------|------|-------------|---------|
73| `Method` | `Method.t` | HTTP methods (GET, POST, etc.) | Yes - RFC 9110 |
74| `Status` | `Status.t` | HTTP status codes | Yes - RFC 9110 |
75| `Headers` | `Headers.t` | Header collection | Yes - with H2 pseudo-header support |
76| `Body` | `Body.t` | Request body construction | Yes |
77| `Response` | `Response.t` | Response representation | Yes |
78| `Uri` | `Uri.t` | URI parsing and manipulation | Yes - RFC 3986 |
79| `Mime` | `Mime.t` | MIME types | Yes |
80| `Error` | `Error.error` | Error types | Extended for H2 |
81| `Timeout` | `Timeout.t` | Timeout configuration | Yes |
82| `Retry` | `Retry.config` | Retry configuration | Yes |
83| `Auth` | `Auth.t` | Authentication | Yes |
84
85### HTTP/2-Specific Types (New)
86
87| Module | Type | Description |
88|--------|------|-------------|
89| `H2_frame` | `frame_type`, `frame` | Frame definitions per RFC 9113 §6 |
90| `H2_stream` | `stream_id`, `state` | Stream state machine per RFC 9113 §5.1 |
91| `H2_settings` | `settings`, `setting` | Connection settings per RFC 9113 §6.5 |
92| `H2_hpack` | `Encoder.t`, `Decoder.t` | HPACK compression per RFC 7541 |
93| `H2_error` | `error_code` | H2 error codes per RFC 9113 §7 |
94
95### Headers Module Extension
96
97The existing `Headers` module needs extension for HTTP/2 pseudo-headers:
98
99```ocaml
100(** HTTP/2 pseudo-headers per RFC 9113 §8.3 *)
101
102val get_pseudo : string -> t -> string option
103(** [get_pseudo name headers] retrieves a pseudo-header (without the colon prefix).
104 Example: [get_pseudo "method" headers] for [:method] *)
105
106val set_pseudo : string -> string -> t -> t
107(** [set_pseudo name value headers] sets a pseudo-header.
108 Pseudo-headers are placed before regular headers per RFC 9113 §8.3. *)
109
110val has_pseudo_headers : t -> bool
111(** [has_pseudo_headers headers] returns true if any pseudo-headers are present. *)
112
113val validate_h2_request : t -> (unit, string) result
114(** Validate headers for HTTP/2 request constraints per RFC 9113 §8.2.1 *)
115
116val validate_h2_response : t -> (unit, string) result
117(** Validate headers for HTTP/2 response constraints per RFC 9113 §8.2.2 *)
118```
119
120### Error Module Extension
121
122Extend `Error.error` with HTTP/2-specific variants:
123
124```ocaml
125(* Add to Error.error type *)
126
127(* HTTP/2 protocol errors *)
128| H2_protocol_error of { code: int; message: string }
129 (** HTTP/2 connection error per RFC 9113 §5.4.1 *)
130| H2_stream_error of { stream_id: int32; code: int; message: string }
131 (** HTTP/2 stream error per RFC 9113 §5.4.2 *)
132| H2_flow_control_error of { stream_id: int32 option }
133 (** Flow control window exceeded per RFC 9113 §5.2 *)
134| H2_compression_error of { message: string }
135 (** HPACK decompression failed per RFC 7541 *)
136| H2_settings_timeout
137 (** SETTINGS acknowledgment timeout per RFC 9113 §6.5.3 *)
138| H2_goaway of { last_stream_id: int32; code: int; debug: string }
139 (** Server sent GOAWAY per RFC 9113 §6.8 *)
140```
141
142## Architecture Overview
143
144### Module Structure
145
146```
147lib/
148├── # ══════════════════════════════════════════════════════════════
149├── # SHARED TYPES (Protocol-Agnostic, RFC 9110)
150├── # ══════════════════════════════════════════════════════════════
151├── method.ml[i] # HTTP methods (existing)
152├── status.ml[i] # Status codes (existing)
153├── headers.ml[i] # Headers - EXTENDED for H2 pseudo-headers
154├── body.ml[i] # Request body (existing)
155├── response.ml[i] # Response type (existing)
156├── uri.ml[i] # URI parsing (existing)
157├── error.ml[i] # Errors - EXTENDED for H2 errors
158├── auth.ml[i] # Authentication (existing)
159├── timeout.ml[i] # Timeout config (existing)
160├── retry.ml[i] # Retry config (existing)
161│
162├── # ══════════════════════════════════════════════════════════════
163├── # HTTP/1.1 IMPLEMENTATION (Existing)
164├── # ══════════════════════════════════════════════════════════════
165├── http_read.ml[i] # HTTP/1.1 response parsing
166├── http_write.ml[i] # HTTP/1.1 request serialization
167├── http_client.ml[i] # HTTP/1.1 client
168│
169├── # ══════════════════════════════════════════════════════════════
170├── # HTTP/2 IMPLEMENTATION (New - BSD-3-Clause where h2-derived)
171├── # ══════════════════════════════════════════════════════════════
172├── h2/
173│ ├── h2_frame.ml[i] # Frame types and serialization (RFC 9113 §4, §6)
174│ ├── h2_hpack.ml[i] # HPACK encoder/decoder (RFC 7541) - BSD-3-Clause
175│ ├── h2_hpack_static.ml # Static table data (RFC 7541 Appendix A)
176│ ├── h2_huffman.ml[i] # Huffman coding (RFC 7541 Appendix B) - BSD-3-Clause
177│ ├── h2_stream.ml[i] # Stream state machine (RFC 9113 §5.1) - BSD-3-Clause
178│ ├── h2_flow_control.ml[i] # Flow control windows (RFC 9113 §5.2)
179│ ├── h2_settings.ml[i] # Settings negotiation (RFC 9113 §6.5)
180│ ├── h2_connection.ml[i] # Connection lifecycle management
181│ └── h2_client.ml[i] # Client-side HTTP/2 implementation
182│
183├── # ══════════════════════════════════════════════════════════════
184├── # PROTOCOL ABSTRACTION LAYER (New)
185├── # ══════════════════════════════════════════════════════════════
186├── http_version.ml[i] # Version enum and ALPN identifiers
187├── connection.ml[i] # Unified HTTP/1.1 + HTTP/2 connection
188├── protocol.ml[i] # Protocol selection and negotiation
189│
190├── # ══════════════════════════════════════════════════════════════
191├── # PUBLIC API (Existing - Updated)
192├── # ══════════════════════════════════════════════════════════════
193├── requests.ml[i] # Session API (protocol-transparent)
194└── one.ml[i] # One-shot API (protocol-transparent)
195```
196
197### File License Summary
198
199| File Pattern | License | Reason |
200|--------------|---------|--------|
201| `h2_hpack.ml[i]` | BSD-3-Clause | Derived from h2 HPACK implementation |
202| `h2_huffman.ml[i]` | BSD-3-Clause | Derived from h2 Huffman tables |
203| `h2_stream.ml[i]` | BSD-3-Clause | State machine patterns from h2 |
204| All other files | ISC | Original implementation |
205
206### Data Flow
207
208```
209┌─────────────────────────────────────────────────────────────────┐
210│ Requests API │
211│ (get, post, put, delete - unchanged interface) │
212└─────────────────────────────┬───────────────────────────────────┘
213 │
214 ▼
215┌─────────────────────────────────────────────────────────────────┐
216│ Connection Router │
217│ - ALPN negotiation for protocol selection │
218│ - Protocol-specific handler dispatch │
219└──────────────┬──────────────────────────────────┬───────────────┘
220 │ │
221 ▼ ▼
222┌──────────────────────────┐ ┌──────────────────────────────┐
223│ HTTP/1.1 Handler │ │ HTTP/2 Handler │
224│ (existing http_client) │ │ (new h2_client) │
225└──────────────────────────┘ └──────────────┬───────────────┘
226 │
227 ┌───────────────────────┼───────────────────────┐
228 │ │ │
229 ▼ ▼ ▼
230 ┌────────────┐ ┌────────────┐ ┌────────────┐
231 │ Stream 1 │ │ Stream 3 │ │ Stream 5 │
232 │ Request A │ │ Request B │ │ Request C │
233 └────────────┘ └────────────┘ └────────────┘
234```
235
236## Phase 1: Core Frame Layer
237
238### h2_frame.ml - Frame Parsing and Serialization
239
240Implements RFC 9113 Section 4 (HTTP Frames) and Section 6 (Frame Definitions).
241
242```ocaml
243(** RFC 9113 Frame Types *)
244
245(** Frame header - 9 octets fixed per RFC 9113 §4.1 *)
246type frame_header = {
247 length : int; (** 24-bit payload length *)
248 frame_type : frame_type; (** 8-bit type *)
249 flags : flags; (** 8-bit flags *)
250 stream_id : stream_id; (** 31-bit stream identifier *)
251}
252
253(** Frame types per RFC 9113 §6 *)
254type frame_type =
255 | Data (** 0x00 - RFC 9113 §6.1 *)
256 | Headers (** 0x01 - RFC 9113 §6.2 *)
257 | Priority (** 0x02 - RFC 9113 §6.3 (deprecated but must parse) *)
258 | Rst_stream (** 0x03 - RFC 9113 §6.4 *)
259 | Settings (** 0x04 - RFC 9113 §6.5 *)
260 | Push_promise (** 0x05 - RFC 9113 §6.6 *)
261 | Ping (** 0x06 - RFC 9113 §6.7 *)
262 | Goaway (** 0x07 - RFC 9113 §6.8 *)
263 | Window_update (** 0x08 - RFC 9113 §6.9 *)
264 | Continuation (** 0x09 - RFC 9113 §6.10 *)
265 | Unknown of int
266
267(** Frame payload variants *)
268type frame_payload =
269 | Data_payload of {
270 data : Cstruct.t;
271 padding : int option;
272 end_stream : bool;
273 }
274 | Headers_payload of {
275 header_block : Cstruct.t;
276 priority : priority option;
277 end_stream : bool;
278 end_headers : bool;
279 }
280 | Settings_payload of setting list
281 | Window_update_payload of int32
282 | Rst_stream_payload of error_code
283 | Ping_payload of Cstruct.t (** 8 bytes exactly *)
284 | Goaway_payload of {
285 last_stream_id : stream_id;
286 error_code : error_code;
287 debug_data : Cstruct.t;
288 }
289 | Continuation_payload of {
290 header_block : Cstruct.t;
291 end_headers : bool;
292 }
293 | Push_promise_payload of {
294 promised_stream_id : stream_id;
295 header_block : Cstruct.t;
296 end_headers : bool;
297 }
298
299type frame = {
300 header : frame_header;
301 payload : frame_payload;
302}
303
304(** Eio-native frame reading *)
305val read_frame :
306 Eio.Buf_read.t ->
307 max_frame_size:int ->
308 (frame, error_code * string) result
309
310(** Eio-native frame writing *)
311val write_frame :
312 (Eio.Buf_write.t -> unit) ->
313 frame ->
314 unit
315```
316
317**Key Implementation Notes:**
318
3191. Use `Cstruct` for zero-copy buffer management
3202. Frame size validation per SETTINGS_MAX_FRAME_SIZE
3213. Reserved bit handling (must ignore on receive, set to 0 on send)
3224. Unknown frame type handling (MUST ignore per §5.5)
323
324### h2_error.ml - Error Codes
325
326Implements RFC 9113 Section 7 (Error Codes).
327
328```ocaml
329type error_code =
330 | No_error (** 0x00 - Graceful shutdown *)
331 | Protocol_error (** 0x01 - Protocol error detected *)
332 | Internal_error (** 0x02 - Implementation fault *)
333 | Flow_control_error (** 0x03 - Flow control violated *)
334 | Settings_timeout (** 0x04 - Settings not acknowledged *)
335 | Stream_closed (** 0x05 - Frame on closed stream *)
336 | Frame_size_error (** 0x06 - Frame size incorrect *)
337 | Refused_stream (** 0x07 - Stream not processed *)
338 | Cancel (** 0x08 - Stream cancelled *)
339 | Compression_error (** 0x09 - Compression state error *)
340 | Connect_error (** 0x0a - TCP connection error for CONNECT *)
341 | Enhance_your_calm (** 0x0b - Processing capacity exceeded *)
342 | Inadequate_security (** 0x0c - Negotiated TLS insufficient *)
343 | Http_1_1_required (** 0x0d - Use HTTP/1.1 *)
344 | Unknown of int32
345
346type error =
347 | Connection_error of error_code * string
348 | Stream_error of stream_id * error_code
349```
350
351## Phase 2: HPACK Header Compression
352
353### h2_hpack.ml - Header Compression
354
355Implements RFC 7541 (HPACK).
356
357```ocaml
358(** HPACK encoding context *)
359module Encoder : sig
360 type t
361
362 val create : capacity:int -> t
363 val set_capacity : t -> int -> unit
364
365 (** Encode headers to a buffer *)
366 val encode :
367 t ->
368 Eio.Buf_write.t ->
369 (string * string) list ->
370 unit
371end
372
373(** HPACK decoding context *)
374module Decoder : sig
375 type t
376
377 val create : capacity:int -> t
378 val set_capacity : t -> int -> unit
379
380 (** Decode a header block fragment *)
381 val decode :
382 t ->
383 Cstruct.t ->
384 ((string * string) list, error_code * string) result
385end
386
387(** Static table (RFC 7541 Appendix A) - 61 entries *)
388val static_table : (string * string) array
389
390(** Huffman encoding/decoding (RFC 7541 Appendix B) *)
391module Huffman : sig
392 val encode : string -> Cstruct.t
393 val decode : Cstruct.t -> (string, string) result
394end
395```
396
397**Key Implementation Notes:**
398
3991. Dynamic table uses FIFO eviction
4002. Static table lookup must be O(1) - use hash table
4013. Never-indexed literals for sensitive headers (cookies, auth)
4024. Integer encoding with variable-length prefix per §5.1
403
404## Phase 3: Stream State Machine
405
406### h2_stream.ml - Stream Management
407
408Implements RFC 9113 Section 5.1 (Stream States).
409
410```ocaml
411(** Stream states per RFC 9113 §5.1 Figure 2 *)
412type state =
413 | Idle
414 | Reserved_local
415 | Reserved_remote
416 | Open
417 | Half_closed_local
418 | Half_closed_remote
419 | Closed of closed_reason
420
421type closed_reason =
422 | Finished (** Normal completion with END_STREAM *)
423 | Reset_by_us of error_code
424 | Reset_by_peer of error_code
425
426(** Stream identifier - odd for client-initiated, even for server *)
427type stream_id = int32
428
429(** A single HTTP/2 stream *)
430type t = {
431 id : stream_id;
432 mutable state : state;
433 mutable send_window : int32;
434 mutable recv_window : int32;
435
436 (** Request data *)
437 request : Request.t option;
438 request_body : Body.Writer.t option;
439
440 (** Response handling *)
441 mutable response : Response.t option;
442 response_body : Eio.Stream.t; (** Backpressure-aware body stream *)
443
444 (** Completion signaling *)
445 promise : (Response.t, error) result Eio.Promise.t;
446 resolver : (Response.t, error) result Eio.Promise.u;
447}
448
449(** State transition validation *)
450val transition : t -> event -> (unit, error_code) result
451
452type event =
453 | Send_headers of { end_stream : bool }
454 | Recv_headers of { end_stream : bool }
455 | Send_data of { end_stream : bool }
456 | Recv_data of { end_stream : bool }
457 | Send_rst_stream
458 | Recv_rst_stream
459 | Send_push_promise
460 | Recv_push_promise
461
462(** Stream identifier allocation *)
463val next_stream_id : t -> stream_id
464```
465
466**Key Implementation Notes:**
467
4681. Client streams use odd IDs starting at 1
4692. Server-pushed streams use even IDs
4703. Stream IDs cannot be reused
4714. Maximum concurrent streams governed by SETTINGS
472
473## Phase 4: Flow Control
474
475### h2_flow_control.ml - Flow Control Windows
476
477Implements RFC 9113 Section 5.2 (Flow Control).
478
479```ocaml
480(** Flow control window *)
481type window = {
482 mutable size : int32;
483 mutable pending_updates : int32;
484}
485
486(** Initial window size: 65535 bytes per RFC 9113 §6.9.2 *)
487val default_window_size : int32
488
489(** Maximum window size: 2^31 - 1 *)
490val max_window_size : int32
491
492(** Connection-level flow control *)
493type connection_flow = {
494 send_window : window;
495 recv_window : window;
496}
497
498(** Stream-level flow control *)
499type stream_flow = {
500 send_window : window;
501 recv_window : window;
502}
503
504(** Consume bytes from send window, blocking if insufficient *)
505val consume_send :
506 connection_flow ->
507 stream_flow ->
508 int ->
509 unit
510
511(** Process received WINDOW_UPDATE *)
512val apply_window_update :
513 window ->
514 int32 ->
515 (unit, error_code) result
516
517(** Generate WINDOW_UPDATE when bytes consumed *)
518val update_recv_window :
519 window ->
520 int ->
521 int32 option (** Returns increment to send, if any *)
522```
523
524**Key Implementation Notes:**
525
5261. DATA frames are the only flow-controlled frames
5272. Both connection and stream level windows must have space
5283. WINDOW_UPDATE overflow is FLOW_CONTROL_ERROR
5294. Zero window size pauses sending (no busy waiting with Eio)
530
531## Phase 5: Settings Negotiation
532
533### h2_settings.ml - Connection Settings
534
535Implements RFC 9113 Section 6.5 (SETTINGS).
536
537```ocaml
538(** Settings parameters per RFC 9113 §6.5.2 *)
539type setting =
540 | Header_table_size of int (** 0x01 - Default: 4096 *)
541 | Enable_push of bool (** 0x02 - Default: true *)
542 | Max_concurrent_streams of int32 (** 0x03 - Default: unlimited *)
543 | Initial_window_size of int32 (** 0x04 - Default: 65535 *)
544 | Max_frame_size of int (** 0x05 - Default: 16384 *)
545 | Max_header_list_size of int (** 0x06 - Default: unlimited *)
546
547type t = {
548 header_table_size : int;
549 enable_push : bool;
550 max_concurrent_streams : int32 option;
551 initial_window_size : int32;
552 max_frame_size : int;
553 max_header_list_size : int option;
554}
555
556val default : t
557
558(** Validate setting values per RFC 9113 §6.5.2 *)
559val validate : setting -> (unit, error_code * string) result
560
561(** Apply settings to connection state *)
562val apply : t -> setting list -> t
563```
564
565**Key Implementation Notes:**
566
5671. SETTINGS must be acknowledged within reasonable time
5682. Initial SETTINGS in connection preface cannot be empty
5693. ENABLE_PUSH=0 from client means server MUST NOT push
5704. MAX_FRAME_SIZE range: 16384 to 16777215
571
572## Phase 6: Connection Management
573
574### h2_connection.ml - Multiplexed Connection
575
576```ocaml
577(** HTTP/2 connection state *)
578type t = {
579 flow : Eio.Flow.two_way_ty Eio.Resource.t;
580 hpack_encoder : Hpack.Encoder.t;
581 hpack_decoder : Hpack.Decoder.t;
582 mutable local_settings : Settings.t;
583 mutable remote_settings : Settings.t;
584 connection_flow : Flow_control.connection_flow;
585 streams : (stream_id, Stream.t) Hashtbl.t;
586 mutable next_stream_id : stream_id;
587 mutable goaway_received : bool;
588 mutable goaway_sent : bool;
589
590 (** Eio synchronization *)
591 write_mutex : Eio.Mutex.t;
592 pending_writes : Frame.frame Eio.Stream.t;
593}
594
595(** Connection preface - client sends magic + SETTINGS *)
596val connection_preface : string
597(** "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n" *)
598
599(** Establish HTTP/2 connection *)
600val create :
601 sw:Eio.Switch.t ->
602 flow:Eio.Flow.two_way_ty Eio.Resource.t ->
603 settings:Settings.t ->
604 t
605
606(** Run the connection (spawns reader/writer fibers) *)
607val run : t -> unit
608
609(** Initiate a new stream for a request *)
610val open_stream :
611 t ->
612 Request.t ->
613 (Stream.t, error) result
614
615(** Graceful shutdown *)
616val shutdown : t -> error_code -> unit
617```
618
619### Connection Lifecycle
620
621```
622Client Server
623 | |
624 |-- Connection Preface (magic string) -------->|
625 |-- SETTINGS frame -------------------------->|
626 | |
627 |<-- SETTINGS frame --------------------------|
628 |-- SETTINGS ACK ---------------------------->|
629 |<-- SETTINGS ACK ----------------------------|
630 | |
631 |== Connection Established ===================|
632 | |
633 |-- HEADERS (stream 1) ---------------------->|
634 |<-- HEADERS (stream 1) ----------------------|
635 |-- DATA (stream 1, END_STREAM) ------------->|
636 |<-- DATA (stream 1, END_STREAM) -------------|
637 | |
638 |-- GOAWAY ---------------------------------->|
639 |<-- GOAWAY ---------------------------------|
640 | |
641```
642
643## Phase 7: Client Implementation
644
645### h2_client.ml - HTTP/2 Client
646
647This module uses shared types throughout, ensuring HTTP/2 responses are
648indistinguishable from HTTP/1.1 responses at the API level.
649
650```ocaml
651(** Make an HTTP/2 request over an existing connection.
652 Uses shared types: Method.t, Uri.t, Headers.t, Body.t → Response.t *)
653val request :
654 sw:Eio.Switch.t ->
655 connection:H2_connection.t ->
656 meth:Method.t -> (* Shared: method.ml *)
657 uri:Uri.t -> (* Shared: uri.ml *)
658 headers:Headers.t -> (* Shared: headers.ml *)
659 body:Body.t -> (* Shared: body.ml *)
660 Response.t Eio.Promise.t (* Shared: response.ml *)
661
662(** HTTP/2 pseudo-headers for requests per RFC 9113 §8.3.1
663 These are extracted from the shared types, not exposed to users *)
664type request_pseudo_headers = {
665 method_ : string; (** :method - from Method.to_string *)
666 scheme : string; (** :scheme - from Uri.scheme *)
667 authority : string; (** :authority - from Uri.host_with_port *)
668 path : string; (** :path - from Uri.path_and_query *)
669}
670
671(** Extract pseudo-headers from shared request types *)
672val pseudo_headers_of_request :
673 meth:Method.t ->
674 uri:Uri.t ->
675 request_pseudo_headers
676
677(** Encode request headers for HEADERS frame.
678 Combines pseudo-headers with regular Headers.t *)
679val encode_request_headers :
680 Hpack.Encoder.t ->
681 pseudo:request_pseudo_headers ->
682 headers:Headers.t -> (* Shared: headers.ml *)
683 Cstruct.t
684
685(** Build Response.t from HTTP/2 response headers and body stream.
686 This is where HTTP/2 frames are converted to the shared Response type. *)
687val response_of_h2 :
688 sw:Eio.Switch.t ->
689 status:int -> (* From :status pseudo-header *)
690 headers:Headers.t -> (* Shared: headers.ml, pseudo-headers stripped *)
691 body:Eio.Flow.source_ty Eio.Resource.t -> (* DATA frames as Eio flow *)
692 url:string -> (* Original request URL *)
693 elapsed:float -> (* Request timing *)
694 Response.t (* Shared: response.ml *)
695(** The returned Response.t is identical in structure to HTTP/1.1 responses.
696 Users cannot distinguish between protocols without explicit inspection. *)
697
698(** Handle PUSH_PROMISE - RFC 9113 §8.4
699 Server push is disabled by default for clients (SETTINGS_ENABLE_PUSH=0) *)
700val handle_push_promise :
701 connection:H2_connection.t ->
702 promised_stream_id:H2_stream.stream_id ->
703 headers:Headers.t ->
704 unit
705```
706
707### Response Construction Flow
708
709```
710HTTP/2 HEADERS Frame HTTP/1.1 Status Line + Headers
711 │ │
712 ▼ ▼
713┌──────────────────┐ ┌──────────────────┐
714│ Decode HPACK │ │ Parse headers │
715│ Extract :status │ │ Parse status │
716│ Strip pseudos │ │ │
717└────────┬─────────┘ └────────┬─────────┘
718 │ │
719 │ ┌────────────────────────┐ │
720 └───►│ Response.Private.make │◄─────────┘
721 │ ~status ~headers │
722 │ ~body ~url ~elapsed │
723 └───────────┬────────────┘
724 │
725 ▼
726 ┌───────────────┐
727 │ Response.t │ ◄── Same type for both protocols
728 │ (shared) │
729 └───────────────┘
730```
731
732## Phase 8: Protocol Selection (ALPN)
733
734### http_version.ml - Protocol Detection
735
736```ocaml
737(** Supported HTTP versions *)
738type version =
739 | Http_1_0
740 | Http_1_1
741 | Http_2
742
743(** Pretty printer *)
744val pp : Format.formatter -> version -> unit
745
746(** String conversion *)
747val to_string : version -> string
748
749(** ALPN protocol identifiers per RFC 9113 §3.1 *)
750val alpn_h2 : string (** "h2" - HTTP/2 over TLS *)
751val alpn_http_1_1 : string (** "http/1.1" *)
752
753(** Build ALPN list for TLS configuration *)
754val alpn_protocols : preferred:version list -> string list
755(** Returns ALPN identifiers in preference order.
756 Example: [alpn_protocols ~preferred:[Http_2; Http_1_1]] returns ["h2"; "http/1.1"] *)
757```
758
759### protocol.ml - Protocol Negotiation
760
761```ocaml
762(** Protocol negotiation and connection establishment *)
763
764(** Preferred protocol configuration *)
765type preference =
766 | Prefer_h2 (** Try HTTP/2 first, fall back to HTTP/1.1 *)
767 | Http2_only (** HTTP/2 only, fail if not supported *)
768 | Http1_only (** HTTP/1.1 only, no ALPN negotiation *)
769 | Auto (** Use ALPN result, default to HTTP/1.1 *)
770
771(** Establish connection with protocol negotiation *)
772val connect :
773 sw:Eio.Switch.t ->
774 net:_ Eio.Net.t ->
775 clock:_ Eio.Time.clock ->
776 tls_config:Tls.Config.client option ->
777 preference:preference ->
778 host:string ->
779 port:int ->
780 Connection.t
781(** Performs:
782 1. TCP connection
783 2. TLS handshake with ALPN (if HTTPS)
784 3. Protocol detection from ALPN result
785 4. HTTP/2 connection preface (if HTTP/2)
786 5. Returns unified Connection.t *)
787
788(** Detect protocol from established TLS connection *)
789val detect_from_alpn : Tls_eio.t -> Http_version.version option
790
791(** Check if server supports HTTP/2 (for connection reuse decisions) *)
792val supports_h2 : Connection.t -> bool
793```
794
795## Phase 9: Unified Connection Abstraction
796
797### connection.ml - Protocol-Agnostic Connection
798
799This module provides a unified interface that works with both HTTP/1.1 and HTTP/2,
800using the shared types from the requests library.
801
802```ocaml
803(** HTTP protocol version *)
804type version =
805 | Http_1_1
806 | Http_2
807
808(** A connection that can be HTTP/1.1 or HTTP/2 *)
809type t =
810 | Http1 of {
811 flow : Eio.Flow.two_way_ty Eio.Resource.t;
812 version : [ `Http_1_0 | `Http_1_1 ];
813 }
814 | Http2 of {
815 connection : H2_connection.t;
816 }
817
818(** Connection pool key - for HTTP/2, one connection handles all streams *)
819type pool_key = {
820 host : string;
821 port : int;
822 scheme : [ `Http | `Https ];
823}
824
825(** Request using shared types - protocol handled internally *)
826type request = {
827 meth : Method.t; (** Shared: method.ml *)
828 uri : Uri.t; (** Shared: uri.ml *)
829 headers : Headers.t; (** Shared: headers.ml *)
830 body : Body.t; (** Shared: body.ml *)
831}
832
833(** Execute a request on the appropriate protocol.
834 Returns Response.t (shared type) regardless of underlying protocol. *)
835val execute :
836 t ->
837 sw:Eio.Switch.t ->
838 clock:_ Eio.Time.clock ->
839 request:request ->
840 auto_decompress:bool ->
841 Response.t
842(** The response uses shared types:
843 - [Response.status] returns [Status.t] (shared)
844 - [Response.headers] returns [Headers.t] (shared)
845 - [Response.body] returns [Eio.Flow.source] (protocol-agnostic stream) *)
846
847(** Check if connection can accept more streams.
848 - HTTP/1.1: Always true (one request at a time, pipelining not supported)
849 - HTTP/2: True if under MAX_CONCURRENT_STREAMS limit *)
850val can_accept_stream : t -> bool
851
852(** Get the negotiated protocol version *)
853val version : t -> version
854
855(** HTTP/2-specific: number of active streams *)
856val active_streams : t -> int
857```
858
859### Type Flow Diagram
860
861```
862 ┌─────────────────────────────────────────┐
863 │ User Code │
864 │ Requests.get session url │
865 └───────────────────┬─────────────────────┘
866 │
867 ┌───────────────────▼─────────────────────┐
868 │ Shared Request Types │
869 │ Method.t, Uri.t, Headers.t, Body.t │
870 └───────────────────┬─────────────────────┘
871 │
872 ┌─────────────────────────┼─────────────────────────┐
873 │ │ │
874 ▼ ▼ ▼
875 ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
876 │ HTTP/1.1 │ │ Connection │ │ HTTP/2 │
877 │ http_write │◄─────│ Abstraction │─────►│ h2_client │
878 │ http_read │ │ connection.ml │ │ h2_frame │
879 └────────┬────────┘ └─────────────────┘ └────────┬────────┘
880 │ │
881 │ ┌─────────────────────────┐ │
882 └────────►│ Shared Response Type │◄─────────────┘
883 │ Response.t │
884 │ Status.t, Headers.t │
885 │ Eio.Flow.source (body) │
886 └─────────────────────────┘
887```
888
889## Phase 10: Updated Requests API
890
891### Changes to requests.ml
892
893The main API remains unchanged from the user's perspective. Internal changes:
894
8951. **Connection Pool Changes**: HTTP/2 connections are multiplexed, so pooling strategy differs
8962. **ALPN Negotiation**: Automatic protocol selection for HTTPS
8973. **Stream Multiplexing**: Multiple concurrent requests on single connection
898
899```ocaml
900(** Internal: choose protocol and execute request *)
901let make_request_internal t ~method_ ~uri ~headers ~body =
902 let scheme = Uri.scheme uri in
903 let pool = match scheme with
904 | "http" -> t.http_pool
905 | "https" -> t.https_pool
906 | _ -> raise (Invalid_argument "unsupported scheme")
907 in
908
909 Conpool.with_connection pool (Uri.host_with_port uri) @@ fun conn ->
910 match conn with
911 | Connection.Http1 _ ->
912 (* Existing HTTP/1.1 code path *)
913 Http_client.make_request ...
914 | Connection.Http2 { connection } ->
915 (* New HTTP/2 code path *)
916 let stream = H2_connection.open_stream connection request in
917 Eio.Promise.await stream.promise
918```
919
920## Implementation Phases
921
922### Phase 0: Shared Type Extensions (Week 1) - COMPLETED
923- [x] Extend `headers.ml` with pseudo-header support for HTTP/2
924- [x] Extend `error.ml` with HTTP/2 error variants
925- [x] Add `http_version.ml` - version type and ALPN identifiers
926- [x] Unit tests for header pseudo-header validation (28 tests passing)
927
928### Phase 1: Frame Layer (Week 1-2) - COMPLETED
929- [x] `h2/h2_frame.ml` - Frame types, parsing, serialization
930- [x] Unit tests for all frame types (37 tests passing)
931- [ ] Fuzz tests for frame parsing robustness
932
933### Phase 2: HPACK (Week 2-3) - BSD-3-Clause files
934- [ ] `h2/h2_hpack_static.ml` - Static table (RFC 7541 Appendix A)
935- [ ] `h2/h2_huffman.ml` - Huffman encode/decode (RFC 7541 Appendix B)
936- [ ] `h2/h2_hpack.ml` - Encoder and Decoder
937 - [ ] Integer encoding with prefix
938 - [ ] String literal encoding
939 - [ ] Dynamic table with eviction
940 - [ ] Indexed vs literal header representations
941- [ ] Unit tests with RFC 7541 Appendix C examples
942
943### Phase 3: Stream State Machine (Week 3-4) - BSD-3-Clause
944- [ ] `h2/h2_stream.ml` - Stream state machine
945 - [ ] State transitions per RFC 9113 §5.1
946 - [ ] Stream ID allocation (odd for client)
947 - [ ] Stream lifecycle (idle → open → half-closed → closed)
948- [ ] `h2/h2_flow_control.ml` - Window management
949 - [ ] Connection and stream level windows
950 - [ ] WINDOW_UPDATE generation
951 - [ ] Backpressure via Eio.Stream
952- [ ] Unit tests for state transitions and flow control
953
954### Phase 4: Connection Management (Week 4-5)
955- [ ] `h2/h2_settings.ml` - Settings frame handling
956- [ ] `h2/h2_connection.ml` - Connection lifecycle
957 - [ ] Connection preface exchange
958 - [ ] Settings negotiation
959 - [ ] GOAWAY handling
960 - [ ] PING/PONG
961 - [ ] Reader/writer fiber management
962- [ ] `h2/h2_client.ml` - Client request handling
963 - [ ] Request → HEADERS frame conversion
964 - [ ] Response handling with shared Response.t
965 - [ ] Trailer support
966- [ ] Integration tests with mock HTTP/2 server
967
968### Phase 5: Protocol Abstraction (Week 5-6)
969- [ ] `protocol.ml` - Protocol negotiation
970- [ ] `connection.ml` - Unified connection type
971- [ ] Update `requests.ml`:
972 - [ ] ALPN preference configuration
973 - [ ] Protocol-transparent request execution
974 - [ ] Connection pool adaptation for HTTP/2 multiplexing
975- [ ] Update `one.ml` for HTTP/2 support
976- [ ] Integration tests with real HTTP/2 servers
977
978### Phase 6: Testing & Documentation (Week 6-7)
979- [ ] h2spec compliance testing (all test groups)
980- [ ] Interoperability testing (nginx, Cloudflare, Google)
981- [ ] Performance benchmarks vs HTTP/1.1
982- [ ] Update module documentation with RFC references
983- [ ] Usage examples in documentation
984- [ ] CHANGELOG entry
985
986## Key Differences from ocaml-h2
987
988| Aspect | ocaml-h2 | Our Implementation |
989|--------|----------|-------------------|
990| I/O Model | Callback-based, runtime-agnostic | Native Eio with structured concurrency |
991| Parsing | Angstrom | Direct Eio.Buf_read parsing |
992| Serialization | Faraday | Eio.Buf_write |
993| Concurrency | External via Lwt/Async/Eio adapters | Built-in Eio fibers and promises |
994| Flow Control | Manual callbacks | Eio.Stream backpressure |
995| Error Handling | Result types + callbacks | Eio exceptions + Result |
996| Types | Standalone Request/Response types | Shared with HTTP/1.1 (Method, Status, Headers, Body, Response) |
997| Integration | Separate library | Built into requests library |
998
999### Shared Type Benefits
1000
10011. **Single API Surface**: Users work with the same types regardless of protocol
10022. **No Conversion Overhead**: Response from HTTP/2 uses same `Response.t` as HTTP/1.1
10033. **Consistent Error Handling**: Extended `Error.error` type covers both protocols
10044. **Unified Headers**: `Headers.t` works for both, with H2 pseudo-header extensions
10055. **Protocol Transparency**: `Requests.get` works the same for HTTP/1.1 and HTTP/2
1006
1007## OCamldoc Templates
1008
1009### Module-level reference:
1010```ocaml
1011(** RFC 9113 HTTP/2 Frame handling.
1012
1013 This module implements HTTP/2 frame parsing and serialization as specified in
1014 {{:https://datatracker.ietf.org/doc/html/rfc9113#section-4}RFC 9113 Section 4}
1015 and {{:https://datatracker.ietf.org/doc/html/rfc9113#section-6}Section 6}. *)
1016```
1017
1018### Section-specific reference:
1019```ocaml
1020(** Stream state machine per
1021 {{:https://datatracker.ietf.org/doc/html/rfc9113#section-5.1}RFC 9113 Section 5.1}. *)
1022```
1023
1024### HPACK reference:
1025```ocaml
1026(** HPACK header compression per
1027 {{:https://datatracker.ietf.org/doc/html/rfc7541}RFC 7541}. *)
1028```
1029
1030## Testing Strategy
1031
1032### Unit Tests
1033- Each module has comprehensive unit tests
1034- Use RFC examples as test vectors (especially HPACK Appendix C)
1035- Property-based tests for frame parsing/serialization roundtrips
1036
1037### Integration Tests
1038- Mock HTTP/2 server for controlled testing
1039- Full request/response cycles
1040- Error condition handling (RST_STREAM, GOAWAY)
1041- Flow control behavior under load
1042
1043### h2spec Compliance
1044Run the [h2spec](https://github.com/summerwind/h2spec) test suite:
1045```bash
1046h2spec -h localhost -p 8443 --tls --insecure
1047```
1048All test groups must pass:
1049- Generic tests
1050- HPACK tests
1051- HTTP/2 tests (all sections)
1052
1053### Interoperability Testing
1054Test against major HTTP/2 servers:
1055- nginx (most common)
1056- Cloudflare (strict implementation)
1057- Google (reference implementation)
1058- Apache (with mod_http2)
1059
1060### Shared Type Verification
1061Verify that responses from HTTP/2 are indistinguishable from HTTP/1.1:
1062```ocaml
1063(* This test should pass regardless of protocol *)
1064let test_shared_response protocol =
1065 let response = match protocol with
1066 | `Http1 -> make_http1_request ()
1067 | `Http2 -> make_http2_request ()
1068 in
1069 (* All these use shared types *)
1070 assert (Response.status response = `OK);
1071 assert (Headers.get `Content_type (Response.headers response) = Some "application/json");
1072 let body = Response.text response in
1073 assert (String.length body > 0)
1074```
1075
1076### Fuzz Testing
1077Property-based testing for:
1078- Frame parsing (malformed frames should not crash)
1079- HPACK decoding (compression bombs, invalid sequences)
1080- Header validation (injection attacks)
1081
1082## Dependencies
1083
1084- `eio` - Structured concurrency runtime (existing)
1085- `cstruct` - Zero-copy buffer management (may already exist)
1086- `tls-eio` - TLS with ALPN support (existing)
1087- No new external dependencies for core HTTP/2
1088
1089## Risks and Mitigations
1090
1091| Risk | Mitigation |
1092|------|------------|
1093| HPACK complexity | Follow RFC 7541 examples exactly, extensive testing |
1094| Flow control deadlocks | Use Eio.Stream for backpressure, careful window management |
1095| Connection multiplexing bugs | State machine validation, property-based testing |
1096| Performance regression | Benchmark early and often, profile critical paths |
1097
1098## References
1099
11001. [RFC 9113 - HTTP/2](https://datatracker.ietf.org/doc/html/rfc9113)
11012. [RFC 7541 - HPACK](https://datatracker.ietf.org/doc/html/rfc7541)
11023. [RFC 9110 - HTTP Semantics](https://datatracker.ietf.org/doc/html/rfc9110)
11034. [ocaml-h2 Reference Implementation](https://github.com/anmonteiro/ocaml-h2)
11045. [h2spec Test Suite](https://github.com/summerwind/h2spec)