anil.recoil.org / ocaml-requests
A batteries included HTTP/1.1 client in OCaml
fork atom

features

anil.recoil.org 42037660 60979f57

+224 -128
unified split
+224 -128
RECOMMENDATIONS.md
··· 2 3 > Auto-generated summary of recommendations from analyzing HTTP client libraries across multiple languages. 4 5 ## Executive Summary 6 7 This comprehensive analysis synthesizes recommendations from 45+ HTTP client libraries across 10 programming languages (Python, JavaScript, Go, Rust, Haskell, PHP, Java, Swift, C++, Bash). The most consistent themes are: (1) Security features like certificate pinning, cross-origin header stripping, and HTTPS enforcement for sensitive auth; (2) Middleware/interceptor systems for extensibility; (3) HTTP proxy support; (4) Progress callbacks for uploads/downloads; (5) Request cancellation capabilities; (6) Detailed timing metrics; (7) Retry logic improvements including Retry-After capping and custom predicates. The prioritization emphasizes security fixes that could lead to credential leakage, followed by missing features that block enterprise adoption (proxy support), then enhancements that improve developer experience. ··· 11 12 ## Security & Spec Compliance 13 14 - ### 1. Strip sensitive headers on cross-origin redirects 15 16 When redirecting to a different origin (host/port/scheme), automatically strip Authorization, Cookie, Proxy-Authorization, and WWW-Authenticate headers to prevent credential leakage to unintended domains. Also strip headers on HTTPS→HTTP protocol downgrade. 17 ··· 21 **Cross-Language Consensus:** 8 libraries 22 **Source Libraries:** reqwest, got, node-fetch, okhttp, axios, superagent, http-client, needle 23 24 - **Affected Files:** 25 - - `lib/requests.ml` 26 - - `lib/one.ml` 27 - - `lib/http_client.ml` 28 29 - **Implementation Notes:** 30 - Implement same_origin check comparing scheme, host, and port. Create a list of sensitive headers to strip. Call strip_sensitive_headers before following any redirect where origin changes. 31 32 - ### 2. Add certificate/public key pinning support 33 34 Allow pinning certificates or public keys (SPKI hashes) to protect against compromised certificate authorities. Support per-host pinning configuration with wildcard patterns. 35 ··· 46 **Implementation Notes:** 47 Add certificate_pins parameter to create() accepting (hostname * sha256_hash list) list. Verify pins during TLS handshake by extracting SPKI from certificate chain. Provide helpful error messages showing expected vs actual hashes. 48 49 - ### 3. Enforce HTTPS for sensitive authentication methods 50 51 Reject Basic, Bearer, and Digest authentication over unencrypted HTTP connections by default. Provide explicit opt-out flag (allow_insecure_auth) for testing environments. 52 ··· 57 **Cross-Language Consensus:** 2 libraries 58 **Source Libraries:** req (Haskell), urllib3 59 60 - **Affected Files:** 61 - - `lib/auth.ml` 62 - - `lib/auth.mli` 63 - - `lib/requests.ml` 64 - - `lib/one.ml` 65 66 - **Implementation Notes:** 67 - Add runtime check in auth application: if scheme is http (not https) and auth is Basic/Bearer/Digest, raise Security_error unless allow_insecure_auth=true. Log warning when flag is used. 68 69 - ### 4. Validate Content-Length vs Transfer-Encoding precedence 70 71 Per RFC 9112, when both Content-Length and Transfer-Encoding headers are present, ignore Content-Length. This prevents HTTP request smuggling attacks. 72 ··· 76 **Cross-Language Consensus:** 2 libraries 77 **Source Libraries:** hyper, http-streams 78 79 - **Affected Files:** 80 - - `lib/http_read.ml` 81 82 - **Implementation Notes:** 83 - Audit http_read.ml to ensure Transfer-Encoding: chunked takes precedence. Log warning when both headers present (potential attack indicator). Add test case for this scenario. 84 85 - ### 5. Cap maximum Retry-After delay to prevent DoS 86 87 Implement a configurable maximum backoff time (default 120s) that caps server-specified Retry-After values, preventing malicious servers from causing indefinite client blocking. 88 ··· 92 **Cross-Language Consensus:** 2 libraries 93 **Source Libraries:** req (Go), got 94 95 - **Affected Files:** 96 - - `lib/retry.ml` 97 - - `lib/retry.mli` 98 - 99 - **Implementation Notes:** 100 - Add backoff_max field to Retry.config. In calculate_backoff, apply: min retry_after backoff_max. Document that very long delays (>backoff_max) result in immediate failure rather than unbounded waiting. 101 102 103 --- 104 105 ## Feature Enhancements 106 107 - ### 6. Add HTTP proxy support with environment variable detection 108 109 Implement HTTP/HTTPS proxy support including: proxy URL configuration, authentication (Basic), CONNECT method for HTTPS tunneling, and automatic detection from HTTP_PROXY, HTTPS_PROXY, NO_PROXY environment variables. 110 ··· 122 - `lib/one.ml` 123 124 **Implementation Notes:** 125 - Add proxy configuration type with host/port/auth. Implement HTTP CONNECT for HTTPS proxies. Parse environment variables on session creation. Add NO_PROXY pattern matching for bypass. Strip Proxy-Authorization on HTTPS connections. 126 127 - ### 7. Implement request/response middleware system 128 129 Add a composable middleware chain that allows intercepting and transforming requests before sending and responses after receiving. Enable cross-cutting concerns like logging, metrics, request signing, and custom caching. 130 ··· 138 **Implementation Notes:** 139 Define type middleware = Request.t -> (Request.t -> Response.t) -> Response.t. Store middleware list in session. Execute chain: apply each middleware wrapping the next. Provide default middleware for retry and redirect as replaceable components. 140 141 - ### 8. Add progress callbacks for uploads and downloads 142 143 Implement optional progress callbacks that report bytes transferred, total size (if known), and percent complete. Rate-limit callbacks to prevent performance degradation. 144 ··· 154 **Implementation Notes:** 155 Add optional progress:(transferred:int64 -> total:int64 option -> unit) parameter. Track bytes during streaming read/write. Throttle callbacks (e.g., max 3/second or every 64KB). Support both upload and download progress. 156 157 - ### 9. Add request cancellation support 158 159 Allow programmatic cancellation of in-flight requests. Integrate with Eio's cancellation system while providing explicit cancel() function or cancel token pattern. 160 ··· 169 **Implementation Notes:** 170 Return (Response.t * cancel_token) from request functions where cancel_token allows aborting the request. Internally use Eio.Cancel or Switch.fail to terminate the fiber. Raise Cancelled error on cancellation. 171 172 - ### 10. Add concurrent request batching API (map/imap) 173 174 Provide map() for executing multiple requests concurrently and returning results in order, and imap() for yielding responses as they complete. Include configurable concurrency limits and batch timeout. 175 ··· 183 **Implementation Notes:** 184 Implement using Eio.Fiber.all for map, Eio.Stream for imap. Add max_concurrent parameter using Eio.Semaphore. Provide exception_handler callback for partial failure handling. Add batch_timeout for overall operation limit. 185 186 - ### 11. Add base URL support with relative path resolution 187 188 Allow configuring a base URL on the session that is prepended to relative request URLs. Use proper RFC 3986 URI resolution for correct path joining. 189 ··· 193 **Cross-Language Consensus:** 5 libraries 194 **Source Libraries:** surf, axios, got, sling, req (Go) 195 196 - **Affected Files:** 197 - - `lib/requests.ml` 198 - - `lib/requests.mli` 199 - 200 - **Implementation Notes:** 201 - Add base_url field to session type. In request methods, check if URL is relative (no scheme); if so, use Uri.resolve to join with base_url. Document trailing slash importance per RFC 3986. 202 203 204 --- 205 206 ## Architectural Improvements 207 208 - ### 12. Add detailed timing metrics to responses 209 210 Expose granular timing breakdown: DNS lookup time, TCP connection time, TLS handshake time, time to first byte (TTFB), download time, and total time. Include connection reuse tracking. 211 212 **Cross-Language Consensus:** 7 libraries 213 **Source Libraries:** curl-rust, got, resty, req (Go), guzzle, okhttp, isahc 214 215 - **Affected Files:** 216 - - `lib/response.ml` 217 - - `lib/response.mli` 218 - - `lib/http_client.ml` 219 220 - **Implementation Notes:** 221 - Define type timings = { dns: float option; connect: float; tls: float option; send: float; first_byte: float; download: float; total: float; connection_reused: bool }. Instrument http_client.ml with timestamps at each phase. 222 223 - ### 13. Add automatic retry on stale connection detection 224 225 Detect when a pooled connection was closed by the server between requests (NoResponseDataReceived) and automatically retry with a fresh connection. Mark this error as always retryable. 226 227 **Cross-Language Consensus:** 2 libraries 228 **Source Libraries:** http-client (Haskell), okhttp 229 230 - **Affected Files:** 231 - - `lib/http_client.ml` 232 - - `lib/error.ml` 233 - - `lib/retry.ml` 234 - - `lib/conpool.ml` 235 236 - **Implementation Notes:** 237 - Add No_response_data error variant. In http_client, catch empty response on reused connection. Track connection reuse status. In retry logic, always retry No_response_data even for non-idempotent methods since request wasn't sent. 238 239 - ### 14. Add custom retry condition predicates 240 241 Allow users to provide custom retry decision functions beyond status codes and error types. Separate predicates for response-based retry (status/headers) and exception-based retry. 242 243 **Cross-Language Consensus:** 4 libraries 244 **Source Libraries:** resty, got, req (Go), req (Haskell) 245 246 - **Affected Files:** 247 - - `lib/retry.ml` 248 - - `lib/retry.mli` 249 - 250 - **Implementation Notes:** 251 - Add retry_condition:(Response.t option -> exn option -> retry_status -> bool) to Retry.config. Provide default that checks status_forcelist and Error.is_retryable. Allow custom logic for specific error messages, custom headers, etc. 252 253 254 --- 255 256 ## Feature Enhancements 257 258 - ### 15. Add response charset/encoding detection and conversion 259 260 Automatically detect character encoding from Content-Type charset parameter and convert response text to UTF-8. Provide BOM sniffing fallback for encoding detection. 261 ··· 273 **Implementation Notes:** 274 Parse charset from Content-Type in Mime module. Add Response.text_utf8 that detects and converts encoding. Use uutf library for encoding conversion. Return raw bytes on conversion failure with access to detected charset. 275 276 - ### 16. Add redirect policy customization 277 278 Allow custom redirect policies with access to attempt status, URL, and previous URLs. Enable controlling method preservation (307/308 vs 301/302), header stripping, and cross-protocol redirect behavior. 279 ··· 295 296 ## Architectural Improvements 297 298 - ### 17. Add TCP keep-alive configuration 299 300 Expose TCP keep-alive settings including enable/disable, interval, and retry count. This helps detect broken connections and maintain long-lived connections through firewalls/NAT. 301 ··· 314 315 ## Feature Enhancements 316 317 - ### 18. Add URL sanitization for error messages 318 319 Automatically sanitize URLs in error messages to prevent leakage of credentials embedded in query parameters (API keys, tokens). Provide without_url() method to strip URLs entirely. 320 ··· 328 **Implementation Notes:** 329 Extend existing sanitize_url function to handle query parameter credentials. Add Error.without_url to remove URL entirely from error. Apply sanitization automatically in Error.to_string and pp functions. 330 331 - ### 19. Add Link header parsing for pagination 332 333 Parse RFC 8288 Link headers to extract relation types (next, prev, first, last) for API pagination. Provide helpers like Response.next_url for common patterns. 334 ··· 338 **Cross-Language Consensus:** 3 libraries 339 **Source Libraries:** got, superagent, just 340 341 - **Affected Files:** 342 - - `lib/headers.ml` 343 - - `lib/headers.mli` 344 - - `lib/response.ml` 345 - 346 - **Implementation Notes:** 347 - Add type link_rel = { url: string; rel: string; params: (string * string) list }. Implement parse_link_header following RFC 8288. Add Response.links and Response.next_url/prev_url convenience functions. 348 349 350 --- 351 352 ## Architectural Improvements 353 354 - ### 20. Add mock handler for testing 355 356 Provide a mock handler that queues pre-defined responses and exceptions for testing HTTP client code without network calls. Support request inspection and assertion. 357 ··· 370 371 ## Feature Enhancements 372 373 - ### 21. Add Result-based error_for_status variant 374 375 Provide check_status() that returns (Response.t, Error.error) result instead of raising, enabling functional error handling alongside the existing raise_for_status. 376 377 **Cross-Language Consensus:** 2 libraries 378 **Source Libraries:** reqwest, grequests 379 380 - **Affected Files:** 381 - - `lib/response.ml` 382 - - `lib/response.mli` 383 384 - **Implementation Notes:** 385 - Add Response.check_status : t -> (t, Error.error) result that returns Ok response for 2xx, Error with Http_error for 4xx/5xx. Complement existing raise_for_status for different coding styles. 386 387 - ### 22. Add request body rewind support for retries 388 389 Track whether request bodies are rewindable (seekable) and automatically replay them on retry. Raise clear error when retry requires body replay but body is a consumed stream. 390 ··· 404 405 ## Architectural Improvements 406 407 - ### 23. Add event listener/hooks for observability 408 409 Provide comprehensive event hooks at each request lifecycle stage: DNS start/end, connection start/end, TLS handshake, request headers/body sent, response headers/body received. Enable metrics and tracing integration. 410 ··· 424 425 ## Feature Enhancements 426 427 - ### 24. Add XSRF/CSRF token handling 428 429 Automatically read CSRF token from a configurable cookie name (default: XSRF-TOKEN) and add it to request headers (default: X-XSRF-TOKEN) for same-origin requests. 430 431 **Cross-Language Consensus:** 1 libraries 432 **Source Libraries:** axios 433 434 - **Affected Files:** 435 - - `lib/requests.ml` 436 - - `lib/headers.ml` 437 - 438 - **Implementation Notes:** 439 - Add xsrf_cookie_name and xsrf_header_name config options. Before request, if cookie exists, add corresponding header. Only for same-origin requests. Disable by setting cookie name to empty. 440 441 442 --- 443 444 ## Architectural Improvements 445 446 - ### 25. Add decompression bomb protection 447 448 Enforce configurable compression ratio limits (e.g., 100:1) to detect and abort decompression bomb attacks. Track compressed vs decompressed bytes during streaming. 449 450 **Cross-Language Consensus:** 1 libraries 451 **Source Libraries:** axios 452 453 - **Affected Files:** 454 - - `lib/http_read.ml` 455 - - `lib/response_limits.ml` 456 - 457 - **Implementation Notes:** 458 - Add max_compression_ratio to Response_limits. During decompression, track input/output bytes. If ratio exceeds limit, raise Decompression_bomb error. Use existing error type from error.ml. 459 460 461 --- 462 463 ## Feature Enhancements 464 465 - ### 26. Add HTTP version to response metadata 466 467 Expose the HTTP version used for the response (HTTP/1.0, HTTP/1.1, HTTP/2) in Response type. Useful for debugging protocol negotiation and monitoring HTTP/2 adoption. 468 469 **Cross-Language Consensus:** 3 libraries 470 **Source Libraries:** reqwest, surf, curl-rust 471 472 - **Affected Files:** 473 - - `lib/response.ml` 474 - - `lib/response.mli` 475 - - `lib/http_read.ml` 476 477 - **Implementation Notes:** 478 - Add http_version field to Response.t. Parse version from status line. Add Response.version accessor. Prepare for future HTTP/2 support. 479 480 - ### 27. Add Unix domain socket support 481 482 Enable connecting to local services via Unix domain sockets (e.g., Docker daemon at /var/run/docker.sock) without TCP overhead. 483 ··· 497 498 ## Security & Spec Compliance 499 500 - ### 28. Restrict URL protocols to HTTP/HTTPS only 501 502 Explicitly whitelist only http:// and https:// protocols to prevent protocol smuggling attacks via file://, ftp://, gopher://, etc. Reject invalid protocols early in URL parsing. 503 504 **Cross-Language Consensus:** 1 libraries 505 **Source Libraries:** buzz 506 507 - **Affected Files:** 508 - - `lib/http_client.ml` 509 - - `lib/one.ml` 510 - - `lib/requests.ml` 511 - 512 - **Implementation Notes:** 513 - Add URL scheme validation early in request processing. Reject any scheme other than http/https with Invalid_url error. Apply validation also to redirect URLs. 514 515 516 --- 517 518 ## Feature Enhancements 519 520 - ### 29. Add path parameter templating 521 522 Support URL templates like /users/{id}/posts/{post_id} with placeholder substitution from a parameter map. Automatically URL-encode substituted values. 523 ··· 527 **Cross-Language Consensus:** 3 libraries 528 **Source Libraries:** sling, resty, req (Go) 529 530 - **Affected Files:** 531 - - `lib/requests.ml` 532 - - `lib/one.ml` 533 - 534 - **Implementation Notes:** 535 - Add path_params:(string * string) list parameter. Before URL resolution, scan for {key} patterns and substitute with URL-encoded values from map. Raise error if template has unmatched placeholders. 536 537 538 --- 539 540 ## Architectural Improvements 541 542 - ### 30. Add default User-Agent header 543 544 Set a default User-Agent identifying the library (ocaml-requests/VERSION) unless user provides one. Helps server-side debugging and follows HTTP best practices. 545 ··· 549 **Cross-Language Consensus:** 5 libraries 550 **Source Libraries:** needle, resty, requests (Go), okhttp, guzzle 551 552 - **Affected Files:** 553 - - `lib/requests.ml` 554 - - `lib/one.ml` 555 - 556 - **Implementation Notes:** 557 - Add default User-Agent in default_headers if not set. Include library version and optionally OCaml version/platform. Allow override via explicit header or user_agent parameter. 558
··· 2 3 > Auto-generated summary of recommendations from analyzing HTTP client libraries across multiple languages. 4 5 + ## Implementation Status Summary 6 + 7 + | Status | Count | Description | 8 + |--------|-------|-------------| 9 + | ✅ Implemented | 15 | Fully implemented and tested | 10 + | 🔧 Partial | 1 | Partially implemented | 11 + | ❌ Not Started | 14 | Not yet implemented | 12 + 13 + ### Implemented Features 14 + 15 + | # | Feature | Location | 16 + |---|---------|----------| 17 + | 1 | Strip sensitive headers on cross-origin redirects | `lib/requests.ml`, `lib/one.ml` | 18 + | 3 | Enforce HTTPS for sensitive authentication | `lib/auth.ml` (`apply_secure`, `Insecure_auth` error) | 19 + | 4 | Content-Length vs Transfer-Encoding precedence | `lib/http_read.ml` (RFC 9112 warning) | 20 + | 5 | Cap maximum Retry-After delay | `lib/retry.ml` (`backoff_max` parameter) | 21 + | 11 | Base URL support with relative path resolution | `lib/requests.ml` (`base_url`, `resolve_url`) | 22 + | 12 | Detailed timing metrics | `lib/timing.ml` (new module) | 23 + | 13 | Stale connection detection | `ocaml-conpool/lib/conpool.ml` (`is_healthy`) | 24 + | 14 | Custom retry condition predicates | `lib/retry.ml` (`response_predicate`, `exception_predicate`) | 25 + | 19 | Link header parsing for pagination | `lib/link.ml` (new module) | 26 + | 21 | Result-based error_for_status variant | `lib/response.ml` (`check_status`) | 27 + | 24 | XSRF/CSRF token handling | `lib/requests.ml` (`apply_xsrf_token`) | 28 + | 25 | Decompression bomb protection | `lib/http_client.ml` (`check_decompression_limits`) | 29 + | 26 | HTTP version in response metadata | `lib/http_read.ml` (`http_version` type) | 30 + | 28 | Restrict URL protocols to HTTP/HTTPS | `lib/requests.ml`, `lib/one.ml` (`validate_redirect_url`) | 31 + | 29 | Path parameter templating | `lib/requests.ml` (`substitute_path_params`) | 32 + | 30 | Default User-Agent header | `lib/version.ml`, applied in `lib/requests.ml` | 33 + 34 + ### Not Yet Implemented 35 + 36 + | # | Feature | Priority | 37 + |---|---------|----------| 38 + | 2 | Certificate/public key pinning | High | 39 + | 6 | HTTP proxy support | High | 40 + | 7 | Request/response middleware system | Medium | 41 + | 8 | Progress callbacks | Medium | 42 + | 9 | Request cancellation | Medium | 43 + | 10 | Concurrent request batching | Low | 44 + | 15 | Charset/encoding detection | Low | 45 + | 16 | Redirect policy customization | Low | 46 + | 17 | TCP keep-alive configuration | Low | 47 + | 18 | URL sanitization for error messages | Low | 48 + | 20 | Mock handler for testing | Medium | 49 + | 22 | Request body rewind for retries | Medium | 50 + | 23 | Event listener/hooks | Low | 51 + | 27 | Unix domain socket support | Low | 52 + 53 + --- 54 + 55 ## Executive Summary 56 57 This comprehensive analysis synthesizes recommendations from 45+ HTTP client libraries across 10 programming languages (Python, JavaScript, Go, Rust, Haskell, PHP, Java, Swift, C++, Bash). The most consistent themes are: (1) Security features like certificate pinning, cross-origin header stripping, and HTTPS enforcement for sensitive auth; (2) Middleware/interceptor systems for extensibility; (3) HTTP proxy support; (4) Progress callbacks for uploads/downloads; (5) Request cancellation capabilities; (6) Detailed timing metrics; (7) Retry logic improvements including Retry-After capping and custom predicates. The prioritization emphasizes security fixes that could lead to credential leakage, followed by missing features that block enterprise adoption (proxy support), then enhancements that improve developer experience. ··· 61 62 ## Security & Spec Compliance 63 64 + ### 1. ✅ Strip sensitive headers on cross-origin redirects 65 + 66 + **Status: IMPLEMENTED** in `lib/requests.ml` and `lib/one.ml` 67 68 When redirecting to a different origin (host/port/scheme), automatically strip Authorization, Cookie, Proxy-Authorization, and WWW-Authenticate headers to prevent credential leakage to unintended domains. Also strip headers on HTTPS→HTTP protocol downgrade. 69 ··· 73 **Cross-Language Consensus:** 8 libraries 74 **Source Libraries:** reqwest, got, node-fetch, okhttp, axios, superagent, http-client, needle 75 76 + **Implementation:** 77 + - `same_origin` function compares scheme and host 78 + - `strip_sensitive_headers` removes Authorization, Cookie, Proxy-Authorization, WWW-Authenticate 79 + - Applied during redirect handling in `make_with_redirects` 80 81 + ### 2. ❌ Add certificate/public key pinning support 82 83 + **Status: NOT IMPLEMENTED** - High priority for enterprise security 84 85 Allow pinning certificates or public keys (SPKI hashes) to protect against compromised certificate authorities. Support per-host pinning configuration with wildcard patterns. 86 ··· 97 **Implementation Notes:** 98 Add certificate_pins parameter to create() accepting (hostname * sha256_hash list) list. Verify pins during TLS handshake by extracting SPKI from certificate chain. Provide helpful error messages showing expected vs actual hashes. 99 100 + ### 3. ✅ Enforce HTTPS for sensitive authentication methods 101 + 102 + **Status: IMPLEMENTED** in `lib/auth.ml` 103 104 Reject Basic, Bearer, and Digest authentication over unencrypted HTTP connections by default. Provide explicit opt-out flag (allow_insecure_auth) for testing environments. 105 ··· 110 **Cross-Language Consensus:** 2 libraries 111 **Source Libraries:** req (Haskell), urllib3 112 113 + **Implementation:** 114 + - `Auth.apply_secure` validates transport security before applying auth 115 + - `Auth.validate_secure_transport` checks if HTTPS is required 116 + - `Error.Insecure_auth` error type for clear error messages 117 + - `allow_insecure_auth` parameter in request functions for testing 118 119 + ### 4. ✅ Validate Content-Length vs Transfer-Encoding precedence 120 121 + **Status: IMPLEMENTED** in `lib/http_read.ml` 122 123 Per RFC 9112, when both Content-Length and Transfer-Encoding headers are present, ignore Content-Length. This prevents HTTP request smuggling attacks. 124 ··· 128 **Cross-Language Consensus:** 2 libraries 129 **Source Libraries:** hyper, http-streams 130 131 + **Implementation:** 132 + - Transfer-Encoding: chunked takes precedence over Content-Length 133 + - Warning logged when both headers present (potential attack indicator) 134 + - Per RFC 9112 Section 6.3 compliance 135 136 + ### 5. ✅ Cap maximum Retry-After delay to prevent DoS 137 138 + **Status: IMPLEMENTED** in `lib/retry.ml` 139 140 Implement a configurable maximum backoff time (default 120s) that caps server-specified Retry-After values, preventing malicious servers from causing indefinite client blocking. 141 ··· 145 **Cross-Language Consensus:** 2 libraries 146 **Source Libraries:** req (Go), got 147 148 + **Implementation:** 149 + - `backoff_max` field in `Retry.config` (default: 120.0 seconds) 150 + - `parse_retry_after` function accepts `?backoff_max` parameter 151 + - Delays exceeding cap are logged as warnings and capped 152 153 154 --- 155 156 ## Feature Enhancements 157 158 + ### 6. ❌ Add HTTP proxy support with environment variable detection 159 + 160 + **Status: NOT IMPLEMENTED** - High priority, see `spec/PROXY_IMPLEMENTATION_PLAN.md` 161 162 Implement HTTP/HTTPS proxy support including: proxy URL configuration, authentication (Basic), CONNECT method for HTTPS tunneling, and automatic detection from HTTP_PROXY, HTTPS_PROXY, NO_PROXY environment variables. 163 ··· 175 - `lib/one.ml` 176 177 **Implementation Notes:** 178 + A comprehensive implementation plan exists in `spec/PROXY_IMPLEMENTATION_PLAN.md` with 8 phases covering proxy configuration, HTTP CONNECT tunneling, environment variable detection, NO_PROXY pattern matching, and proxy authentication. 179 + 180 + ### 7. ❌ Implement request/response middleware system 181 182 + **Status: NOT IMPLEMENTED** - Medium priority 183 184 Add a composable middleware chain that allows intercepting and transforming requests before sending and responses after receiving. Enable cross-cutting concerns like logging, metrics, request signing, and custom caching. 185 ··· 193 **Implementation Notes:** 194 Define type middleware = Request.t -> (Request.t -> Response.t) -> Response.t. Store middleware list in session. Execute chain: apply each middleware wrapping the next. Provide default middleware for retry and redirect as replaceable components. 195 196 + ### 8. ❌ Add progress callbacks for uploads and downloads 197 + 198 + **Status: NOT IMPLEMENTED** - Medium priority 199 200 Implement optional progress callbacks that report bytes transferred, total size (if known), and percent complete. Rate-limit callbacks to prevent performance degradation. 201 ··· 211 **Implementation Notes:** 212 Add optional progress:(transferred:int64 -> total:int64 option -> unit) parameter. Track bytes during streaming read/write. Throttle callbacks (e.g., max 3/second or every 64KB). Support both upload and download progress. 213 214 + ### 9. ❌ Add request cancellation support 215 + 216 + **Status: NOT IMPLEMENTED** - Medium priority (Eio provides natural cancellation via switches) 217 218 Allow programmatic cancellation of in-flight requests. Integrate with Eio's cancellation system while providing explicit cancel() function or cancel token pattern. 219 ··· 228 **Implementation Notes:** 229 Return (Response.t * cancel_token) from request functions where cancel_token allows aborting the request. Internally use Eio.Cancel or Switch.fail to terminate the fiber. Raise Cancelled error on cancellation. 230 231 + ### 10. ❌ Add concurrent request batching API (map/imap) 232 + 233 + **Status: NOT IMPLEMENTED** - Low priority (Eio.Fiber.all/both provides this) 234 235 Provide map() for executing multiple requests concurrently and returning results in order, and imap() for yielding responses as they complete. Include configurable concurrency limits and batch timeout. 236 ··· 244 **Implementation Notes:** 245 Implement using Eio.Fiber.all for map, Eio.Stream for imap. Add max_concurrent parameter using Eio.Semaphore. Provide exception_handler callback for partial failure handling. Add batch_timeout for overall operation limit. 246 247 + ### 11. ✅ Add base URL support with relative path resolution 248 + 249 + **Status: IMPLEMENTED** in `lib/requests.ml` 250 251 Allow configuring a base URL on the session that is prepended to relative request URLs. Use proper RFC 3986 URI resolution for correct path joining. 252 ··· 256 **Cross-Language Consensus:** 5 libraries 257 **Source Libraries:** surf, axios, got, sling, req (Go) 258 259 + **Implementation:** 260 + - `base_url` parameter in `create` function 261 + - `is_relative_url` checks if URL has scheme 262 + - `resolve_url` uses `Uri.resolve` for RFC 3986 compliant resolution 263 + - Trailing slashes automatically normalized 264 265 266 --- 267 268 ## Architectural Improvements 269 270 + ### 12. ✅ Add detailed timing metrics to responses 271 + 272 + **Status: IMPLEMENTED** in new `lib/timing.ml` module 273 274 Expose granular timing breakdown: DNS lookup time, TCP connection time, TLS handshake time, time to first byte (TTFB), download time, and total time. Include connection reuse tracking. 275 276 **Cross-Language Consensus:** 7 libraries 277 **Source Libraries:** curl-rust, got, resty, req (Go), guzzle, okhttp, isahc 278 279 + **Implementation:** 280 + - New `Timing` module with detailed metrics type 281 + - Phases: `dns_lookup`, `tcp_connect`, `tls_handshake`, `request_sent`, `time_to_first_byte`, `content_transfer`, `total` 282 + - `timer` type for incremental collection during requests 283 + - `connection_time` and `server_time` computed metrics 284 + - Pretty-printing with `pp` and `to_string` 285 286 + ### 13. ✅ Add automatic retry on stale connection detection 287 288 + **Status: IMPLEMENTED** in `ocaml-conpool` library 289 290 Detect when a pooled connection was closed by the server between requests (NoResponseDataReceived) and automatically retry with a fresh connection. Mark this error as always retryable. 291 292 **Cross-Language Consensus:** 2 libraries 293 **Source Libraries:** http-client (Haskell), okhttp 294 295 + **Implementation:** 296 + The `ocaml-conpool` library provides comprehensive stale connection detection via: 297 + - `is_healthy` function checking age, idle time, and use count 298 + - `max_idle_time` configuration (default: 60s) 299 + - `max_connection_lifetime` configuration (default: 300s) 300 + - `max_connection_uses` configuration (optional) 301 + - Custom `health_check` callback support 302 + - Pool `validate` function runs health checks before connection reuse 303 304 + ### 14. ✅ Add custom retry condition predicates 305 306 + **Status: IMPLEMENTED** in `lib/retry.ml` 307 308 Allow users to provide custom retry decision functions beyond status codes and error types. Separate predicates for response-based retry (status/headers) and exception-based retry. 309 310 **Cross-Language Consensus:** 4 libraries 311 **Source Libraries:** resty, got, req (Go), req (Haskell) 312 313 + **Implementation:** 314 + - `response_predicate` type: `Method.t -> int -> Headers.t -> bool` 315 + - `exception_predicate` type: `exn -> bool` 316 + - `retry_response` and `retry_exception` fields in `Retry.config` 317 + - `should_retry_response` checks both built-in rules and custom predicates 318 + - `should_retry_exn` for exception-based retry decisions 319 + - Examples in module documentation 320 321 322 --- 323 324 ## Feature Enhancements 325 326 + ### 15. ❌ Add response charset/encoding detection and conversion 327 + 328 + **Status: NOT IMPLEMENTED** - Low priority 329 330 Automatically detect character encoding from Content-Type charset parameter and convert response text to UTF-8. Provide BOM sniffing fallback for encoding detection. 331 ··· 343 **Implementation Notes:** 344 Parse charset from Content-Type in Mime module. Add Response.text_utf8 that detects and converts encoding. Use uutf library for encoding conversion. Return raw bytes on conversion failure with access to detected charset. 345 346 + ### 16. ❌ Add redirect policy customization 347 + 348 + **Status: NOT IMPLEMENTED** - Low priority 349 350 Allow custom redirect policies with access to attempt status, URL, and previous URLs. Enable controlling method preservation (307/308 vs 301/302), header stripping, and cross-protocol redirect behavior. 351 ··· 367 368 ## Architectural Improvements 369 370 + ### 17. ❌ Add TCP keep-alive configuration 371 + 372 + **Status: NOT IMPLEMENTED** - Low priority 373 374 Expose TCP keep-alive settings including enable/disable, interval, and retry count. This helps detect broken connections and maintain long-lived connections through firewalls/NAT. 375 ··· 388 389 ## Feature Enhancements 390 391 + ### 18. ❌ Add URL sanitization for error messages 392 + 393 + **Status: NOT IMPLEMENTED** - Low priority 394 395 Automatically sanitize URLs in error messages to prevent leakage of credentials embedded in query parameters (API keys, tokens). Provide without_url() method to strip URLs entirely. 396 ··· 404 **Implementation Notes:** 405 Extend existing sanitize_url function to handle query parameter credentials. Add Error.without_url to remove URL entirely from error. Apply sanitization automatically in Error.to_string and pp functions. 406 407 + ### 19. ✅ Add Link header parsing for pagination 408 + 409 + **Status: IMPLEMENTED** in new `lib/link.ml` module 410 411 Parse RFC 8288 Link headers to extract relation types (next, prev, first, last) for API pagination. Provide helpers like Response.next_url for common patterns. 412 ··· 416 **Cross-Language Consensus:** 3 libraries 417 **Source Libraries:** got, superagent, just 418 419 + **Implementation:** 420 + - New `Link` module with RFC 8288 compliant parsing 421 + - `Link.t` type with `uri`, `rel`, `title`, `media_type`, `hreflang`, `params` 422 + - `parse` function handles comma-separated links with quoted values 423 + - `from_headers` extracts Link header from response headers 424 + - `pagination` returns `(first, prev, next, last)` tuple 425 + - `next_url`, `prev_url`, `has_next` convenience functions 426 + - Pretty-printing with `pp` and `to_string` 427 428 429 --- 430 431 ## Architectural Improvements 432 433 + ### 20. ❌ Add mock handler for testing 434 + 435 + **Status: NOT IMPLEMENTED** - Medium priority 436 437 Provide a mock handler that queues pre-defined responses and exceptions for testing HTTP client code without network calls. Support request inspection and assertion. 438 ··· 451 452 ## Feature Enhancements 453 454 + ### 21. ✅ Add Result-based error_for_status variant 455 + 456 + **Status: IMPLEMENTED** in `lib/response.ml` 457 458 Provide check_status() that returns (Response.t, Error.error) result instead of raising, enabling functional error handling alongside the existing raise_for_status. 459 460 **Cross-Language Consensus:** 2 libraries 461 **Source Libraries:** reqwest, grequests 462 463 + **Implementation:** 464 + - `Response.check_status : t -> (t, Error.t) result` 465 + - Returns `Ok response` for 2xx status codes 466 + - Returns `Error (Http_error {...})` for 4xx/5xx status codes 467 + - Complements existing `raise_for_status` for functional style 468 469 + ### 22. ❌ Add request body rewind support for retries 470 471 + **Status: NOT IMPLEMENTED** - Medium priority 472 473 Track whether request bodies are rewindable (seekable) and automatically replay them on retry. Raise clear error when retry requires body replay but body is a consumed stream. 474 ··· 488 489 ## Architectural Improvements 490 491 + ### 23. ❌ Add event listener/hooks for observability 492 + 493 + **Status: NOT IMPLEMENTED** - Low priority 494 495 Provide comprehensive event hooks at each request lifecycle stage: DNS start/end, connection start/end, TLS handshake, request headers/body sent, response headers/body received. Enable metrics and tracing integration. 496 ··· 510 511 ## Feature Enhancements 512 513 + ### 24. ✅ Add XSRF/CSRF token handling 514 + 515 + **Status: IMPLEMENTED** in `lib/requests.ml` 516 517 Automatically read CSRF token from a configurable cookie name (default: XSRF-TOKEN) and add it to request headers (default: X-XSRF-TOKEN) for same-origin requests. 518 519 **Cross-Language Consensus:** 1 libraries 520 **Source Libraries:** axios 521 522 + **Implementation:** 523 + - `xsrf_cookie_name` parameter (default: `Some "XSRF-TOKEN"`) 524 + - `xsrf_header_name` parameter (default: `"X-XSRF-TOKEN"`) 525 + - `apply_xsrf_token` function reads cookie and injects header 526 + - Set `xsrf_cookie_name` to `None` to disable 527 + - Applied automatically in `make_request_internal` 528 529 530 --- 531 532 ## Architectural Improvements 533 534 + ### 25. ✅ Add decompression bomb protection 535 + 536 + **Status: IMPLEMENTED** in `lib/http_client.ml` 537 538 Enforce configurable compression ratio limits (e.g., 100:1) to detect and abort decompression bomb attacks. Track compressed vs decompressed bytes during streaming. 539 540 **Cross-Language Consensus:** 1 libraries 541 **Source Libraries:** axios 542 543 + **Implementation:** 544 + - `check_decompression_limits` function validates size and ratio 545 + - `max_decompressed_size` in `Response_limits` (absolute size limit) 546 + - `max_compression_ratio` in `Response_limits` (ratio limit) 547 + - `Error.Decompression_bomb` error type with limit and ratio info 548 + - Applied during gzip/deflate/zlib decompression 549 550 551 --- 552 553 ## Feature Enhancements 554 555 + ### 26. ✅ Add HTTP version to response metadata 556 + 557 + **Status: IMPLEMENTED** in `lib/http_read.ml` 558 559 Expose the HTTP version used for the response (HTTP/1.0, HTTP/1.1, HTTP/2) in Response type. Useful for debugging protocol negotiation and monitoring HTTP/2 adoption. 560 561 **Cross-Language Consensus:** 3 libraries 562 **Source Libraries:** reqwest, surf, curl-rust 563 564 + **Implementation:** 565 + - `http_version` type: `HTTP_1_0 | HTTP_1_1` 566 + - `http_version_to_string` for display 567 + - `status_line` returns `(http_version, status_code)` tuple 568 + - `response` function returns `(http_version, status, headers, body)` 569 + - `stream_response` includes `http_version` field 570 571 + ### 27. ❌ Add Unix domain socket support 572 573 + **Status: NOT IMPLEMENTED** - Low priority 574 575 Enable connecting to local services via Unix domain sockets (e.g., Docker daemon at /var/run/docker.sock) without TCP overhead. 576 ··· 590 591 ## Security & Spec Compliance 592 593 + ### 28. ✅ Restrict URL protocols to HTTP/HTTPS only 594 + 595 + **Status: IMPLEMENTED** in `lib/requests.ml` and `lib/one.ml` 596 597 Explicitly whitelist only http:// and https:// protocols to prevent protocol smuggling attacks via file://, ftp://, gopher://, etc. Reject invalid protocols early in URL parsing. 598 599 **Cross-Language Consensus:** 1 libraries 600 **Source Libraries:** buzz 601 602 + **Implementation:** 603 + - `allowed_redirect_schemes = ["http"; "https"]` 604 + - `validate_redirect_url` checks scheme on redirects 605 + - `Error.Invalid_redirect` raised for disallowed schemes 606 + - Applied during redirect following to prevent SSRF 607 608 609 --- 610 611 ## Feature Enhancements 612 613 + ### 29. ✅ Add path parameter templating 614 + 615 + **Status: IMPLEMENTED** in `lib/requests.ml` 616 617 Support URL templates like /users/{id}/posts/{post_id} with placeholder substitution from a parameter map. Automatically URL-encode substituted values. 618 ··· 622 **Cross-Language Consensus:** 3 libraries 623 **Source Libraries:** sling, resty, req (Go) 624 625 + **Implementation:** 626 + - `substitute_path_params` function for RFC 6570 templating 627 + - `path_params:(string * string) list` parameter on all request methods 628 + - Automatic URL encoding via `Uri.pct_encode` 629 + - Applied before base URL resolution in request pipeline 630 + - Example: `get ~path_params:[("id", "123")] "/users/{id}"` 631 632 633 --- 634 635 ## Architectural Improvements 636 637 + ### 30. ✅ Add default User-Agent header 638 + 639 + **Status: IMPLEMENTED** in `lib/version.ml` 640 641 Set a default User-Agent identifying the library (ocaml-requests/VERSION) unless user provides one. Helps server-side debugging and follows HTTP best practices. 642 ··· 646 **Cross-Language Consensus:** 5 libraries 647 **Source Libraries:** needle, resty, requests (Go), okhttp, guzzle 648 649 + **Implementation:** 650 + - New `Version` module with library version info 651 + - `Version.user_agent` returns `"ocaml-requests/0.1.0 (OCaml X.XX.X)"` 652 + - Applied automatically if not already set in `make_request_internal` 653 + - Can be overridden via explicit User-Agent header 654