anil.recoil.org / ocaml-requests
A batteries included HTTP/1.1 client in OCaml
fork atom

more

anil.recoil.org bb08e520 08bff2d3

-654
-654
RECOMMENDATIONS.md
··· 1 - # OCaml HTTP Client Library - Priority Feature Recommendations 2 - 3 - > Auto-generated summary of recommendations from analyzing HTTP client libraries across multiple languages. 4 - 5 - ## Implementation Status Summary 6 - 7 - | Status | Count | Description | 8 - |--------|-------|-------------| 9 - | ✅ Implemented | 15 | Fully implemented and tested | 10 - | 🔧 Partial | 1 | Partially implemented | 11 - | ❌ Not Started | 14 | Not yet implemented | 12 - 13 - ### Implemented Features 14 - 15 - | # | Feature | Location | 16 - |---|---------|----------| 17 - | 1 | Strip sensitive headers on cross-origin redirects | `lib/requests.ml`, `lib/one.ml` | 18 - | 3 | Enforce HTTPS for sensitive authentication | `lib/auth.ml` (`apply_secure`, `Insecure_auth` error) | 19 - | 4 | Content-Length vs Transfer-Encoding precedence | `lib/http_read.ml` (RFC 9112 warning) | 20 - | 5 | Cap maximum Retry-After delay | `lib/retry.ml` (`backoff_max` parameter) | 21 - | 11 | Base URL support with relative path resolution | `lib/requests.ml` (`base_url`, `resolve_url`) | 22 - | 12 | Detailed timing metrics | `lib/timing.ml` (new module) | 23 - | 13 | Stale connection detection | `ocaml-conpool/lib/conpool.ml` (`is_healthy`) | 24 - | 14 | Custom retry condition predicates | `lib/retry.ml` (`response_predicate`, `exception_predicate`) | 25 - | 19 | Link header parsing for pagination | `lib/link.ml` (new module) | 26 - | 21 | Result-based error_for_status variant | `lib/response.ml` (`check_status`) | 27 - | 24 | XSRF/CSRF token handling | `lib/requests.ml` (`apply_xsrf_token`) | 28 - | 25 | Decompression bomb protection | `lib/http_client.ml` (`check_decompression_limits`) | 29 - | 26 | HTTP version in response metadata | `lib/http_read.ml` (`http_version` type) | 30 - | 28 | Restrict URL protocols to HTTP/HTTPS | `lib/requests.ml`, `lib/one.ml` (`validate_redirect_url`) | 31 - | 29 | Path parameter templating | `lib/requests.ml` (`substitute_path_params`) | 32 - | 30 | Default User-Agent header | `lib/version.ml`, applied in `lib/requests.ml` | 33 - 34 - ### Not Yet Implemented 35 - 36 - | # | Feature | Priority | 37 - |---|---------|----------| 38 - | 2 | Certificate/public key pinning | High | 39 - | 6 | HTTP proxy support | High | 40 - | 7 | Request/response middleware system | Medium | 41 - | 8 | Progress callbacks | Medium | 42 - | 9 | Request cancellation | Medium | 43 - | 10 | Concurrent request batching | Low | 44 - | 15 | Charset/encoding detection | Low | 45 - | 16 | Redirect policy customization | Low | 46 - | 17 | TCP keep-alive configuration | Low | 47 - | 18 | URL sanitization for error messages | Low | 48 - | 20 | Mock handler for testing | Medium | 49 - | 22 | Request body rewind for retries | Medium | 50 - | 23 | Event listener/hooks | Low | 51 - | 27 | Unix domain socket support | Low | 52 - 53 - --- 54 - 55 - ## Executive Summary 56 - 57 - This comprehensive analysis synthesizes recommendations from 45+ HTTP client libraries across 10 programming languages (Python, JavaScript, Go, Rust, Haskell, PHP, Java, Swift, C++, Bash). The most consistent themes are: (1) Security features like certificate pinning, cross-origin header stripping, and HTTPS enforcement for sensitive auth; (2) Middleware/interceptor systems for extensibility; (3) HTTP proxy support; (4) Progress callbacks for uploads/downloads; (5) Request cancellation capabilities; (6) Detailed timing metrics; (7) Retry logic improvements including Retry-After capping and custom predicates. The prioritization emphasizes security fixes that could lead to credential leakage, followed by missing features that block enterprise adoption (proxy support), then enhancements that improve developer experience. 58 - 59 - 60 - --- 61 - 62 - ## Security & Spec Compliance 63 - 64 - ### 1. ✅ Strip sensitive headers on cross-origin redirects 65 - 66 - **Status: IMPLEMENTED** in `lib/requests.ml` and `lib/one.ml` 67 - 68 - When redirecting to a different origin (host/port/scheme), automatically strip Authorization, Cookie, Proxy-Authorization, and WWW-Authenticate headers to prevent credential leakage to unintended domains. Also strip headers on HTTPS→HTTP protocol downgrade. 69 - 70 - **RFC References:** 71 - - RFC 9110 Section 15.4 (Redirection) 72 - 73 - **Cross-Language Consensus:** 8 libraries 74 - **Source Libraries:** reqwest, got, node-fetch, okhttp, axios, superagent, http-client, needle 75 - 76 - **Implementation:** 77 - - `same_origin` function compares scheme and host 78 - - `strip_sensitive_headers` removes Authorization, Cookie, Proxy-Authorization, WWW-Authenticate 79 - - Applied during redirect handling in `make_with_redirects` 80 - 81 - ### 2. ❌ Add certificate/public key pinning support 82 - 83 - **Status: NOT IMPLEMENTED** - High priority for enterprise security 84 - 85 - Allow pinning certificates or public keys (SPKI hashes) to protect against compromised certificate authorities. Support per-host pinning configuration with wildcard patterns. 86 - 87 - **RFC References:** 88 - - RFC 7469 (Public Key Pinning for HTTP) 89 - 90 - **Cross-Language Consensus:** 4 libraries 91 - **Source Libraries:** okhttp, swifthttp, urllib3, alamofire 92 - 93 - **Affected Files:** 94 - - `lib/requests.ml` 95 - - `lib/requests.mli` 96 - 97 - **Implementation Notes:** 98 - Add certificate_pins parameter to create() accepting (hostname * sha256_hash list) list. Verify pins during TLS handshake by extracting SPKI from certificate chain. Provide helpful error messages showing expected vs actual hashes. 99 - 100 - ### 3. ✅ Enforce HTTPS for sensitive authentication methods 101 - 102 - **Status: IMPLEMENTED** in `lib/auth.ml` 103 - 104 - Reject Basic, Bearer, and Digest authentication over unencrypted HTTP connections by default. Provide explicit opt-out flag (allow_insecure_auth) for testing environments. 105 - 106 - **RFC References:** 107 - - RFC 7617 Section 4 (Basic Authentication Security) 108 - - RFC 6750 Section 5.1 (Bearer Token Security) 109 - 110 - **Cross-Language Consensus:** 2 libraries 111 - **Source Libraries:** req (Haskell), urllib3 112 - 113 - **Implementation:** 114 - - `Auth.apply_secure` validates transport security before applying auth 115 - - `Auth.validate_secure_transport` checks if HTTPS is required 116 - - `Error.Insecure_auth` error type for clear error messages 117 - - `allow_insecure_auth` parameter in request functions for testing 118 - 119 - ### 4. ✅ Validate Content-Length vs Transfer-Encoding precedence 120 - 121 - **Status: IMPLEMENTED** in `lib/http_read.ml` 122 - 123 - Per RFC 9112, when both Content-Length and Transfer-Encoding headers are present, ignore Content-Length. This prevents HTTP request smuggling attacks. 124 - 125 - **RFC References:** 126 - - RFC 9112 Section 6.3 (Message Body Length) 127 - 128 - **Cross-Language Consensus:** 2 libraries 129 - **Source Libraries:** hyper, http-streams 130 - 131 - **Implementation:** 132 - - Transfer-Encoding: chunked takes precedence over Content-Length 133 - - Warning logged when both headers present (potential attack indicator) 134 - - Per RFC 9112 Section 6.3 compliance 135 - 136 - ### 5. ✅ Cap maximum Retry-After delay to prevent DoS 137 - 138 - **Status: IMPLEMENTED** in `lib/retry.ml` 139 - 140 - Implement a configurable maximum backoff time (default 120s) that caps server-specified Retry-After values, preventing malicious servers from causing indefinite client blocking. 141 - 142 - **RFC References:** 143 - - RFC 9110 Section 10.2.3 (Retry-After) 144 - 145 - **Cross-Language Consensus:** 2 libraries 146 - **Source Libraries:** req (Go), got 147 - 148 - **Implementation:** 149 - - `backoff_max` field in `Retry.config` (default: 120.0 seconds) 150 - - `parse_retry_after` function accepts `?backoff_max` parameter 151 - - Delays exceeding cap are logged as warnings and capped 152 - 153 - 154 - --- 155 - 156 - ## Feature Enhancements 157 - 158 - ### 6. ❌ Add HTTP proxy support with environment variable detection 159 - 160 - **Status: NOT IMPLEMENTED** - High priority, see `spec/PROXY_IMPLEMENTATION_PLAN.md` 161 - 162 - Implement HTTP/HTTPS proxy support including: proxy URL configuration, authentication (Basic), CONNECT method for HTTPS tunneling, and automatic detection from HTTP_PROXY, HTTPS_PROXY, NO_PROXY environment variables. 163 - 164 - **RFC References:** 165 - - RFC 9110 Section 9.3.6 (CONNECT) 166 - - RFC 7235 (HTTP Authentication) 167 - 168 - **Cross-Language Consensus:** 7 libraries 169 - **Source Libraries:** reqwest, axios, guzzle, http-client, requests (Go), resty, isahc 170 - 171 - **Affected Files:** 172 - - `lib/requests.ml` 173 - - `lib/requests.mli` 174 - - `lib/http_client.ml` 175 - - `lib/one.ml` 176 - 177 - **Implementation Notes:** 178 - A comprehensive implementation plan exists in `spec/PROXY_IMPLEMENTATION_PLAN.md` with 8 phases covering proxy configuration, HTTP CONNECT tunneling, environment variable detection, NO_PROXY pattern matching, and proxy authentication. 179 - 180 - ### 7. ❌ Implement request/response middleware system 181 - 182 - **Status: NOT IMPLEMENTED** - Medium priority 183 - 184 - Add a composable middleware chain that allows intercepting and transforming requests before sending and responses after receiving. Enable cross-cutting concerns like logging, metrics, request signing, and custom caching. 185 - 186 - **Cross-Language Consensus:** 9 libraries 187 - **Source Libraries:** surf, got, axios, guzzle, okhttp, resty, buzz, moya, req (Go) 188 - 189 - **Affected Files:** 190 - - `lib/requests.ml` 191 - - `lib/requests.mli` 192 - 193 - **Implementation Notes:** 194 - Define type middleware = Request.t -> (Request.t -> Response.t) -> Response.t. Store middleware list in session. Execute chain: apply each middleware wrapping the next. Provide default middleware for retry and redirect as replaceable components. 195 - 196 - ### 8. ❌ Add progress callbacks for uploads and downloads 197 - 198 - **Status: NOT IMPLEMENTED** - Medium priority 199 - 200 - Implement optional progress callbacks that report bytes transferred, total size (if known), and percent complete. Rate-limit callbacks to prevent performance degradation. 201 - 202 - **Cross-Language Consensus:** 9 libraries 203 - **Source Libraries:** curl-rust, got, axios, guzzle, resty, req (Go), kingfisher, net (Swift), http-request (Java) 204 - 205 - **Affected Files:** 206 - - `lib/requests.ml` 207 - - `lib/body.ml` 208 - - `lib/http_client.ml` 209 - - `lib/one.ml` 210 - 211 - **Implementation Notes:** 212 - Add optional progress:(transferred:int64 -> total:int64 option -> unit) parameter. Track bytes during streaming read/write. Throttle callbacks (e.g., max 3/second or every 64KB). Support both upload and download progress. 213 - 214 - ### 9. ❌ Add request cancellation support 215 - 216 - **Status: NOT IMPLEMENTED** - Medium priority (Eio provides natural cancellation via switches) 217 - 218 - Allow programmatic cancellation of in-flight requests. Integrate with Eio's cancellation system while providing explicit cancel() function or cancel token pattern. 219 - 220 - **Cross-Language Consensus:** 6 libraries 221 - **Source Libraries:** got, axios, okhttp, needle, just, hyper 222 - 223 - **Affected Files:** 224 - - `lib/requests.ml` 225 - - `lib/requests.mli` 226 - - `lib/one.ml` 227 - 228 - **Implementation Notes:** 229 - Return (Response.t * cancel_token) from request functions where cancel_token allows aborting the request. Internally use Eio.Cancel or Switch.fail to terminate the fiber. Raise Cancelled error on cancellation. 230 - 231 - ### 10. ❌ Add concurrent request batching API (map/imap) 232 - 233 - **Status: NOT IMPLEMENTED** - Low priority (Eio.Fiber.all/both provides this) 234 - 235 - Provide map() for executing multiple requests concurrently and returning results in order, and imap() for yielding responses as they complete. Include configurable concurrency limits and batch timeout. 236 - 237 - **Cross-Language Consensus:** 2 libraries 238 - **Source Libraries:** grequests, aiohttp 239 - 240 - **Affected Files:** 241 - - `lib/requests.ml` 242 - - `lib/requests.mli` 243 - 244 - **Implementation Notes:** 245 - Implement using Eio.Fiber.all for map, Eio.Stream for imap. Add max_concurrent parameter using Eio.Semaphore. Provide exception_handler callback for partial failure handling. Add batch_timeout for overall operation limit. 246 - 247 - ### 11. ✅ Add base URL support with relative path resolution 248 - 249 - **Status: IMPLEMENTED** in `lib/requests.ml` 250 - 251 - Allow configuring a base URL on the session that is prepended to relative request URLs. Use proper RFC 3986 URI resolution for correct path joining. 252 - 253 - **RFC References:** 254 - - RFC 3986 Section 5 (Reference Resolution) 255 - 256 - **Cross-Language Consensus:** 5 libraries 257 - **Source Libraries:** surf, axios, got, sling, req (Go) 258 - 259 - **Implementation:** 260 - - `base_url` parameter in `create` function 261 - - `is_relative_url` checks if URL has scheme 262 - - `resolve_url` uses `Uri.resolve` for RFC 3986 compliant resolution 263 - - Trailing slashes automatically normalized 264 - 265 - 266 - --- 267 - 268 - ## Architectural Improvements 269 - 270 - ### 12. ✅ Add detailed timing metrics to responses 271 - 272 - **Status: IMPLEMENTED** in new `lib/timing.ml` module 273 - 274 - Expose granular timing breakdown: DNS lookup time, TCP connection time, TLS handshake time, time to first byte (TTFB), download time, and total time. Include connection reuse tracking. 275 - 276 - **Cross-Language Consensus:** 7 libraries 277 - **Source Libraries:** curl-rust, got, resty, req (Go), guzzle, okhttp, isahc 278 - 279 - **Implementation:** 280 - - New `Timing` module with detailed metrics type 281 - - Phases: `dns_lookup`, `tcp_connect`, `tls_handshake`, `request_sent`, `time_to_first_byte`, `content_transfer`, `total` 282 - - `timer` type for incremental collection during requests 283 - - `connection_time` and `server_time` computed metrics 284 - - Pretty-printing with `pp` and `to_string` 285 - 286 - ### 13. ✅ Add automatic retry on stale connection detection 287 - 288 - **Status: IMPLEMENTED** in `ocaml-conpool` library 289 - 290 - Detect when a pooled connection was closed by the server between requests (NoResponseDataReceived) and automatically retry with a fresh connection. Mark this error as always retryable. 291 - 292 - **Cross-Language Consensus:** 2 libraries 293 - **Source Libraries:** http-client (Haskell), okhttp 294 - 295 - **Implementation:** 296 - The `ocaml-conpool` library provides comprehensive stale connection detection via: 297 - - `is_healthy` function checking age, idle time, and use count 298 - - `max_idle_time` configuration (default: 60s) 299 - - `max_connection_lifetime` configuration (default: 300s) 300 - - `max_connection_uses` configuration (optional) 301 - - Custom `health_check` callback support 302 - - Pool `validate` function runs health checks before connection reuse 303 - 304 - ### 14. ✅ Add custom retry condition predicates 305 - 306 - **Status: IMPLEMENTED** in `lib/retry.ml` 307 - 308 - Allow users to provide custom retry decision functions beyond status codes and error types. Separate predicates for response-based retry (status/headers) and exception-based retry. 309 - 310 - **Cross-Language Consensus:** 4 libraries 311 - **Source Libraries:** resty, got, req (Go), req (Haskell) 312 - 313 - **Implementation:** 314 - - `response_predicate` type: `Method.t -> int -> Headers.t -> bool` 315 - - `exception_predicate` type: `exn -> bool` 316 - - `retry_response` and `retry_exception` fields in `Retry.config` 317 - - `should_retry_response` checks both built-in rules and custom predicates 318 - - `should_retry_exn` for exception-based retry decisions 319 - - Examples in module documentation 320 - 321 - 322 - --- 323 - 324 - ## Feature Enhancements 325 - 326 - ### 15. ❌ Add response charset/encoding detection and conversion 327 - 328 - **Status: NOT IMPLEMENTED** - Low priority 329 - 330 - Automatically detect character encoding from Content-Type charset parameter and convert response text to UTF-8. Provide BOM sniffing fallback for encoding detection. 331 - 332 - **RFC References:** 333 - - RFC 9110 Section 8.3.1 (Content-Type) 334 - 335 - **Cross-Language Consensus:** 6 libraries 336 - **Source Libraries:** reqwest, surf, isahc, needle, got, http-request (Java) 337 - 338 - **Affected Files:** 339 - - `lib/response.ml` 340 - - `lib/response.mli` 341 - - `lib/http_read.ml` 342 - 343 - **Implementation Notes:** 344 - Parse charset from Content-Type in Mime module. Add Response.text_utf8 that detects and converts encoding. Use uutf library for encoding conversion. Return raw bytes on conversion failure with access to detected charset. 345 - 346 - ### 16. ❌ Add redirect policy customization 347 - 348 - **Status: NOT IMPLEMENTED** - Low priority 349 - 350 - Allow custom redirect policies with access to attempt status, URL, and previous URLs. Enable controlling method preservation (307/308 vs 301/302), header stripping, and cross-protocol redirect behavior. 351 - 352 - **RFC References:** 353 - - RFC 9110 Section 15.4 (Redirection 3xx) 354 - 355 - **Cross-Language Consensus:** 5 libraries 356 - **Source Libraries:** reqwest, needle, node-fetch, guzzle, http-client (Haskell) 357 - 358 - **Affected Files:** 359 - - `lib/requests.ml` 360 - - `lib/requests.mli` 361 - 362 - **Implementation Notes:** 363 - Add redirect_policy type with hooks: before_redirect:(request -> response -> uri -> bool), strip_headers:(origin_changed -> header_name -> bool). Implement RFC-correct 301/302 POST→GET conversion. Provide presets: follow_all, same_host_only, same_protocol_only. 364 - 365 - 366 - --- 367 - 368 - ## Architectural Improvements 369 - 370 - ### 17. ❌ Add TCP keep-alive configuration 371 - 372 - **Status: NOT IMPLEMENTED** - Low priority 373 - 374 - Expose TCP keep-alive settings including enable/disable, interval, and retry count. This helps detect broken connections and maintain long-lived connections through firewalls/NAT. 375 - 376 - **Cross-Language Consensus:** 3 libraries 377 - **Source Libraries:** reqwest, isahc, curl-rust 378 - 379 - **Affected Files:** 380 - - `lib/requests.mli` 381 - - `lib/http_client.ml` 382 - 383 - **Implementation Notes:** 384 - Add tcp_keepalive_interval and tcp_keepalive option to session config. Configure socket options via Eio.Net during connection establishment. Default to reasonable values (e.g., 60s interval). 385 - 386 - 387 - --- 388 - 389 - ## Feature Enhancements 390 - 391 - ### 18. ❌ Add URL sanitization for error messages 392 - 393 - **Status: NOT IMPLEMENTED** - Low priority 394 - 395 - Automatically sanitize URLs in error messages to prevent leakage of credentials embedded in query parameters (API keys, tokens). Provide without_url() method to strip URLs entirely. 396 - 397 - **Cross-Language Consensus:** 1 libraries 398 - **Source Libraries:** reqwest 399 - 400 - **Affected Files:** 401 - - `lib/error.ml` 402 - - `lib/error.mli` 403 - 404 - **Implementation Notes:** 405 - Extend existing sanitize_url function to handle query parameter credentials. Add Error.without_url to remove URL entirely from error. Apply sanitization automatically in Error.to_string and pp functions. 406 - 407 - ### 19. ✅ Add Link header parsing for pagination 408 - 409 - **Status: IMPLEMENTED** in new `lib/link.ml` module 410 - 411 - Parse RFC 8288 Link headers to extract relation types (next, prev, first, last) for API pagination. Provide helpers like Response.next_url for common patterns. 412 - 413 - **RFC References:** 414 - - RFC 8288 (Web Linking) 415 - 416 - **Cross-Language Consensus:** 3 libraries 417 - **Source Libraries:** got, superagent, just 418 - 419 - **Implementation:** 420 - - New `Link` module with RFC 8288 compliant parsing 421 - - `Link.t` type with `uri`, `rel`, `title`, `media_type`, `hreflang`, `params` 422 - - `parse` function handles comma-separated links with quoted values 423 - - `from_headers` extracts Link header from response headers 424 - - `pagination` returns `(first, prev, next, last)` tuple 425 - - `next_url`, `prev_url`, `has_next` convenience functions 426 - - Pretty-printing with `pp` and `to_string` 427 - 428 - 429 - --- 430 - 431 - ## Architectural Improvements 432 - 433 - ### 20. ❌ Add mock handler for testing 434 - 435 - **Status: NOT IMPLEMENTED** - Medium priority 436 - 437 - Provide a mock handler that queues pre-defined responses and exceptions for testing HTTP client code without network calls. Support request inspection and assertion. 438 - 439 - **Cross-Language Consensus:** 3 libraries 440 - **Source Libraries:** guzzle, okhttp, resty 441 - 442 - **Affected Files:** 443 - - `lib/http_client.ml` 444 - - `test/` 445 - 446 - **Implementation Notes:** 447 - Create Mock module with queue of responses/exceptions. Replace actual HTTP client in test mode. Track all requests made for assertions. Allow configuring delays for timeout testing. 448 - 449 - 450 - --- 451 - 452 - ## Feature Enhancements 453 - 454 - ### 21. ✅ Add Result-based error_for_status variant 455 - 456 - **Status: IMPLEMENTED** in `lib/response.ml` 457 - 458 - Provide check_status() that returns (Response.t, Error.error) result instead of raising, enabling functional error handling alongside the existing raise_for_status. 459 - 460 - **Cross-Language Consensus:** 2 libraries 461 - **Source Libraries:** reqwest, grequests 462 - 463 - **Implementation:** 464 - - `Response.check_status : t -> (t, Error.t) result` 465 - - Returns `Ok response` for 2xx status codes 466 - - Returns `Error (Http_error {...})` for 4xx/5xx status codes 467 - - Complements existing `raise_for_status` for functional style 468 - 469 - ### 22. ❌ Add request body rewind support for retries 470 - 471 - **Status: NOT IMPLEMENTED** - Medium priority 472 - 473 - Track whether request bodies are rewindable (seekable) and automatically replay them on retry. Raise clear error when retry requires body replay but body is a consumed stream. 474 - 475 - **Cross-Language Consensus:** 4 libraries 476 - **Source Libraries:** isahc, curl-rust, surf, node-fetch 477 - 478 - **Affected Files:** 479 - - `lib/body.ml` 480 - - `lib/retry.ml` 481 - - `lib/error.ml` 482 - 483 - **Implementation Notes:** 484 - Add is_rewindable field to Body.t. For Body.String/Form, rewindable=true. For Stream without seek, rewindable=false. In retry, check rewindability; raise Body_not_rewindable if retry needed but body consumed. 485 - 486 - 487 - --- 488 - 489 - ## Architectural Improvements 490 - 491 - ### 23. ❌ Add event listener/hooks for observability 492 - 493 - **Status: NOT IMPLEMENTED** - Low priority 494 - 495 - Provide comprehensive event hooks at each request lifecycle stage: DNS start/end, connection start/end, TLS handshake, request headers/body sent, response headers/body received. Enable metrics and tracing integration. 496 - 497 - **Cross-Language Consensus:** 3 libraries 498 - **Source Libraries:** okhttp, resty, http-streams 499 - 500 - **Affected Files:** 501 - - `lib/requests.ml` 502 - - `lib/requests.mli` 503 - - `lib/http_client.ml` 504 - 505 - **Implementation Notes:** 506 - Define event types: Dns_start, Dns_end, Connect_start, etc. Add event_listener callback to session config. Emit events at appropriate points in http_client.ml. Include timing and context in each event. 507 - 508 - 509 - --- 510 - 511 - ## Feature Enhancements 512 - 513 - ### 24. ✅ Add XSRF/CSRF token handling 514 - 515 - **Status: IMPLEMENTED** in `lib/requests.ml` 516 - 517 - Automatically read CSRF token from a configurable cookie name (default: XSRF-TOKEN) and add it to request headers (default: X-XSRF-TOKEN) for same-origin requests. 518 - 519 - **Cross-Language Consensus:** 1 libraries 520 - **Source Libraries:** axios 521 - 522 - **Implementation:** 523 - - `xsrf_cookie_name` parameter (default: `Some "XSRF-TOKEN"`) 524 - - `xsrf_header_name` parameter (default: `"X-XSRF-TOKEN"`) 525 - - `apply_xsrf_token` function reads cookie and injects header 526 - - Set `xsrf_cookie_name` to `None` to disable 527 - - Applied automatically in `make_request_internal` 528 - 529 - 530 - --- 531 - 532 - ## Architectural Improvements 533 - 534 - ### 25. ✅ Add decompression bomb protection 535 - 536 - **Status: IMPLEMENTED** in `lib/http_client.ml` 537 - 538 - Enforce configurable compression ratio limits (e.g., 100:1) to detect and abort decompression bomb attacks. Track compressed vs decompressed bytes during streaming. 539 - 540 - **Cross-Language Consensus:** 1 libraries 541 - **Source Libraries:** axios 542 - 543 - **Implementation:** 544 - - `check_decompression_limits` function validates size and ratio 545 - - `max_decompressed_size` in `Response_limits` (absolute size limit) 546 - - `max_compression_ratio` in `Response_limits` (ratio limit) 547 - - `Error.Decompression_bomb` error type with limit and ratio info 548 - - Applied during gzip/deflate/zlib decompression 549 - 550 - 551 - --- 552 - 553 - ## Feature Enhancements 554 - 555 - ### 26. ✅ Add HTTP version to response metadata 556 - 557 - **Status: IMPLEMENTED** in `lib/http_read.ml` 558 - 559 - Expose the HTTP version used for the response (HTTP/1.0, HTTP/1.1, HTTP/2) in Response type. Useful for debugging protocol negotiation and monitoring HTTP/2 adoption. 560 - 561 - **Cross-Language Consensus:** 3 libraries 562 - **Source Libraries:** reqwest, surf, curl-rust 563 - 564 - **Implementation:** 565 - - `http_version` type: `HTTP_1_0 | HTTP_1_1` 566 - - `http_version_to_string` for display 567 - - `status_line` returns `(http_version, status_code)` tuple 568 - - `response` function returns `(http_version, status, headers, body)` 569 - - `stream_response` includes `http_version` field 570 - 571 - ### 27. ❌ Add Unix domain socket support 572 - 573 - **Status: NOT IMPLEMENTED** - Low priority 574 - 575 - Enable connecting to local services via Unix domain sockets (e.g., Docker daemon at /var/run/docker.sock) without TCP overhead. 576 - 577 - **Cross-Language Consensus:** 2 libraries 578 - **Source Libraries:** curl-rust, http-streams 579 - 580 - **Affected Files:** 581 - - `lib/http_client.ml` 582 - - `lib/requests.ml` 583 - - `lib/one.ml` 584 - 585 - **Implementation Notes:** 586 - Add unix:// URL scheme support. Use Eio_unix for socket connection. Handle Host header correctly (use socket path or override). Add to connection pool keyed by socket path. 587 - 588 - 589 - --- 590 - 591 - ## Security & Spec Compliance 592 - 593 - ### 28. ✅ Restrict URL protocols to HTTP/HTTPS only 594 - 595 - **Status: IMPLEMENTED** in `lib/requests.ml` and `lib/one.ml` 596 - 597 - Explicitly whitelist only http:// and https:// protocols to prevent protocol smuggling attacks via file://, ftp://, gopher://, etc. Reject invalid protocols early in URL parsing. 598 - 599 - **Cross-Language Consensus:** 1 libraries 600 - **Source Libraries:** buzz 601 - 602 - **Implementation:** 603 - - `allowed_redirect_schemes = ["http"; "https"]` 604 - - `validate_redirect_url` checks scheme on redirects 605 - - `Error.Invalid_redirect` raised for disallowed schemes 606 - - Applied during redirect following to prevent SSRF 607 - 608 - 609 - --- 610 - 611 - ## Feature Enhancements 612 - 613 - ### 29. ✅ Add path parameter templating 614 - 615 - **Status: IMPLEMENTED** in `lib/requests.ml` 616 - 617 - Support URL templates like /users/{id}/posts/{post_id} with placeholder substitution from a parameter map. Automatically URL-encode substituted values. 618 - 619 - **RFC References:** 620 - - RFC 6570 (URI Template) 621 - 622 - **Cross-Language Consensus:** 3 libraries 623 - **Source Libraries:** sling, resty, req (Go) 624 - 625 - **Implementation:** 626 - - `substitute_path_params` function for RFC 6570 templating 627 - - `path_params:(string * string) list` parameter on all request methods 628 - - Automatic URL encoding via `Uri.pct_encode` 629 - - Applied before base URL resolution in request pipeline 630 - - Example: `get ~path_params:[("id", "123")] "/users/{id}"` 631 - 632 - 633 - --- 634 - 635 - ## Architectural Improvements 636 - 637 - ### 30. ✅ Add default User-Agent header 638 - 639 - **Status: IMPLEMENTED** in `lib/version.ml` 640 - 641 - Set a default User-Agent identifying the library (ocaml-requests/VERSION) unless user provides one. Helps server-side debugging and follows HTTP best practices. 642 - 643 - **RFC References:** 644 - - RFC 9110 Section 10.1.5 (User-Agent) 645 - 646 - **Cross-Language Consensus:** 5 libraries 647 - **Source Libraries:** needle, resty, requests (Go), okhttp, guzzle 648 - 649 - **Implementation:** 650 - - New `Version` module with library version info 651 - - `Version.user_agent` returns `"ocaml-requests/0.1.0 (OCaml X.XX.X)"` 652 - - Applied automatically if not already set in `make_request_internal` 653 - - Can be overridden via explicit User-Agent header 654 -
···