anil.recoil.org / ocaml-requests
A batteries included HTTP/1.1 client in OCaml
fork atom

features

anil.recoil.org 42037660 60979f57

+224 -128
unified split
+224 -128
RECOMMENDATIONS.md
··· 2 2 3 3 > Auto-generated summary of recommendations from analyzing HTTP client libraries across multiple languages. 4 4 5 + ## Implementation Status Summary 6 + 7 + | Status | Count | Description | 8 + |--------|-------|-------------| 9 + | ✅ Implemented | 15 | Fully implemented and tested | 10 + | 🔧 Partial | 1 | Partially implemented | 11 + | ❌ Not Started | 14 | Not yet implemented | 12 + 13 + ### Implemented Features 14 + 15 + | # | Feature | Location | 16 + |---|---------|----------| 17 + | 1 | Strip sensitive headers on cross-origin redirects | `lib/requests.ml`, `lib/one.ml` | 18 + | 3 | Enforce HTTPS for sensitive authentication | `lib/auth.ml` (`apply_secure`, `Insecure_auth` error) | 19 + | 4 | Content-Length vs Transfer-Encoding precedence | `lib/http_read.ml` (RFC 9112 warning) | 20 + | 5 | Cap maximum Retry-After delay | `lib/retry.ml` (`backoff_max` parameter) | 21 + | 11 | Base URL support with relative path resolution | `lib/requests.ml` (`base_url`, `resolve_url`) | 22 + | 12 | Detailed timing metrics | `lib/timing.ml` (new module) | 23 + | 13 | Stale connection detection | `ocaml-conpool/lib/conpool.ml` (`is_healthy`) | 24 + | 14 | Custom retry condition predicates | `lib/retry.ml` (`response_predicate`, `exception_predicate`) | 25 + | 19 | Link header parsing for pagination | `lib/link.ml` (new module) | 26 + | 21 | Result-based error_for_status variant | `lib/response.ml` (`check_status`) | 27 + | 24 | XSRF/CSRF token handling | `lib/requests.ml` (`apply_xsrf_token`) | 28 + | 25 | Decompression bomb protection | `lib/http_client.ml` (`check_decompression_limits`) | 29 + | 26 | HTTP version in response metadata | `lib/http_read.ml` (`http_version` type) | 30 + | 28 | Restrict URL protocols to HTTP/HTTPS | `lib/requests.ml`, `lib/one.ml` (`validate_redirect_url`) | 31 + | 29 | Path parameter templating | `lib/requests.ml` (`substitute_path_params`) | 32 + | 30 | Default User-Agent header | `lib/version.ml`, applied in `lib/requests.ml` | 33 + 34 + ### Not Yet Implemented 35 + 36 + | # | Feature | Priority | 37 + |---|---------|----------| 38 + | 2 | Certificate/public key pinning | High | 39 + | 6 | HTTP proxy support | High | 40 + | 7 | Request/response middleware system | Medium | 41 + | 8 | Progress callbacks | Medium | 42 + | 9 | Request cancellation | Medium | 43 + | 10 | Concurrent request batching | Low | 44 + | 15 | Charset/encoding detection | Low | 45 + | 16 | Redirect policy customization | Low | 46 + | 17 | TCP keep-alive configuration | Low | 47 + | 18 | URL sanitization for error messages | Low | 48 + | 20 | Mock handler for testing | Medium | 49 + | 22 | Request body rewind for retries | Medium | 50 + | 23 | Event listener/hooks | Low | 51 + | 27 | Unix domain socket support | Low | 52 + 53 + --- 54 + 5 55 ## Executive Summary 6 56 7 57 This comprehensive analysis synthesizes recommendations from 45+ HTTP client libraries across 10 programming languages (Python, JavaScript, Go, Rust, Haskell, PHP, Java, Swift, C++, Bash). The most consistent themes are: (1) Security features like certificate pinning, cross-origin header stripping, and HTTPS enforcement for sensitive auth; (2) Middleware/interceptor systems for extensibility; (3) HTTP proxy support; (4) Progress callbacks for uploads/downloads; (5) Request cancellation capabilities; (6) Detailed timing metrics; (7) Retry logic improvements including Retry-After capping and custom predicates. The prioritization emphasizes security fixes that could lead to credential leakage, followed by missing features that block enterprise adoption (proxy support), then enhancements that improve developer experience. ··· 11 61 12 62 ## Security & Spec Compliance 13 63 14 - ### 1. Strip sensitive headers on cross-origin redirects 64 + ### 1. ✅ Strip sensitive headers on cross-origin redirects 65 + 66 + **Status: IMPLEMENTED** in `lib/requests.ml` and `lib/one.ml` 15 67 16 68 When redirecting to a different origin (host/port/scheme), automatically strip Authorization, Cookie, Proxy-Authorization, and WWW-Authenticate headers to prevent credential leakage to unintended domains. Also strip headers on HTTPS→HTTP protocol downgrade. 17 69 ··· 21 73 **Cross-Language Consensus:** 8 libraries 22 74 **Source Libraries:** reqwest, got, node-fetch, okhttp, axios, superagent, http-client, needle 23 75 24 - **Affected Files:** 25 - - `lib/requests.ml` 26 - - `lib/one.ml` 27 - - `lib/http_client.ml` 76 + **Implementation:** 77 + - `same_origin` function compares scheme and host 78 + - `strip_sensitive_headers` removes Authorization, Cookie, Proxy-Authorization, WWW-Authenticate 79 + - Applied during redirect handling in `make_with_redirects` 28 80 29 - **Implementation Notes:** 30 - Implement same_origin check comparing scheme, host, and port. Create a list of sensitive headers to strip. Call strip_sensitive_headers before following any redirect where origin changes. 81 + ### 2. ❌ Add certificate/public key pinning support 31 82 32 - ### 2. Add certificate/public key pinning support 83 + **Status: NOT IMPLEMENTED** - High priority for enterprise security 33 84 34 85 Allow pinning certificates or public keys (SPKI hashes) to protect against compromised certificate authorities. Support per-host pinning configuration with wildcard patterns. 35 86 ··· 46 97 **Implementation Notes:** 47 98 Add certificate_pins parameter to create() accepting (hostname * sha256_hash list) list. Verify pins during TLS handshake by extracting SPKI from certificate chain. Provide helpful error messages showing expected vs actual hashes. 48 99 49 - ### 3. Enforce HTTPS for sensitive authentication methods 100 + ### 3. ✅ Enforce HTTPS for sensitive authentication methods 101 + 102 + **Status: IMPLEMENTED** in `lib/auth.ml` 50 103 51 104 Reject Basic, Bearer, and Digest authentication over unencrypted HTTP connections by default. Provide explicit opt-out flag (allow_insecure_auth) for testing environments. 52 105 ··· 57 110 **Cross-Language Consensus:** 2 libraries 58 111 **Source Libraries:** req (Haskell), urllib3 59 112 60 - **Affected Files:** 61 - - `lib/auth.ml` 62 - - `lib/auth.mli` 63 - - `lib/requests.ml` 64 - - `lib/one.ml` 113 + **Implementation:** 114 + - `Auth.apply_secure` validates transport security before applying auth 115 + - `Auth.validate_secure_transport` checks if HTTPS is required 116 + - `Error.Insecure_auth` error type for clear error messages 117 + - `allow_insecure_auth` parameter in request functions for testing 65 118 66 - **Implementation Notes:** 67 - Add runtime check in auth application: if scheme is http (not https) and auth is Basic/Bearer/Digest, raise Security_error unless allow_insecure_auth=true. Log warning when flag is used. 119 + ### 4. ✅ Validate Content-Length vs Transfer-Encoding precedence 68 120 69 - ### 4. Validate Content-Length vs Transfer-Encoding precedence 121 + **Status: IMPLEMENTED** in `lib/http_read.ml` 70 122 71 123 Per RFC 9112, when both Content-Length and Transfer-Encoding headers are present, ignore Content-Length. This prevents HTTP request smuggling attacks. 72 124 ··· 76 128 **Cross-Language Consensus:** 2 libraries 77 129 **Source Libraries:** hyper, http-streams 78 130 79 - **Affected Files:** 80 - - `lib/http_read.ml` 131 + **Implementation:** 132 + - Transfer-Encoding: chunked takes precedence over Content-Length 133 + - Warning logged when both headers present (potential attack indicator) 134 + - Per RFC 9112 Section 6.3 compliance 81 135 82 - **Implementation Notes:** 83 - Audit http_read.ml to ensure Transfer-Encoding: chunked takes precedence. Log warning when both headers present (potential attack indicator). Add test case for this scenario. 136 + ### 5. ✅ Cap maximum Retry-After delay to prevent DoS 84 137 85 - ### 5. Cap maximum Retry-After delay to prevent DoS 138 + **Status: IMPLEMENTED** in `lib/retry.ml` 86 139 87 140 Implement a configurable maximum backoff time (default 120s) that caps server-specified Retry-After values, preventing malicious servers from causing indefinite client blocking. 88 141 ··· 92 145 **Cross-Language Consensus:** 2 libraries 93 146 **Source Libraries:** req (Go), got 94 147 95 - **Affected Files:** 96 - - `lib/retry.ml` 97 - - `lib/retry.mli` 98 - 99 - **Implementation Notes:** 100 - Add backoff_max field to Retry.config. In calculate_backoff, apply: min retry_after backoff_max. Document that very long delays (>backoff_max) result in immediate failure rather than unbounded waiting. 148 + **Implementation:** 149 + - `backoff_max` field in `Retry.config` (default: 120.0 seconds) 150 + - `parse_retry_after` function accepts `?backoff_max` parameter 151 + - Delays exceeding cap are logged as warnings and capped 101 152 102 153 103 154 --- 104 155 105 156 ## Feature Enhancements 106 157 107 - ### 6. Add HTTP proxy support with environment variable detection 158 + ### 6. ❌ Add HTTP proxy support with environment variable detection 159 + 160 + **Status: NOT IMPLEMENTED** - High priority, see `spec/PROXY_IMPLEMENTATION_PLAN.md` 108 161 109 162 Implement HTTP/HTTPS proxy support including: proxy URL configuration, authentication (Basic), CONNECT method for HTTPS tunneling, and automatic detection from HTTP_PROXY, HTTPS_PROXY, NO_PROXY environment variables. 110 163 ··· 122 175 - `lib/one.ml` 123 176 124 177 **Implementation Notes:** 125 - Add proxy configuration type with host/port/auth. Implement HTTP CONNECT for HTTPS proxies. Parse environment variables on session creation. Add NO_PROXY pattern matching for bypass. Strip Proxy-Authorization on HTTPS connections. 178 + A comprehensive implementation plan exists in `spec/PROXY_IMPLEMENTATION_PLAN.md` with 8 phases covering proxy configuration, HTTP CONNECT tunneling, environment variable detection, NO_PROXY pattern matching, and proxy authentication. 179 + 180 + ### 7. ❌ Implement request/response middleware system 126 181 127 - ### 7. Implement request/response middleware system 182 + **Status: NOT IMPLEMENTED** - Medium priority 128 183 129 184 Add a composable middleware chain that allows intercepting and transforming requests before sending and responses after receiving. Enable cross-cutting concerns like logging, metrics, request signing, and custom caching. 130 185 ··· 138 193 **Implementation Notes:** 139 194 Define type middleware = Request.t -> (Request.t -> Response.t) -> Response.t. Store middleware list in session. Execute chain: apply each middleware wrapping the next. Provide default middleware for retry and redirect as replaceable components. 140 195 141 - ### 8. Add progress callbacks for uploads and downloads 196 + ### 8. ❌ Add progress callbacks for uploads and downloads 197 + 198 + **Status: NOT IMPLEMENTED** - Medium priority 142 199 143 200 Implement optional progress callbacks that report bytes transferred, total size (if known), and percent complete. Rate-limit callbacks to prevent performance degradation. 144 201 ··· 154 211 **Implementation Notes:** 155 212 Add optional progress:(transferred:int64 -> total:int64 option -> unit) parameter. Track bytes during streaming read/write. Throttle callbacks (e.g., max 3/second or every 64KB). Support both upload and download progress. 156 213 157 - ### 9. Add request cancellation support 214 + ### 9. ❌ Add request cancellation support 215 + 216 + **Status: NOT IMPLEMENTED** - Medium priority (Eio provides natural cancellation via switches) 158 217 159 218 Allow programmatic cancellation of in-flight requests. Integrate with Eio's cancellation system while providing explicit cancel() function or cancel token pattern. 160 219 ··· 169 228 **Implementation Notes:** 170 229 Return (Response.t * cancel_token) from request functions where cancel_token allows aborting the request. Internally use Eio.Cancel or Switch.fail to terminate the fiber. Raise Cancelled error on cancellation. 171 230 172 - ### 10. Add concurrent request batching API (map/imap) 231 + ### 10. ❌ Add concurrent request batching API (map/imap) 232 + 233 + **Status: NOT IMPLEMENTED** - Low priority (Eio.Fiber.all/both provides this) 173 234 174 235 Provide map() for executing multiple requests concurrently and returning results in order, and imap() for yielding responses as they complete. Include configurable concurrency limits and batch timeout. 175 236 ··· 183 244 **Implementation Notes:** 184 245 Implement using Eio.Fiber.all for map, Eio.Stream for imap. Add max_concurrent parameter using Eio.Semaphore. Provide exception_handler callback for partial failure handling. Add batch_timeout for overall operation limit. 185 246 186 - ### 11. Add base URL support with relative path resolution 247 + ### 11. ✅ Add base URL support with relative path resolution 248 + 249 + **Status: IMPLEMENTED** in `lib/requests.ml` 187 250 188 251 Allow configuring a base URL on the session that is prepended to relative request URLs. Use proper RFC 3986 URI resolution for correct path joining. 189 252 ··· 193 256 **Cross-Language Consensus:** 5 libraries 194 257 **Source Libraries:** surf, axios, got, sling, req (Go) 195 258 196 - **Affected Files:** 197 - - `lib/requests.ml` 198 - - `lib/requests.mli` 199 - 200 - **Implementation Notes:** 201 - Add base_url field to session type. In request methods, check if URL is relative (no scheme); if so, use Uri.resolve to join with base_url. Document trailing slash importance per RFC 3986. 259 + **Implementation:** 260 + - `base_url` parameter in `create` function 261 + - `is_relative_url` checks if URL has scheme 262 + - `resolve_url` uses `Uri.resolve` for RFC 3986 compliant resolution 263 + - Trailing slashes automatically normalized 202 264 203 265 204 266 --- 205 267 206 268 ## Architectural Improvements 207 269 208 - ### 12. Add detailed timing metrics to responses 270 + ### 12. ✅ Add detailed timing metrics to responses 271 + 272 + **Status: IMPLEMENTED** in new `lib/timing.ml` module 209 273 210 274 Expose granular timing breakdown: DNS lookup time, TCP connection time, TLS handshake time, time to first byte (TTFB), download time, and total time. Include connection reuse tracking. 211 275 212 276 **Cross-Language Consensus:** 7 libraries 213 277 **Source Libraries:** curl-rust, got, resty, req (Go), guzzle, okhttp, isahc 214 278 215 - **Affected Files:** 216 - - `lib/response.ml` 217 - - `lib/response.mli` 218 - - `lib/http_client.ml` 279 + **Implementation:** 280 + - New `Timing` module with detailed metrics type 281 + - Phases: `dns_lookup`, `tcp_connect`, `tls_handshake`, `request_sent`, `time_to_first_byte`, `content_transfer`, `total` 282 + - `timer` type for incremental collection during requests 283 + - `connection_time` and `server_time` computed metrics 284 + - Pretty-printing with `pp` and `to_string` 219 285 220 - **Implementation Notes:** 221 - Define type timings = { dns: float option; connect: float; tls: float option; send: float; first_byte: float; download: float; total: float; connection_reused: bool }. Instrument http_client.ml with timestamps at each phase. 286 + ### 13. ✅ Add automatic retry on stale connection detection 222 287 223 - ### 13. Add automatic retry on stale connection detection 288 + **Status: IMPLEMENTED** in `ocaml-conpool` library 224 289 225 290 Detect when a pooled connection was closed by the server between requests (NoResponseDataReceived) and automatically retry with a fresh connection. Mark this error as always retryable. 226 291 227 292 **Cross-Language Consensus:** 2 libraries 228 293 **Source Libraries:** http-client (Haskell), okhttp 229 294 230 - **Affected Files:** 231 - - `lib/http_client.ml` 232 - - `lib/error.ml` 233 - - `lib/retry.ml` 234 - - `lib/conpool.ml` 295 + **Implementation:** 296 + The `ocaml-conpool` library provides comprehensive stale connection detection via: 297 + - `is_healthy` function checking age, idle time, and use count 298 + - `max_idle_time` configuration (default: 60s) 299 + - `max_connection_lifetime` configuration (default: 300s) 300 + - `max_connection_uses` configuration (optional) 301 + - Custom `health_check` callback support 302 + - Pool `validate` function runs health checks before connection reuse 235 303 236 - **Implementation Notes:** 237 - Add No_response_data error variant. In http_client, catch empty response on reused connection. Track connection reuse status. In retry logic, always retry No_response_data even for non-idempotent methods since request wasn't sent. 304 + ### 14. ✅ Add custom retry condition predicates 238 305 239 - ### 14. Add custom retry condition predicates 306 + **Status: IMPLEMENTED** in `lib/retry.ml` 240 307 241 308 Allow users to provide custom retry decision functions beyond status codes and error types. Separate predicates for response-based retry (status/headers) and exception-based retry. 242 309 243 310 **Cross-Language Consensus:** 4 libraries 244 311 **Source Libraries:** resty, got, req (Go), req (Haskell) 245 312 246 - **Affected Files:** 247 - - `lib/retry.ml` 248 - - `lib/retry.mli` 249 - 250 - **Implementation Notes:** 251 - Add retry_condition:(Response.t option -> exn option -> retry_status -> bool) to Retry.config. Provide default that checks status_forcelist and Error.is_retryable. Allow custom logic for specific error messages, custom headers, etc. 313 + **Implementation:** 314 + - `response_predicate` type: `Method.t -> int -> Headers.t -> bool` 315 + - `exception_predicate` type: `exn -> bool` 316 + - `retry_response` and `retry_exception` fields in `Retry.config` 317 + - `should_retry_response` checks both built-in rules and custom predicates 318 + - `should_retry_exn` for exception-based retry decisions 319 + - Examples in module documentation 252 320 253 321 254 322 --- 255 323 256 324 ## Feature Enhancements 257 325 258 - ### 15. Add response charset/encoding detection and conversion 326 + ### 15. ❌ Add response charset/encoding detection and conversion 327 + 328 + **Status: NOT IMPLEMENTED** - Low priority 259 329 260 330 Automatically detect character encoding from Content-Type charset parameter and convert response text to UTF-8. Provide BOM sniffing fallback for encoding detection. 261 331 ··· 273 343 **Implementation Notes:** 274 344 Parse charset from Content-Type in Mime module. Add Response.text_utf8 that detects and converts encoding. Use uutf library for encoding conversion. Return raw bytes on conversion failure with access to detected charset. 275 345 276 - ### 16. Add redirect policy customization 346 + ### 16. ❌ Add redirect policy customization 347 + 348 + **Status: NOT IMPLEMENTED** - Low priority 277 349 278 350 Allow custom redirect policies with access to attempt status, URL, and previous URLs. Enable controlling method preservation (307/308 vs 301/302), header stripping, and cross-protocol redirect behavior. 279 351 ··· 295 367 296 368 ## Architectural Improvements 297 369 298 - ### 17. Add TCP keep-alive configuration 370 + ### 17. ❌ Add TCP keep-alive configuration 371 + 372 + **Status: NOT IMPLEMENTED** - Low priority 299 373 300 374 Expose TCP keep-alive settings including enable/disable, interval, and retry count. This helps detect broken connections and maintain long-lived connections through firewalls/NAT. 301 375 ··· 314 388 315 389 ## Feature Enhancements 316 390 317 - ### 18. Add URL sanitization for error messages 391 + ### 18. ❌ Add URL sanitization for error messages 392 + 393 + **Status: NOT IMPLEMENTED** - Low priority 318 394 319 395 Automatically sanitize URLs in error messages to prevent leakage of credentials embedded in query parameters (API keys, tokens). Provide without_url() method to strip URLs entirely. 320 396 ··· 328 404 **Implementation Notes:** 329 405 Extend existing sanitize_url function to handle query parameter credentials. Add Error.without_url to remove URL entirely from error. Apply sanitization automatically in Error.to_string and pp functions. 330 406 331 - ### 19. Add Link header parsing for pagination 407 + ### 19. ✅ Add Link header parsing for pagination 408 + 409 + **Status: IMPLEMENTED** in new `lib/link.ml` module 332 410 333 411 Parse RFC 8288 Link headers to extract relation types (next, prev, first, last) for API pagination. Provide helpers like Response.next_url for common patterns. 334 412 ··· 338 416 **Cross-Language Consensus:** 3 libraries 339 417 **Source Libraries:** got, superagent, just 340 418 341 - **Affected Files:** 342 - - `lib/headers.ml` 343 - - `lib/headers.mli` 344 - - `lib/response.ml` 345 - 346 - **Implementation Notes:** 347 - Add type link_rel = { url: string; rel: string; params: (string * string) list }. Implement parse_link_header following RFC 8288. Add Response.links and Response.next_url/prev_url convenience functions. 419 + **Implementation:** 420 + - New `Link` module with RFC 8288 compliant parsing 421 + - `Link.t` type with `uri`, `rel`, `title`, `media_type`, `hreflang`, `params` 422 + - `parse` function handles comma-separated links with quoted values 423 + - `from_headers` extracts Link header from response headers 424 + - `pagination` returns `(first, prev, next, last)` tuple 425 + - `next_url`, `prev_url`, `has_next` convenience functions 426 + - Pretty-printing with `pp` and `to_string` 348 427 349 428 350 429 --- 351 430 352 431 ## Architectural Improvements 353 432 354 - ### 20. Add mock handler for testing 433 + ### 20. ❌ Add mock handler for testing 434 + 435 + **Status: NOT IMPLEMENTED** - Medium priority 355 436 356 437 Provide a mock handler that queues pre-defined responses and exceptions for testing HTTP client code without network calls. Support request inspection and assertion. 357 438 ··· 370 451 371 452 ## Feature Enhancements 372 453 373 - ### 21. Add Result-based error_for_status variant 454 + ### 21. ✅ Add Result-based error_for_status variant 455 + 456 + **Status: IMPLEMENTED** in `lib/response.ml` 374 457 375 458 Provide check_status() that returns (Response.t, Error.error) result instead of raising, enabling functional error handling alongside the existing raise_for_status. 376 459 377 460 **Cross-Language Consensus:** 2 libraries 378 461 **Source Libraries:** reqwest, grequests 379 462 380 - **Affected Files:** 381 - - `lib/response.ml` 382 - - `lib/response.mli` 463 + **Implementation:** 464 + - `Response.check_status : t -> (t, Error.t) result` 465 + - Returns `Ok response` for 2xx status codes 466 + - Returns `Error (Http_error {...})` for 4xx/5xx status codes 467 + - Complements existing `raise_for_status` for functional style 383 468 384 - **Implementation Notes:** 385 - Add Response.check_status : t -> (t, Error.error) result that returns Ok response for 2xx, Error with Http_error for 4xx/5xx. Complement existing raise_for_status for different coding styles. 469 + ### 22. ❌ Add request body rewind support for retries 386 470 387 - ### 22. Add request body rewind support for retries 471 + **Status: NOT IMPLEMENTED** - Medium priority 388 472 389 473 Track whether request bodies are rewindable (seekable) and automatically replay them on retry. Raise clear error when retry requires body replay but body is a consumed stream. 390 474 ··· 404 488 405 489 ## Architectural Improvements 406 490 407 - ### 23. Add event listener/hooks for observability 491 + ### 23. ❌ Add event listener/hooks for observability 492 + 493 + **Status: NOT IMPLEMENTED** - Low priority 408 494 409 495 Provide comprehensive event hooks at each request lifecycle stage: DNS start/end, connection start/end, TLS handshake, request headers/body sent, response headers/body received. Enable metrics and tracing integration. 410 496 ··· 424 510 425 511 ## Feature Enhancements 426 512 427 - ### 24. Add XSRF/CSRF token handling 513 + ### 24. ✅ Add XSRF/CSRF token handling 514 + 515 + **Status: IMPLEMENTED** in `lib/requests.ml` 428 516 429 517 Automatically read CSRF token from a configurable cookie name (default: XSRF-TOKEN) and add it to request headers (default: X-XSRF-TOKEN) for same-origin requests. 430 518 431 519 **Cross-Language Consensus:** 1 libraries 432 520 **Source Libraries:** axios 433 521 434 - **Affected Files:** 435 - - `lib/requests.ml` 436 - - `lib/headers.ml` 437 - 438 - **Implementation Notes:** 439 - Add xsrf_cookie_name and xsrf_header_name config options. Before request, if cookie exists, add corresponding header. Only for same-origin requests. Disable by setting cookie name to empty. 522 + **Implementation:** 523 + - `xsrf_cookie_name` parameter (default: `Some "XSRF-TOKEN"`) 524 + - `xsrf_header_name` parameter (default: `"X-XSRF-TOKEN"`) 525 + - `apply_xsrf_token` function reads cookie and injects header 526 + - Set `xsrf_cookie_name` to `None` to disable 527 + - Applied automatically in `make_request_internal` 440 528 441 529 442 530 --- 443 531 444 532 ## Architectural Improvements 445 533 446 - ### 25. Add decompression bomb protection 534 + ### 25. ✅ Add decompression bomb protection 535 + 536 + **Status: IMPLEMENTED** in `lib/http_client.ml` 447 537 448 538 Enforce configurable compression ratio limits (e.g., 100:1) to detect and abort decompression bomb attacks. Track compressed vs decompressed bytes during streaming. 449 539 450 540 **Cross-Language Consensus:** 1 libraries 451 541 **Source Libraries:** axios 452 542 453 - **Affected Files:** 454 - - `lib/http_read.ml` 455 - - `lib/response_limits.ml` 456 - 457 - **Implementation Notes:** 458 - Add max_compression_ratio to Response_limits. During decompression, track input/output bytes. If ratio exceeds limit, raise Decompression_bomb error. Use existing error type from error.ml. 543 + **Implementation:** 544 + - `check_decompression_limits` function validates size and ratio 545 + - `max_decompressed_size` in `Response_limits` (absolute size limit) 546 + - `max_compression_ratio` in `Response_limits` (ratio limit) 547 + - `Error.Decompression_bomb` error type with limit and ratio info 548 + - Applied during gzip/deflate/zlib decompression 459 549 460 550 461 551 --- 462 552 463 553 ## Feature Enhancements 464 554 465 - ### 26. Add HTTP version to response metadata 555 + ### 26. ✅ Add HTTP version to response metadata 556 + 557 + **Status: IMPLEMENTED** in `lib/http_read.ml` 466 558 467 559 Expose the HTTP version used for the response (HTTP/1.0, HTTP/1.1, HTTP/2) in Response type. Useful for debugging protocol negotiation and monitoring HTTP/2 adoption. 468 560 469 561 **Cross-Language Consensus:** 3 libraries 470 562 **Source Libraries:** reqwest, surf, curl-rust 471 563 472 - **Affected Files:** 473 - - `lib/response.ml` 474 - - `lib/response.mli` 475 - - `lib/http_read.ml` 564 + **Implementation:** 565 + - `http_version` type: `HTTP_1_0 | HTTP_1_1` 566 + - `http_version_to_string` for display 567 + - `status_line` returns `(http_version, status_code)` tuple 568 + - `response` function returns `(http_version, status, headers, body)` 569 + - `stream_response` includes `http_version` field 476 570 477 - **Implementation Notes:** 478 - Add http_version field to Response.t. Parse version from status line. Add Response.version accessor. Prepare for future HTTP/2 support. 571 + ### 27. ❌ Add Unix domain socket support 479 572 480 - ### 27. Add Unix domain socket support 573 + **Status: NOT IMPLEMENTED** - Low priority 481 574 482 575 Enable connecting to local services via Unix domain sockets (e.g., Docker daemon at /var/run/docker.sock) without TCP overhead. 483 576 ··· 497 590 498 591 ## Security & Spec Compliance 499 592 500 - ### 28. Restrict URL protocols to HTTP/HTTPS only 593 + ### 28. ✅ Restrict URL protocols to HTTP/HTTPS only 594 + 595 + **Status: IMPLEMENTED** in `lib/requests.ml` and `lib/one.ml` 501 596 502 597 Explicitly whitelist only http:// and https:// protocols to prevent protocol smuggling attacks via file://, ftp://, gopher://, etc. Reject invalid protocols early in URL parsing. 503 598 504 599 **Cross-Language Consensus:** 1 libraries 505 600 **Source Libraries:** buzz 506 601 507 - **Affected Files:** 508 - - `lib/http_client.ml` 509 - - `lib/one.ml` 510 - - `lib/requests.ml` 511 - 512 - **Implementation Notes:** 513 - Add URL scheme validation early in request processing. Reject any scheme other than http/https with Invalid_url error. Apply validation also to redirect URLs. 602 + **Implementation:** 603 + - `allowed_redirect_schemes = ["http"; "https"]` 604 + - `validate_redirect_url` checks scheme on redirects 605 + - `Error.Invalid_redirect` raised for disallowed schemes 606 + - Applied during redirect following to prevent SSRF 514 607 515 608 516 609 --- 517 610 518 611 ## Feature Enhancements 519 612 520 - ### 29. Add path parameter templating 613 + ### 29. ✅ Add path parameter templating 614 + 615 + **Status: IMPLEMENTED** in `lib/requests.ml` 521 616 522 617 Support URL templates like /users/{id}/posts/{post_id} with placeholder substitution from a parameter map. Automatically URL-encode substituted values. 523 618 ··· 527 622 **Cross-Language Consensus:** 3 libraries 528 623 **Source Libraries:** sling, resty, req (Go) 529 624 530 - **Affected Files:** 531 - - `lib/requests.ml` 532 - - `lib/one.ml` 533 - 534 - **Implementation Notes:** 535 - Add path_params:(string * string) list parameter. Before URL resolution, scan for {key} patterns and substitute with URL-encoded values from map. Raise error if template has unmatched placeholders. 625 + **Implementation:** 626 + - `substitute_path_params` function for RFC 6570 templating 627 + - `path_params:(string * string) list` parameter on all request methods 628 + - Automatic URL encoding via `Uri.pct_encode` 629 + - Applied before base URL resolution in request pipeline 630 + - Example: `get ~path_params:[("id", "123")] "/users/{id}"` 536 631 537 632 538 633 --- 539 634 540 635 ## Architectural Improvements 541 636 542 - ### 30. Add default User-Agent header 637 + ### 30. ✅ Add default User-Agent header 638 + 639 + **Status: IMPLEMENTED** in `lib/version.ml` 543 640 544 641 Set a default User-Agent identifying the library (ocaml-requests/VERSION) unless user provides one. Helps server-side debugging and follows HTTP best practices. 545 642 ··· 549 646 **Cross-Language Consensus:** 5 libraries 550 647 **Source Libraries:** needle, resty, requests (Go), okhttp, guzzle 551 648 552 - **Affected Files:** 553 - - `lib/requests.ml` 554 - - `lib/one.ml` 555 - 556 - **Implementation Notes:** 557 - Add default User-Agent in default_headers if not set. Include library version and optionally OCaml version/platform. Allow override via explicit header or user_agent parameter. 649 + **Implementation:** 650 + - New `Version` module with library version info 651 + - `Version.user_agent` returns `"ocaml-requests/0.1.0 (OCaml X.XX.X)"` 652 + - Applied automatically if not already set in `make_request_internal` 653 + - Can be overridden via explicit User-Agent header 558 654