OCaml HTTP Client Library - Priority Feature Recommendations#

Auto-generated summary of recommendations from analyzing HTTP client libraries across multiple languages.

Implementation Status Summary#

Implemented Features#

Not Yet Implemented#

Executive Summary#

This comprehensive analysis synthesizes recommendations from 45+ HTTP client libraries across 10 programming languages (Python, JavaScript, Go, Rust, Haskell, PHP, Java, Swift, C++, Bash). The most consistent themes are: (1) Security features like certificate pinning, cross-origin header stripping, and HTTPS enforcement for sensitive auth; (2) Middleware/interceptor systems for extensibility; (3) HTTP proxy support; (4) Progress callbacks for uploads/downloads; (5) Request cancellation capabilities; (6) Detailed timing metrics; (7) Retry logic improvements including Retry-After capping and custom predicates. The prioritization emphasizes security fixes that could lead to credential leakage, followed by missing features that block enterprise adoption (proxy support), then enhancements that improve developer experience.

Security & Spec Compliance#

1. ✅ Strip sensitive headers on cross-origin redirects#

Status: IMPLEMENTED in lib/requests.ml and lib/one.ml

When redirecting to a different origin (host/port/scheme), automatically strip Authorization, Cookie, Proxy-Authorization, and WWW-Authenticate headers to prevent credential leakage to unintended domains. Also strip headers on HTTPS→HTTP protocol downgrade.

RFC References:

• RFC 9110 Section 15.4 (Redirection)

Cross-Language Consensus: 8 libraries Source Libraries: reqwest, got, node-fetch, okhttp, axios, superagent, http-client, needle

Implementation:

• same_origin function compares scheme and host
• strip_sensitive_headers removes Authorization, Cookie, Proxy-Authorization, WWW-Authenticate
• Applied during redirect handling in make_with_redirects

2. ❌ Add certificate/public key pinning support#

Status: NOT IMPLEMENTED - High priority for enterprise security

Allow pinning certificates or public keys (SPKI hashes) to protect against compromised certificate authorities. Support per-host pinning configuration with wildcard patterns.

RFC References:

• RFC 7469 (Public Key Pinning for HTTP)

Cross-Language Consensus: 4 libraries Source Libraries: okhttp, swifthttp, urllib3, alamofire

Affected Files:

• lib/requests.ml
• lib/requests.mli

Implementation Notes: Add certificate_pins parameter to create() accepting (hostname * sha256_hash list) list. Verify pins during TLS handshake by extracting SPKI from certificate chain. Provide helpful error messages showing expected vs actual hashes.

3. ✅ Enforce HTTPS for sensitive authentication methods#

Status: IMPLEMENTED in lib/auth.ml

Reject Basic, Bearer, and Digest authentication over unencrypted HTTP connections by default. Provide explicit opt-out flag (allow_insecure_auth) for testing environments.

RFC References:

• RFC 7617 Section 4 (Basic Authentication Security)
• RFC 6750 Section 5.1 (Bearer Token Security)

Cross-Language Consensus: 2 libraries Source Libraries: req (Haskell), urllib3

Implementation:

• Auth.apply_secure validates transport security before applying auth
• Auth.validate_secure_transport checks if HTTPS is required
• Error.Insecure_auth error type for clear error messages
• allow_insecure_auth parameter in request functions for testing

4. ✅ Validate Content-Length vs Transfer-Encoding precedence#

Status: IMPLEMENTED in lib/http_read.ml

Per RFC 9112, when both Content-Length and Transfer-Encoding headers are present, ignore Content-Length. This prevents HTTP request smuggling attacks.

RFC References:

• RFC 9112 Section 6.3 (Message Body Length)

Cross-Language Consensus: 2 libraries Source Libraries: hyper, http-streams

Implementation:

• Transfer-Encoding: chunked takes precedence over Content-Length
• Warning logged when both headers present (potential attack indicator)
• Per RFC 9112 Section 6.3 compliance

5. ✅ Cap maximum Retry-After delay to prevent DoS#

Status: IMPLEMENTED in lib/retry.ml

Implement a configurable maximum backoff time (default 120s) that caps server-specified Retry-After values, preventing malicious servers from causing indefinite client blocking.

RFC References:

• RFC 9110 Section 10.2.3 (Retry-After)

Cross-Language Consensus: 2 libraries Source Libraries: req (Go), got

Implementation:

• backoff_max field in Retry.config (default: 120.0 seconds)
• parse_retry_after function accepts ?backoff_max parameter
• Delays exceeding cap are logged as warnings and capped

Feature Enhancements#

6. ❌ Add HTTP proxy support with environment variable detection#

Status: NOT IMPLEMENTED - High priority, see spec/PROXY_IMPLEMENTATION_PLAN.md

Implement HTTP/HTTPS proxy support including: proxy URL configuration, authentication (Basic), CONNECT method for HTTPS tunneling, and automatic detection from HTTP_PROXY, HTTPS_PROXY, NO_PROXY environment variables.

RFC References:

• RFC 9110 Section 9.3.6 (CONNECT)
• RFC 7235 (HTTP Authentication)

Cross-Language Consensus: 7 libraries Source Libraries: reqwest, axios, guzzle, http-client, requests (Go), resty, isahc

Affected Files:

• lib/requests.ml
• lib/requests.mli
• lib/http_client.ml
• lib/one.ml

Implementation Notes: A comprehensive implementation plan exists in spec/PROXY_IMPLEMENTATION_PLAN.md with 8 phases covering proxy configuration, HTTP CONNECT tunneling, environment variable detection, NO_PROXY pattern matching, and proxy authentication.

7. ❌ Implement request/response middleware system#

Status: NOT IMPLEMENTED - Medium priority

Add a composable middleware chain that allows intercepting and transforming requests before sending and responses after receiving. Enable cross-cutting concerns like logging, metrics, request signing, and custom caching.

Cross-Language Consensus: 9 libraries Source Libraries: surf, got, axios, guzzle, okhttp, resty, buzz, moya, req (Go)

Affected Files:

• lib/requests.ml
• lib/requests.mli

Implementation Notes: Define type middleware = Request.t -> (Request.t -> Response.t) -> Response.t. Store middleware list in session. Execute chain: apply each middleware wrapping the next. Provide default middleware for retry and redirect as replaceable components.

8. ❌ Add progress callbacks for uploads and downloads#

Status: NOT IMPLEMENTED - Medium priority

Implement optional progress callbacks that report bytes transferred, total size (if known), and percent complete. Rate-limit callbacks to prevent performance degradation.

Cross-Language Consensus: 9 libraries Source Libraries: curl-rust, got, axios, guzzle, resty, req (Go), kingfisher, net (Swift), http-request (Java)

Affected Files:

• lib/requests.ml
• lib/body.ml
• lib/http_client.ml
• lib/one.ml

Implementation Notes: Add optional progress:(transferred:int64 -> total:int64 option -> unit) parameter. Track bytes during streaming read/write. Throttle callbacks (e.g., max 3/second or every 64KB). Support both upload and download progress.

9. ❌ Add request cancellation support#

Status: NOT IMPLEMENTED - Medium priority (Eio provides natural cancellation via switches)

Allow programmatic cancellation of in-flight requests. Integrate with Eio's cancellation system while providing explicit cancel() function or cancel token pattern.

Cross-Language Consensus: 6 libraries Source Libraries: got, axios, okhttp, needle, just, hyper

Affected Files:

• lib/requests.ml
• lib/requests.mli
• lib/one.ml

Implementation Notes: Return (Response.t * cancel_token) from request functions where cancel_token allows aborting the request. Internally use Eio.Cancel or Switch.fail to terminate the fiber. Raise Cancelled error on cancellation.

10. ❌ Add concurrent request batching API (map/imap)#

Status: NOT IMPLEMENTED - Low priority (Eio.Fiber.all/both provides this)

Provide map() for executing multiple requests concurrently and returning results in order, and imap() for yielding responses as they complete. Include configurable concurrency limits and batch timeout.

Cross-Language Consensus: 2 libraries Source Libraries: grequests, aiohttp

Affected Files:

• lib/requests.ml
• lib/requests.mli

Implementation Notes: Implement using Eio.Fiber.all for map, Eio.Stream for imap. Add max_concurrent parameter using Eio.Semaphore. Provide exception_handler callback for partial failure handling. Add batch_timeout for overall operation limit.

11. ✅ Add base URL support with relative path resolution#

Status: IMPLEMENTED in lib/requests.ml

Allow configuring a base URL on the session that is prepended to relative request URLs. Use proper RFC 3986 URI resolution for correct path joining.

RFC References:

• RFC 3986 Section 5 (Reference Resolution)

Cross-Language Consensus: 5 libraries Source Libraries: surf, axios, got, sling, req (Go)

Implementation:

• base_url parameter in create function
• is_relative_url checks if URL has scheme
• resolve_url uses Uri.resolve for RFC 3986 compliant resolution
• Trailing slashes automatically normalized

Architectural Improvements#

12. ✅ Add detailed timing metrics to responses#

Status: IMPLEMENTED in new lib/timing.ml module

Expose granular timing breakdown: DNS lookup time, TCP connection time, TLS handshake time, time to first byte (TTFB), download time, and total time. Include connection reuse tracking.

Cross-Language Consensus: 7 libraries Source Libraries: curl-rust, got, resty, req (Go), guzzle, okhttp, isahc

Implementation:

• New Timing module with detailed metrics type
• Phases: dns_lookup, tcp_connect, tls_handshake, request_sent, time_to_first_byte, content_transfer, total
• timer type for incremental collection during requests
• connection_time and server_time computed metrics
• Pretty-printing with pp and to_string

13. ✅ Add automatic retry on stale connection detection#

Status: IMPLEMENTED in ocaml-conpool library

Detect when a pooled connection was closed by the server between requests (NoResponseDataReceived) and automatically retry with a fresh connection. Mark this error as always retryable.

Cross-Language Consensus: 2 libraries Source Libraries: http-client (Haskell), okhttp

Implementation: The ocaml-conpool library provides comprehensive stale connection detection via:

• is_healthy function checking age, idle time, and use count
• max_idle_time configuration (default: 60s)
• max_connection_lifetime configuration (default: 300s)
• max_connection_uses configuration (optional)
• Custom health_check callback support
• Pool validate function runs health checks before connection reuse

14. ✅ Add custom retry condition predicates#

Status: IMPLEMENTED in lib/retry.ml

Allow users to provide custom retry decision functions beyond status codes and error types. Separate predicates for response-based retry (status/headers) and exception-based retry.

Cross-Language Consensus: 4 libraries Source Libraries: resty, got, req (Go), req (Haskell)

Implementation:

• response_predicate type: Method.t -> int -> Headers.t -> bool
• exception_predicate type: exn -> bool
• retry_response and retry_exception fields in Retry.config
• should_retry_response checks both built-in rules and custom predicates
• should_retry_exn for exception-based retry decisions
• Examples in module documentation

Feature Enhancements#

15. ❌ Add response charset/encoding detection and conversion#

Status: NOT IMPLEMENTED - Low priority

Automatically detect character encoding from Content-Type charset parameter and convert response text to UTF-8. Provide BOM sniffing fallback for encoding detection.

RFC References:

• RFC 9110 Section 8.3.1 (Content-Type)

Cross-Language Consensus: 6 libraries Source Libraries: reqwest, surf, isahc, needle, got, http-request (Java)

Affected Files:

• lib/response.ml
• lib/response.mli
• lib/http_read.ml

Implementation Notes: Parse charset from Content-Type in Mime module. Add Response.text_utf8 that detects and converts encoding. Use uutf library for encoding conversion. Return raw bytes on conversion failure with access to detected charset.

16. ❌ Add redirect policy customization#

Status: NOT IMPLEMENTED - Low priority

Allow custom redirect policies with access to attempt status, URL, and previous URLs. Enable controlling method preservation (307/308 vs 301/302), header stripping, and cross-protocol redirect behavior.

RFC References:

• RFC 9110 Section 15.4 (Redirection 3xx)

Cross-Language Consensus: 5 libraries Source Libraries: reqwest, needle, node-fetch, guzzle, http-client (Haskell)

Affected Files:

• lib/requests.ml
• lib/requests.mli

Implementation Notes: Add redirect_policy type with hooks: before_redirect:(request -> response -> uri -> bool), strip_headers:(origin_changed -> header_name -> bool). Implement RFC-correct 301/302 POST→GET conversion. Provide presets: follow_all, same_host_only, same_protocol_only.

Architectural Improvements#

17. ❌ Add TCP keep-alive configuration#

Status: NOT IMPLEMENTED - Low priority

Expose TCP keep-alive settings including enable/disable, interval, and retry count. This helps detect broken connections and maintain long-lived connections through firewalls/NAT.

Cross-Language Consensus: 3 libraries Source Libraries: reqwest, isahc, curl-rust

Affected Files:

• lib/requests.mli
• lib/http_client.ml

Implementation Notes: Add tcp_keepalive_interval and tcp_keepalive option to session config. Configure socket options via Eio.Net during connection establishment. Default to reasonable values (e.g., 60s interval).

Feature Enhancements#

18. ❌ Add URL sanitization for error messages#

Status: NOT IMPLEMENTED - Low priority

Automatically sanitize URLs in error messages to prevent leakage of credentials embedded in query parameters (API keys, tokens). Provide without_url() method to strip URLs entirely.

Cross-Language Consensus: 1 libraries Source Libraries: reqwest

Affected Files:

• lib/error.ml
• lib/error.mli

Implementation Notes: Extend existing sanitize_url function to handle query parameter credentials. Add Error.without_url to remove URL entirely from error. Apply sanitization automatically in Error.to_string and pp functions.

19. ✅ Add Link header parsing for pagination#

Status: IMPLEMENTED in new lib/link.ml module

Parse RFC 8288 Link headers to extract relation types (next, prev, first, last) for API pagination. Provide helpers like Response.next_url for common patterns.

RFC References:

• RFC 8288 (Web Linking)

Cross-Language Consensus: 3 libraries Source Libraries: got, superagent, just

Implementation:

• New Link module with RFC 8288 compliant parsing
• Link.t type with uri, rel, title, media_type, hreflang, params
• parse function handles comma-separated links with quoted values
• from_headers extracts Link header from response headers
• pagination returns (first, prev, next, last) tuple
• next_url, prev_url, has_next convenience functions
• Pretty-printing with pp and to_string

Architectural Improvements#

20. ❌ Add mock handler for testing#

Status: NOT IMPLEMENTED - Medium priority

Provide a mock handler that queues pre-defined responses and exceptions for testing HTTP client code without network calls. Support request inspection and assertion.

Cross-Language Consensus: 3 libraries Source Libraries: guzzle, okhttp, resty

Affected Files:

• lib/http_client.ml
• test/

Implementation Notes: Create Mock module with queue of responses/exceptions. Replace actual HTTP client in test mode. Track all requests made for assertions. Allow configuring delays for timeout testing.

Feature Enhancements#

21. ✅ Add Result-based error_for_status variant#

Status: IMPLEMENTED in lib/response.ml

Provide check_status() that returns (Response.t, Error.error) result instead of raising, enabling functional error handling alongside the existing raise_for_status.

Cross-Language Consensus: 2 libraries Source Libraries: reqwest, grequests

Implementation:

• Response.check_status : t -> (t, Error.t) result
• Returns Ok response for 2xx status codes
• Returns Error (Http_error {...}) for 4xx/5xx status codes
• Complements existing raise_for_status for functional style

22. ❌ Add request body rewind support for retries#

Status: NOT IMPLEMENTED - Medium priority

Track whether request bodies are rewindable (seekable) and automatically replay them on retry. Raise clear error when retry requires body replay but body is a consumed stream.

Cross-Language Consensus: 4 libraries Source Libraries: isahc, curl-rust, surf, node-fetch

Affected Files:

• lib/body.ml
• lib/retry.ml
• lib/error.ml

Implementation Notes: Add is_rewindable field to Body.t. For Body.String/Form, rewindable=true. For Stream without seek, rewindable=false. In retry, check rewindability; raise Body_not_rewindable if retry needed but body consumed.

Architectural Improvements#

23. ❌ Add event listener/hooks for observability#

Status: NOT IMPLEMENTED - Low priority

Provide comprehensive event hooks at each request lifecycle stage: DNS start/end, connection start/end, TLS handshake, request headers/body sent, response headers/body received. Enable metrics and tracing integration.

Cross-Language Consensus: 3 libraries Source Libraries: okhttp, resty, http-streams

Affected Files:

• lib/requests.ml
• lib/requests.mli
• lib/http_client.ml

Implementation Notes: Define event types: Dns_start, Dns_end, Connect_start, etc. Add event_listener callback to session config. Emit events at appropriate points in http_client.ml. Include timing and context in each event.

Feature Enhancements#

24. ✅ Add XSRF/CSRF token handling#

Status: IMPLEMENTED in lib/requests.ml

Automatically read CSRF token from a configurable cookie name (default: XSRF-TOKEN) and add it to request headers (default: X-XSRF-TOKEN) for same-origin requests.

Cross-Language Consensus: 1 libraries Source Libraries: axios

Implementation:

• xsrf_cookie_name parameter (default: Some "XSRF-TOKEN")
• xsrf_header_name parameter (default: "X-XSRF-TOKEN")
• apply_xsrf_token function reads cookie and injects header
• Set xsrf_cookie_name to None to disable
• Applied automatically in make_request_internal

Architectural Improvements#

25. ✅ Add decompression bomb protection#

Status: IMPLEMENTED in lib/http_client.ml

Enforce configurable compression ratio limits (e.g., 100:1) to detect and abort decompression bomb attacks. Track compressed vs decompressed bytes during streaming.

Cross-Language Consensus: 1 libraries Source Libraries: axios

Implementation:

• check_decompression_limits function validates size and ratio
• max_decompressed_size in Response_limits (absolute size limit)
• max_compression_ratio in Response_limits (ratio limit)
• Error.Decompression_bomb error type with limit and ratio info
• Applied during gzip/deflate/zlib decompression

Feature Enhancements#

26. ✅ Add HTTP version to response metadata#

Status: IMPLEMENTED in lib/http_read.ml

Expose the HTTP version used for the response (HTTP/1.0, HTTP/1.1, HTTP/2) in Response type. Useful for debugging protocol negotiation and monitoring HTTP/2 adoption.

Cross-Language Consensus: 3 libraries Source Libraries: reqwest, surf, curl-rust

Implementation:

• http_version type: HTTP_1_0 | HTTP_1_1
• http_version_to_string for display
• status_line returns (http_version, status_code) tuple
• response function returns (http_version, status, headers, body)
• stream_response includes http_version field

27. ❌ Add Unix domain socket support#

Status: NOT IMPLEMENTED - Low priority

Enable connecting to local services via Unix domain sockets (e.g., Docker daemon at /var/run/docker.sock) without TCP overhead.

Cross-Language Consensus: 2 libraries Source Libraries: curl-rust, http-streams

Affected Files:

• lib/http_client.ml
• lib/requests.ml
• lib/one.ml

Implementation Notes: Add unix:// URL scheme support. Use Eio_unix for socket connection. Handle Host header correctly (use socket path or override). Add to connection pool keyed by socket path.

Security & Spec Compliance#

28. ✅ Restrict URL protocols to HTTP/HTTPS only#

Status: IMPLEMENTED in lib/requests.ml and lib/one.ml

Explicitly whitelist only http:// and https:// protocols to prevent protocol smuggling attacks via file://, ftp://, gopher://, etc. Reject invalid protocols early in URL parsing.

Cross-Language Consensus: 1 libraries Source Libraries: buzz

Implementation:

• allowed_redirect_schemes = ["http"; "https"]
• validate_redirect_url checks scheme on redirects
• Error.Invalid_redirect raised for disallowed schemes
• Applied during redirect following to prevent SSRF

Feature Enhancements#

29. ✅ Add path parameter templating#

Status: IMPLEMENTED in lib/requests.ml

Support URL templates like /users/{id}/posts/{post_id} with placeholder substitution from a parameter map. Automatically URL-encode substituted values.

RFC References:

• RFC 6570 (URI Template)

Cross-Language Consensus: 3 libraries Source Libraries: sling, resty, req (Go)

Implementation:

• substitute_path_params function for RFC 6570 templating
• path_params:(string * string) list parameter on all request methods
• Automatic URL encoding via Uri.pct_encode
• Applied before base URL resolution in request pipeline
• Example: get ~path_params:[("id", "123")] "/users/{id}"

Architectural Improvements#

30. ✅ Add default User-Agent header#

Status: IMPLEMENTED in lib/version.ml

Set a default User-Agent identifying the library (ocaml-requests/VERSION) unless user provides one. Helps server-side debugging and follows HTTP best practices.

RFC References:

• RFC 9110 Section 10.1.5 (User-Agent)

Cross-Language Consensus: 5 libraries Source Libraries: needle, resty, requests (Go), okhttp, guzzle

Implementation:

• New Version module with library version info
• Version.user_agent returns "ocaml-requests/0.1.0 (OCaml X.XX.X)"
• Applied automatically if not already set in make_request_internal
• Can be overridden via explicit User-Agent header