std.http.proxy reverse proxy

nginx-class outbound HTTP forwarding for the Aether server. Forward inbound requests to a pool of upstream HTTP servers, with:

• five load-balancing algorithms (round-robin, least-conn, ip-hash, smooth-weighted RR, cookie-hash sticky)
• active health checks (one thread per pool, lock-free hot-path read)
• per-upstream circuit breaker (closed / open / half-open)
• in-memory LRU response cache with Vary-aware keying and TTL eviction
• idempotent retry on 5xx + transport with exponential backoff + full jitter, re-picks a different upstream per attempt (nginx proxy_next_upstream semantics)
• per-upstream token-bucket rate limiting
• active drain (take a host out of rotation without removing it from the pool)
• W3C Trace-Context propagation (passthrough or generated traceparent)
• Hop-by-Hop header handling per RFC 7230 §6.1
• Prometheus 0.0.4 metrics surface (per-upstream + per-pool)

The proxy is a middleware plugged into the existing http_server_use_middleware chain. It short-circuits the chain (returns 0) and owns the response.

Quick start, single upstream

import std.http
import std.http.proxy

main() {
    server = http.server_create(8080)
    err = proxy.mount_simple(server, "/", "http://localhost:9000", 30)
    if err != "" { println("proxy: ${err}"); return }
    http.server_start(server)
}

mount_simple(server, path_prefix, upstream_url, timeout_sec) builds a single-upstream pool with round-robin (no other upstream to balance against), default opts, and the given timeout. Mounts the middleware on path_prefix. "/" forwards everything; "/api" forwards just the /api subtree. The server owns that pool and those opts, and http.server_free frees them (#3171). proxy.mount, mount_methods and mount_match take opts the caller made: the server uses them while it runs, and the caller frees them (opts_free, then upstream_pool_free) after http.server_free.

Pool form, load balancing + health + cache + breaker

import std.http
import std.http.proxy

main() {
    server = http.server_create(8080)

    // Three upstreams, smooth weighted RR with 3:2:1 ratio.
    pool = proxy.upstream_pool_new("weighted_rr", 30, 0, 100)
    proxy.upstream_add(pool, "http://10.0.0.1:8080", 3)
    proxy.upstream_add(pool, "http://10.0.0.2:8080", 2)
    proxy.upstream_add(pool, "http://10.0.0.3:8080", 1)

    // Active health checks every 5s. Two consecutive 200s flips an
    // upstream up; three consecutive failures flips it down.
    proxy.health_checks_enable(pool, "/health", 200, 5000, 1000, 2, 3)

    // Per-upstream circuit breaker: 5 consecutive failures opens the
    // breaker for 30s; one half-open test request closes it on success.
    proxy.breaker_configure(pool, 5, 30000, 1)

    // Response cache: 1000 entries, 64 KiB body cap, 60s default TTL,
    // canonical key includes Vary headers from upstream.
    cache = proxy.cache_new(1000, 65536, 60, "method_url_vary")

    // Per-mount options.
    opts = proxy.opts_new()
    proxy.opts_bind_cache(opts, cache)
    proxy.opts_set_strip_prefix(opts, "/api")  // /api/users → /users upstream

    err = proxy.mount(server, "/api", pool, opts)
    if err != "" { println("proxy: ${err}"); return }

    http.server_start(server)
}

Method-Aware Routing

Multiple proxy mounts can share one listener. Registration order is execution order; a mount whose method or path pattern does not match passes through to the next middleware, so narrower carve-outs should be registered first.

Middleware registered before a proxy (a rate limiter, basic auth, use_real_ip) runs on every request the proxy forwards, and the request hooks (the access log, metrics) see every response, a proxied one and one a middleware refused included. Response transformers (use_gzip, use_error_pages) are the route handler's: a proxied response passes on as the upstream sent it, as with nginx's gzip_proxied off. On Linux and macOS the event driver runs all of that itself when each piece is registered as never blocking; see Middleware and hooks on the driver.

proxy.mount_match(server, "/repos/:repo/info", primary_pool, primary_opts, "GET,HEAD")
proxy.mount_methods(server, "/repos", primary_pool, write_opts, "POST,PUT,DELETE")
proxy.mount_methods(server, "/repos", replica_pool, read_opts, "GET,HEAD")

mount_methods keeps the normal path-prefix match. mount_match uses std.http route-pattern syntax (:param, *) against the whole request path, which covers cases like GET /repos/:repo/info going to a primary while other reads fan out to replicas.

Load-balancing algorithms

"Eligible" means: healthy && !draining && breaker not OPEN && inflight < max_inflight_per_up && rate_limit admits.

// cookie-hash sticky: bind requests carrying SESSIONID=foo to the
// same upstream every time. Different cookie values may hash to
// different upstreams; identical values are deterministic.
pool = proxy.upstream_pool_new("cookie_hash", 30, 0, 0)
proxy.upstream_add(pool, "http://10.0.0.1:8080", 1)
proxy.upstream_add(pool, "http://10.0.0.2:8080", 1)
proxy.pool_set_cookie_name(pool, "SESSIONID")

Health checks

One pthread per pool (not per upstream, N-thread blow-up). The thread iterates the upstream list every interval_ms, fires GET probe_path against each base_url, classifies the response, and flips _Atomic int healthy once the threshold trips:

• healthy_threshold consecutive OK probes flip an upstream up.
• unhealthy_threshold consecutive failed probes flip it down.

expect_status = 0 accepts any 2xx; non-zero is the exact status. Down upstreams are skipped by the LB picker (every algorithm).

Configure once via proxy.health_checks_enable(pool, ...); calling again on the same pool stops the prior thread and starts a new one with the new config. proxy.upstream_pool_free joins the thread.

Circuit breaker

Per-upstream three-state machine:

   CLOSED ── consecutive_failures ≥ failure_threshold ──→ OPEN
                                                          │
   OPEN   ── now - opened_at ≥ open_duration_ms ─────────→ HALF_OPEN
                                                          │
   HALF_OPEN ── any test request succeeds ───────────────→ CLOSED
   HALF_OPEN ── any test request fails ──────────────────→ OPEN (reset opened_at)

The hot path is one atomic load per request (cb_state). Transitions are CAS-based for the OPEN→HALF_OPEN race so two threads simultaneously timing out the open window don't both admit "the test request"; half_open_max caps concurrent test requests.

5xx responses and transport errors (DNS, connect, TLS, timeout) count as failures. 4xx is treated as ok (client error, not upstream fault).

failure_threshold = 0 disables the breaker (default, opt in via proxy.breaker_configure).

Response cache

In-memory LRU with TTL eviction. Open-chained hash + doubly-linked LRU list. Cacheability gates per RFC 7234 conservative subset:

• Methods: GET, HEAD only.
• Status codes: 200, 203, 204, 300, 301, 404, 410.
• Request Cache-Control: no-store → bypass.
• Response Cache-Control: no-store / private → bypass.
• Response Vary: * → uncacheable.
• Body length ≤ max_body_bytes.

TTL resolution: response Cache-Control: max-age (clamped to 1 hour for v1 conservatism), else default_ttl_sec. s-maxage and Expires are not consulted in v1.

Key strategies

Idempotent retry, proxy_next_upstream semantics

Off by default. Opt in with:

proxy.opts_set_retry_policy(opts, 3, 100)   // max_retries=3, backoff_base_ms=100

When a 5xx or transport error comes back from an upstream and the request is idempotent (GET / HEAD / PUT / DELETE / OPTIONS), the middleware:

1. Charges the failure against the failing upstream's circuit breaker (so a flapping host trips the breaker normally).
2. Releases the inflight count on that upstream.
3. Calls the load-balancer picker again, usually selecting a different upstream.
4. Backs off for backoff_base_ms × 2^(attempt-1) with full jitter (uniform random in [0, current_backoff]), capped at 10 s. On the event loop the backoff is a timer: the driver serves its other connections meanwhile (#3232).
5. Retries.

POST and PATCH are never retried, at-most-once delivery is preserved so the proxy never silently double-applies a non-idempotent request. The total retry budget is 1 + max_retries attempts; if no eligible upstream remains, the proxy returns 503 with X-Aether-Proxy-Error: no_upstream_after_retry and Retry-After: 1.

Per-upstream rate limit (token bucket)

Off by default. Configure once on the pool:

proxy.rate_limit_set(pool, 200, 50)   // max_rps=200, burst=50

Each upstream gets an independent token bucket: tokens refill at max_rps per second, capped at burst. The LB picker decrements one token before admitting a request; when the bucket is empty the upstream is treated as ineligible (RR / weighted-RR / etc. fall through to the next eligible upstream, or 503 if none).

Set max_rps = 0 to disable. The burst capacity matters at start- up: with burst = 0 the first request is rejected, so for typical deployments burst should be at least equal to expected concurrent connection bursts (a small multiple of max_rps is conservative).

Active drain

Take a host out of rotation without removing it from the pool:

proxy.upstream_drain(pool,   "http://10.0.0.1:8080")   // stop sending new requests
proxy.upstream_undrain(pool, "http://10.0.0.1:8080")   // re-admit

Drained upstreams are skipped by the LB picker (every algorithm). In-flight requests finish naturally; when they complete the upstream is fully idle and safe to take down for a deploy. Drain state is operator-driven and orthogonal to healthy (which the health-check thread owns).

W3C Trace-Context propagation

Two modes per mount:

proxy.opts_set_trace_inject(opts, 0)   // default, passthrough
proxy.opts_set_trace_inject(opts, 1)   // generate when missing

Passthrough (inject=0): if the inbound request carries traceparent / tracestate, the proxy forwards them unchanged to the upstream. End-to-end traces propagate naturally. If the inbound has no traceparent, none is added.

Inject (inject=1): same passthrough behaviour for incoming trace headers, plus, when the inbound request has no traceparent the proxy generates a fresh W3C-compliant one (00-<32-hex>-<16-hex>-01) and stamps it on the outbound request. Use this on the edge of your service mesh so every upstream gets a trace context even from trace-naïve clients.

Prometheus metrics

proxy.pool_metrics_text(pool) -> string    // 0.0.4 exposition

Wire it to a route to expose Grafana-scrapable metrics:

handle_metrics(req: ptr, res: ptr, ud: ptr) {
    body = proxy.pool_metrics_text(ud)
    http.response_set_status(res, 200)
    http.response_set_header(res, "Content-Type", "text/plain; version=0.0.4")
    http.response_set_body(res, body)
}

http.server_get(server, "/proxy-metrics", handle_metrics, pool)
proxy.mount(server, "/api", pool, opts)  // mount AFTER routes

(Mount the proxy on a more specific prefix than "/" so it doesn't shadow the metrics route.)

Surface (HELP + TYPE blocks elided in this table):

Latency is exposed as _sum + _count so a Grafana panel can compute average latency as rate(_sum) / rate(_count).

How a proxied request is served

A proxying server runs an event driver: a small number of threads, one per core, each holding a poller and many connections at once. A connection stays with the driver that accepted it for its whole life, and so does the upstream connection it uses.

Requests move through it as a state machine:

read request -> dial or reuse upstream -> write upstream
             -> read upstream -> write client -> read request

Every step either makes progress or names the descriptor it needs ready before it can continue. A thread only waits when nothing it holds can make progress, so an upstream that is slow to answer occupies a descriptor and some state rather than a thread. Each response it writes keeps or closes the connection as the server's keep-alive policy says (http.server_set_keepalive, the client's Connection header, HTTP/1.0), and says so in its Connection header, as on the per-connection path (#3229). A client that sends nothing for the keep-alive idle timeout (30 seconds unless set), before its first request, between requests or partway through one, has its connection closed, as the per-connection path's receive timeout closes it (#3243). So does one that takes no bytes of its response for as long, as the per-connection path's send timeout does (#3257).

A driver never waits on a name lookup, which takes seconds when a name does not exist or the resolver is slow. An upstream given by name is looked up by the loop's own resolver, on a thread of its own, the first time a request dials it: the connection waits for the answer while the driver serves the rest, and stays on the driver. Requests waiting on one name share one lookup. The answer is kept and used for every dial after it; one older than ten seconds is looked up again behind the dial that uses it, so a changed record is seen about that soon. While lookups of the name fail, the last answer stays in use, for up to a day, as RFC 8767 serves stale answers (#3286); when the resolver says the name does not exist, it goes. A name that does not resolve is answered 502, as on the worker path, and is asked about again a second later. The request's upstream timeout (request_timeout_sec) covers the wait and the call after it together (#3283). Up to four lookups run at once, each on a thread of its own, so a slow name holds up only its own requests until four are slow together (#3284). Adding an upstream looks nothing up, and a program that does not proxy never starts the resolver (#3267, #3275, #3276, #3277). Stopping the server waits for a lookup under way to end, as it did when the driver looked names up itself.

The driver serves HTTP/1.1 proxying, on a TLS-terminating server too: it carries the client's TLS handshake without waiting and reads and writes through the session. These keep the per-connection worker path, and need no configuration to do so:

• HTTP/2
• connection upgrades and tunnels
• streaming request bodies
• a chunked request body, which the worker path decodes (#3228), and any request the worker path refuses (#3233, #3236)
• a request body larger than the mount's body cap, which the worker path streams and the proxy refuses unread, closing the connection so the body is never read as the next request (#3255), and a head that has not ended in 16 KiB, which it refuses with 431 (#3252): the driver never buffers more (#3237)
• a request body over 16 KiB once the driver already holds as many bodies of the cap as the worker path has workers (AETHER_HTTP_WORKERS, else twice the cores, eight at least): its total is the worker path's, not the cap times every connection a client opens (#3263). Every connection keeps at most 64 KiB of buffers between requests, on either path (#3262).
• a server that terminates TLS with the pure-Aether stack (AETHER_PURE_TLS=1, or a build without OpenSSL): the driver's TLS is OpenSSL's (#3256)
• a server whose first proxy mount has an https upstream: the driver dials plain HTTP only, and the worker path's client speaks TLS. An https upstream added while the server runs moves it there from the next request, and a retry that reaches one hands its request over (#3231)

A request the proxy does not own, a health endpoint, an admin route, or anything another middleware answers, is handed to the general server path along with whatever has already been read from the socket, and a pool worker serves the connection from then on, as it does an accepted one: a slow route handler never holds up the driver's other connections (#3230). The pool adds workers while all of them are busy, up to 64, and queues up to 256 connections; past that a driver handing a connection over waits for a place in the queue, as the accept thread does, and its other connections wait with it. That is the server's backpressure, the same one an accepted connection meets, rather than a queue without end. Routing, middleware and handlers behave exactly as they do without the driver: a request the middleware before the first mount already ran on goes on from the middleware after the mount, so none of it runs twice and a rate limiter counts it once.

Middleware and hooks on the driver

The driver runs everything a proxied request gets on the per-connection path (#3214): the middleware registered before the first proxy mount, in order, one of which may answer instead of the proxy (a 429 from the rate limiter, a CORS preflight, an unknown host), then the proxy exchange, then the request hooks on each response it sends. A middleware's answer leaves the connection kept alive or closed as the per-connection path would.

Whatever the driver runs holds up every connection on its thread while it runs, so it runs only what was registered as never blocking: loop-safe. std's own use_real_ip, use_rate_limit, use_cors, use_vhost and use_rewrite are, and so are the access log (a line appended to a file, as nginx writes its access log from its event loop) and the metrics. use_basic_auth, use_bearer_auth and use_session_auth call your verifier, which may wait on a database, and use_static_files reads files: they are not. A middleware your program registers with http.server_use_middleware is not either, because it may block. One that does not (CPU work, a short mutex) says so by registering with http.server_use_middleware_loop_safe:

import std.http
import std.http.proxy

// Refuses a request that names no tenant. CPU work only: loop-safe.
@c_callback tenant_header(req: ptr, res: ptr, ud: ptr) -> int {
    if http.get_header(req, "X-Tenant") == null {
        http.response_set_status(res, 400)
        http.response_set_body(res, "X-Tenant is required")
        return 0
    }
    return 1
}

main() {
    server = http.server_create(8080)
    http.server_set_host(server, "127.0.0.1")
    // The last argument would free the user data with the server.
    http.server_use_middleware_loop_safe(server, tenant_header, null, null)
    err = proxy.mount_simple(server, "/", "http://127.0.0.1:9000", 30)
    if err != "" {
        println("proxy: ${err}")
        return
    }
    http.server_start(server)
}

A server with a middleware before the first mount, or a request hook, that is not loop-safe keeps the per-connection path for every connection, where everything runs as it always has. The responses are the same on either path; what changes is how many threads it takes to serve them. A loop-safe middleware that blocks after all stalls every connection on its thread. When a gateway uses the request hooks, or a middleware before the mount, the driver answers from a response object (the hooks observe it, and a middleware may have put headers on it), where a bare proxy answers straight from the upstream's bytes. Windows has no event loop, and always uses the per-connection path.

The upstream timeout (proxy.pool_set_request_timeout) applies as it always has. The driver keeps the deadline itself rather than taking it from the socket, and an upstream that does not answer within it produces the same 504 with X-Aether-Proxy-Error: upstream_timeout. The timeout counts against the upstream the same way too: its in-flight count goes down, its breaker records the failure, and a retry policy retries it (#3250).

Hop-by-Hop headers (RFC 7230 §6.1)

Stripped on both directions:

Connection, Keep-Alive, Proxy-Authenticate, Proxy-Authorization,
TE, Trailer, Transfer-Encoding, Upgrade, Proxy-Connection

Plus any header listed in the request's Connection: value.

Headers added by the proxy

The entry the proxy appends is the peer the request reports (http.request_remote_addr): the connection's peer address, or, when middleware.use_real_ip ran before the proxy, the client it resolved from the trusted chain. A request that reaches the proxy over a Unix-domain socket has no IP peer and adds unknown. The proxy never reads X-Real-IP: any client can send that header, and taking it let a client write its own entry into the chain the upstream trusts (#3169). Mount middleware.use_real_ip before the proxy when clients reach it through a trusted edge, so the entry names the client rather than the edge.

Error responses

The middleware emits one of these and short-circuits the chain. Each carries an X-Aether-Proxy-Error header for log aggregators:

Limitations (v1)

• Request and response bodies are buffered in memory up to opts.max_body_bytes (default 8 MiB). Streaming pass-through is the next major feature.
• WebSocket and SSE upstreams are unsupported: requests with Upgrade: headers are refused with 502. The upstream can still serve Server-Sent Events to clients via the existing std.http surface, so long as the proxy is not in front of those routes.
• Upstreams are called over HTTP/1.1 via std.http.client. Inbound HTTP/2 works (the inbound side is independent); upstream HTTP/2 is a v2 follow-up.
• In-memory cache only. No disk-backed cache, no shared cache across processes.
• Sharded cache lock is a v2 follow-up. Single mutex around the whole cache works under typical workloads; revisit if contention shows in profiling.

Performance

Reference

import std.http.proxy

// ---- Pool + upstream lifecycle
proxy.upstream_pool_new(lb_algo, request_timeout_sec, dial_timeout_ms, max_inflight_per_up) -> ptr
proxy.upstream_pool_free(pool)
proxy.upstream_add(pool, base_url, weight) -> string
proxy.upstream_remove(pool, base_url) -> string
proxy.upstream_drain(pool, base_url) -> string       // skip in LB
proxy.upstream_undrain(pool, base_url) -> string     // re-admit

// ---- Per-pool tuning
proxy.pool_set_cookie_name(pool, name) -> string     // for "cookie_hash"
proxy.rate_limit_set(pool, max_rps, burst) -> string // 0 = disable

// ---- Health checks
proxy.health_checks_enable(pool, probe_path, expect_status,
    interval_ms, timeout_ms, healthy_threshold, unhealthy_threshold) -> string

// ---- Circuit breaker
proxy.breaker_configure(pool, failure_threshold, open_duration_ms, half_open_max) -> string

// ---- Cache
proxy.cache_new(max_entries, max_body_bytes, default_ttl_sec, key_strategy) -> ptr
proxy.cache_free(cache)

// ---- Per-mount opts
proxy.opts_new() -> ptr
proxy.opts_free(opts)
proxy.opts_set_strip_prefix(opts, prefix) -> string
proxy.opts_set_preserve_host(opts, on) -> string
proxy.opts_set_xforwarded(opts, xff, xfp, xfh) -> string
proxy.opts_bind_cache(opts, cache) -> string
proxy.opts_set_body_cap(opts, max_body_bytes) -> string
proxy.opts_set_retry_policy(opts, max_retries, backoff_base_ms) -> string
proxy.opts_set_trace_inject(opts, on) -> string

// ---- Install
proxy.mount(server, path_prefix, pool, opts) -> string
proxy.mount_simple(server, path_prefix, upstream_url, request_timeout_sec) -> string

// ---- Observability
proxy.pool_metrics_text(pool) -> string              // Prometheus 0.0.4

All string-returning functions are Go-style: "" is success; non-empty is an error message.

See also

• docs/http-server.md, the inbound side (routes, middleware, TLS, h2, WS, SSE, health probes, metrics, graceful shutdown).
• examples/03-std/07-reverse-proxy.ae one upstream behind proxy.mount_simple, upstream and client in the same program.
• examples/03-std/08-proxy-pool.ae a weighted pool of three upstreams configured with the proxy.pool block, mounted with proxy.mount.