Repository navigation
Conversation
A host with "h3" in Protocols now serves HTTP/3 with the certificate mod_ssl already resolved for it (SSLCertificateFile and mod_md alike), the way mod_http2 rides mod_ssl. The certificate and key are loaded in the ap_ssl_add_cert_files hook, during startup while httpd still runs privileged, so a key readable only by root loads for QUIC as it does for mod_ssl instead of failing in the unprivileged child. H3CertificatePath and H3CertificateKeyPath are removed; drop them from existing configurations. Each virtual host on a shared port now presents its own certificate over HTTP/3, chosen by the client SNI; an unmatched name gets the listener default. Names come from ServerName, ServerAlias and the wildcard ServerAlias names that httpd keeps in wild_names, and every entry is matched with ap_strcasecmp_match, the matcher httpd uses for its own virtual hosts. A cert_cb on the listener context applies the matched host certificate, key and chain to the connection: SSL_set_SSL_CTX does not switch the certificate of a QUIC connection. Landing it exposed that the module ran with AP_MODULE_FLAG_NONE, so a host without H3 directives shared the main server configuration; per-host H3AltSvc and H3AltSvcMaxAge were silently ignored. Set ALWAYS_MERGE.
httpd prepends virtual hosts to the server list, so the first host with an HTTP/3 context in that list is the last one defined. The listener took that host's settings and certificate, and a client whose SNI matched no host got that certificate, where mod_ssl serves the first virtual host defined for the address. Take the last match in both h3_post_config and find_h3_server instead.
Each child bound the UDP port itself after httpd dropped root, so a port below 1024 needed CAP_NET_BIND_SERVICE on the httpd binary. On Linux the parent now binds the socket in post_config, while it still runs as root, and keeps it in the process pool, so it stays bound across graceful restarts. The children inherit it. One child at a time serves it: the child that binds an abstract Unix socket name owns the port, and the kernel releases the name when that child exits. Other platforms keep the old per-child bind.
With ListenCoresBucketsRatio, httpd splits its TCP listeners into buckets. The parent now binds one SO_REUSEPORT UDP socket per bucket, and each child serves the first bucket that no other child owns. So more than one child serves HTTP/3. The parent keeps the sockets while children start and stop, so the kernel hash does not change and each peer stays on the same socket. Without buckets, there is one socket as before.
Some MPMs count their listener buckets in the run step, after post_config, so ap_num_listen_buckets was still 1 there. The module now uses the same count as httpd: online cores / ratio, capped at the MPM daemon limit, because a bucket with no child loses its peers.
Worker threads allocate from stream subpools while the event thread allocates from the session pool. All of them use one allocator, which had no mutex, so concurrent allocations corrupted the heap and crashed the child under load. Set a mutex on the allocator, as mod_http2 does.
A pending handshake counts as an MPM connection. The event thread advanced pending handshakes only when not draining, so a handshake in flight at a graceful restart did not finish or time out. The old child then never reached zero connections and kept the port.
nghttp3_conn_resume_stream puts a blocked stream back in the write queue. The module blocked the stream only on the first refusal, so when the peer kept its window shut, nghttp3 offered the stream again and again and the event thread used 100% CPU.
OpenSSL's own QUIC server drops 0-RTT packets, so 0-RTT cannot work on it. This change runs the QUIC transport on ngtcp2 and keeps OpenSSL only for the TLS 1.3 handshake, through its QUIC TLS API. This is the same split that nginx uses. The h3q_ API does not change. Its implementation now reads the UDP socket, maps connection IDs to connections, answers Retry and Version Negotiation, and runs the ngtcp2 timers on a monotonic clock. The BIO filter that recovered peer addresses from the OpenSSL listener is removed. The engine does not copy stream data: it reports each acknowledgement through stream_acked, and nghttp3 releases the buffer when the peer acknowledges it. Connection close follows RFC 9000 section 10. A local close keeps its CONNECTION_CLOSE packet and resends it, at most once per PTO, for three PTOs. An idle timeout closes silently. The transport idle timer runs 2 seconds past H3IdleTimeout, so the module's clean close comes first. H3EarlyData (default off) turns on 0-RTT. With it on, a returning client sends its first request with the handshake and gets the response one round trip sooner. Only safe methods run before the handshake completes; other methods wait for it, because 0-RTT data can be replayed (RFC 8470). The TLS layer sets SSL_OP_NO_ANTI_REPLAY, because OpenSSL's own anti-replay is built for TLS over TCP and refuses QUIC resumption with early data. ngtcp2 >= 1.22.1 with its OpenSSL crypto helper (ngtcp2_crypto_ossl) is a new build dependency: --with-ngtcp2 for configure, NGTCP2_INCLUDE_DIR and NGTCP2_LIBRARIES for CMake.
The upload windows were fixed at 1 MiB per connection and 256 KiB per stream, so an upload on a long, fast path stalled on flow control. Turn on ngtcp2 window autotuning: when a client fills a window within a few round trips, ngtcp2 doubles it. H3MaxWindow (default 6 MiB, the ngtcp2 example default) caps the growth, per connection and per stream. The windows still start at 1 MiB and 256 KiB.
The engine sends the packets of one flush to one peer with one sendmsg and UDP_SEGMENT. Reads use UDP_GRO, and the engine splits a joined read into datagrams. If the kernel or the device refuses GSO, the engine turns it off and sends each packet alone. A stream read now sends the new flow-control window at once. Before, the update waited for the next packet from the peer. A blocked peer sends no packet, and with GRO an upload could stop.
… buckets On Linux the [::] socket now reads the address each packet came to (IP_PKTINFO, IPV6_PKTINFO) and sends the reply from it. Before, the kernel picked the source, and a host with more than one address could reply from another one, which a client drops. ECN: the ECN mark of each packet goes to ngtcp2, which reports it in ACK_ECN frames, and the mark ngtcp2 asks for goes on each sent packet. The parent makes the key for Retry and stateless reset tokens and keeps it across graceful restarts, so a token from one child is good in all. With more than one SO_REUSEPORT socket the server sends disable_active_migration.
The QUIC connection keeps its connection IDs in 16 slots of its own, not in an APR pool of 8 KB or more. It makes the CONNECTION_CLOSE buffer only when it closes. The session and stream allocators keep at most MaxMemFree, as in mod_http2. cid_add does not add a CID that another connection has. Before, the map could keep a key that pointed into the pool of a freed connection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #947, #948 and #949. This branch is stacked on #949, which contains #948. The first three commits are #947's; the five after them are this PR.
Why
modules/http3/README_FIRST.mdsays:#949 removes the single-process limit. This PR removes the other one.
With OpenSSL's QUIC server, OpenSSL owns the transport and mod_http3 works around it:
OPENSSL_memdupinh3q_addr.c), then pumpsSSL_handle_eventsin budgeted loops until the queue drains.sendmsg.ngtcp2 comes from the same project as nghttp2 and nghttp3, which httpd already uses. With it, OpenSSL still does TLS 1.3 through its QUIC TLS API, and the module owns the socket and the packet path.
Measured
httpd trunk with
--enable-http3=shared, OpenSSL 4.0, nghttp3 1.19, clienth2load --h3, medians of 3 runs. Left column is trunk with #947–#949. Client and server shared one 16-vCPU VM over loopback, so the ratios matter more than the absolute numbers.Per connection and per packet, ngtcp2 is 1.8× to 11.7× faster and uses 2.4× to 8× less server CPU per request. With few connections and many parallel requests the transport is not the bottleneck, and the two are even. On that host OpenSSL QUIC also failed some of the large downloads; ngtcp2 completed all of them.
What changes
H3EarlyData(0-RTT). The internalh3q_API is unchanged. The per-datagram BIO copy is gone, and stream data is not copied: nghttp3 releases each buffer when the peer acknowledges it. Connection close follows RFC 9000 §10.H3EarlyData(default off) lets a resumed client send its first request as 0-RTT data, one round trip sooner. Only safe methods run before the handshake completes (RFC 8470).H3MaxWindow(default 6 MiB), so uploads on long, fast paths are no longer held to 1 MiB.[::], replies come from the address the client sent to (IP_PKTINFO/IPV6_PKTINFO). ECN marks are passed through. The parent makes the Retry and stateless-reset token key, so tokens are valid in every child and across graceful restarts. With more than one bucket the server sendsdisable_active_migration.MaxMemFree, as mod_http2 does. A client cannot insert an Initial DCID that belongs to another connection.What this opens
Because the module now creates its own connection IDs, it can put the bucket number in them. That is what the eBPF idea from #935 ("eBPF linux kernel filters to allow multi process on the UDP socket") needs. A follow-up PR adds an
SK_REUSEPORTeBPF program that routes each packet to its bucket's socket by connection ID, so a client can change address and keep its connection across processes. Until then, the server sendsdisable_active_migrationwhen it uses more than one bucket.Dependency
This adds ngtcp2 ≥ 1.22.1 with its OpenSSL crypto helper:
--with-ngtcp2for configure,NGTCP2_INCLUDE_DIR/NGTCP2_LIBRARIESfor CMake. Without it mod_http3 is not built, the same as a missing nghttp3 today. ngtcp2 is MIT-licensed C with no dependencies beyond OpenSSL, and curl uses it for HTTP/3. EPEL 9 ships it asngtcp2-crypto-ossl-devel1.22.1.Tested on EL9 with nghttp3 1.19: OpenSSL 4.0 with ngtcp2 1.26, and the stock EL9 OpenSSL 3.5.8 with EPEL's ngtcp2 1.22.1, which gives the same numbers. No new warnings.