Refactor the HTTP Engine by yosuke-wolfssl · Pull Request #59 · wolfSSL/wolfCert

yosuke-wolfssl · 2026-10-19T03:01:35Z

Problem

src/http.c had two response engines, a blocking reader and the _nb state machine, and they had drifted: chunked bodies, bytes past a body, handshake errors and bodies ending at the close were each handled differently. The state machine also copied every body, so a response peaked at about twice its size on the heap.

Fix (`src/http.c`)

The state machine is the only engine; the blocking reader (about 450 lines) is deleted.

5f1f26d runs blocking, _nb and one-shot calls on it, chunked bodies included; bytes past a response close the session.
bf4fcfd runs the TLS handshake on the first request and removes nonblocking.
22a83e3 (@Frauschi) caps the header block at 8 KiB and hands the receive buffer over as resp->body.
68ae328 trims the session struct and says why a truncated response failed.

Change	Effect on callers
`WolfCertHttpSessionCfg.nonblocking` removed	Code setting it no longer compiles; on 32-bit targets later fields move 4 bytes. Must land before the first major release
Handshake on the first request	`*_session_open()` no longer verifies the server; the first request returns `ERR_TLS` (blocking used to get `ERR_CRYPTO`) or `ERR_MEMORY`, with `wolfcert_last_wolfssl_err()` set
A call while a request is in flight	Only `_nb` with the same `resp` continues it; any other call gets `BAD_ARG`
A closed session (after an error, a body ending at the close, or bytes past a reply)	`ERR_CONN_CLOSED` (was `BAD_ARG`)
Limits	A header block over 8 KiB is `ERR_PROTOCOL`; a body ending at the close is capped at `max_response_bytes`
Static memory pool	A chunked or close-delimited reply first tries one block of `max_response_bytes` + 8 KiB, else grows as before (up to 8192 bytes on default buckets); beyond that, `ERR_MEMORY` naming both settings (`EMBEDDED.md` §4)
EST `_open` / `_open_async`	Open the same session
Custom `WolfCertTransport`	May see timeouts `0` and `-1` on one connection (`ARCHITECTURE.md` §4.6)

Verification

Each commit passes 30/30 with -Werror. The tip is clean under ASan + UBSan, ASan + LeakSanitizer (Ubuntu 24.04) and the nightly ThreadSanitizer set, and passes autotools make check and the static-memory build.
The ten test_nb_* copies merge into cases run both ways. New cases cover chunked _nb replies, truncation, bytes past a reply, the header budget, in-flight calls and handshake failures. Reverting a change fails a test, except the non-waiting close_notify, the resumed header search and the global heap hint branch.

Not in this PR

The EST/SCEP protocol layer (one-shots over sessions, SCEP's mode split, removing _open_async, ARCHITECTURE.md §4.3) is the next PR.
Follow-ups: bytes past a reply are caught only when they arrive with its headers; a server's Connection: close; timeouts on blocking reads; Transfer-Encoding with Content-Length; short-write and WANT_WRITE tests.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

wolfSSL-Fenrir-bot

Fenrir Automated Review — PR #59

Scan targets checked: wolfcert-src, wolfcert-bugs
Not reviewed (no scan target covers them): docs/EMBEDDED.md, src/http.c
Coverage: 5 of 8 in-scope changed file(s) opened by the reviewer; not opened: src/scep/scep_client.c, tests/unit/test_http.c, wolfcert/est.h

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Lite

Frauschi

Built the head and ran the suite plain with -Werror and under ASan+UBSan (30/30 both), plus leaks on the transport, TLS and EST/SCEP session tests. All clean. Having the state machine as the only engine is a nice simplification. A few smaller items inline.

- sm_read_head() reads at most WOLFCERT_HTTP_HEADER_BUDGET bytes of header block and fails with WOLFCERT_ERR_PROTOCOL and a message once the budget is reached; nb_read_some() takes the limit sm_rx may fill to and fails when sm_rx already holds more. - chunked_decode_into() decodes a chunked body in place; wolfcert_http_chunked_decode() wraps it, and DynBuf and dyn_append() are removed. - Chunked, Content-Length and close-delimited bodies hand sm_rx over as the response body through sm_take_body(). A Content-Length body grows sm_rx once to the whole response; a close-delimited body is capped at max_response_bytes. - With WOLFSSL_STATIC_MEMORY and a pool in use, sm_rx grows to the full allowance in one step for a chunked or close-delimited body. - docs/EMBEDDED.md and README.md describe the header budget, the client response buffer and pool sizing. - Tests cover a header block over the budget and a close-delimited body over the cap, and the cap tests pad the body instead of the headers. Based on the patch from the wolfSSL#59 review, without its separate Content-Length buffer and in-place header parsing.

- The non-blocking state machine, with a new chunked body state, is the only response engine; the blocking reader is removed. - Blocking and _nb session requests share http_session_call(); while a request is in flight, only _nb with its resp continues it. - wolfcert_http_request() runs over a session with Connection: close. - Bytes past a response close the session, and a closed session gets WOLFCERT_ERR_CONN_CLOSED; residual and its replay are removed. - The request head is wiped once sent, and the header search resumes where the previous read stopped. - wolfcert/http.h, src/internal.h and docs/EMBEDDED.md describe the changes, and tests cover each of them.

- Remove WolfCertHttpSessionCfg.nonblocking; a TLS session starts in SM_HANDSHAKE and either request call works on any session. - A failed handshake returns WOLFCERT_ERR_TLS, or WOLFCERT_ERR_MEMORY when memory runs out, with the wolfSSL code; session close sends close_notify without waiting. - EST's two session opens now give the same session. - Docs cover mode changes between requests for transports, the new WOLFCERT_ERR_RC_WC(), session open not verifying the server, and req->body staying valid across _nb polls. - est_client_cases.h prints the rc, the wolfSSL code and the message when session_get_cacerts() fails in session_basic_auth(). - Tests drop the test_nb_* copies and cover truncated replies, the TLS error, an allocation failure in the handshake and mixed calls on EST sessions.

- sm_read_head() reads at most WOLFCERT_HTTP_HEADER_BUDGET bytes of header block and fails with WOLFCERT_ERR_PROTOCOL and a message once the budget is reached; nb_read_some() takes the limit sm_rx may fill to and fails when sm_rx already holds more. - chunked_decode_into() decodes a chunked body in place; wolfcert_http_chunked_decode() wraps it, and DynBuf and dyn_append() are removed. - Chunked, Content-Length and close-delimited bodies hand sm_rx over as the response body through sm_take_body(). A Content-Length body grows sm_rx once to the whole response; a close-delimited body is capped at max_response_bytes. - With WOLFSSL_STATIC_MEMORY and a pool in use, sm_rx first tries to grow to the full allowance in one step for a chunked or close-delimited body; when growing fails, nb_read_some() returns WOLFCERT_ERR_MEMORY with a message naming the settings. - docs/EMBEDDED.md describes the header budget, the client response buffer, pool sizing and how to fit the one-step block into the pool's largest bucket. - Tests cover a header block over the budget, a close-delimited body over the cap and chunked replies under a cap no pool block fits, and the cap tests pad the body instead of the headers. Based on the patch from the wolfSSL#59 review, without its separate Content-Length buffer and in-place header parsing.

- inspect_headers() picks the body state and resets sm_scan; sm_chunked is removed. - sm_tx_off replaces sm_head_off and sm_body_off. - closed and sm_head_request are one-bit fields. - sm_read_head(), sm_read_body_cl() and sm_read_body_chunked() report a close before the response ends as WOLFCERT_ERR_IO with a message; the Content-Length one gives the bytes received and expected. - test_truncated_reply_is_io() adds a cut-off chunked reply and checks each message.

- sm_read_head() reads at most WOLFCERT_HTTP_HEADER_BUDGET bytes of header block and fails with WOLFCERT_ERR_PROTOCOL and a message once the budget is reached; nb_read_some() takes the limit sm_rx may fill to and fails when sm_rx already holds more. - chunked_decode_into() decodes a chunked body in place; wolfcert_http_chunked_decode() wraps it, and DynBuf and dyn_append() are removed. - Chunked, Content-Length and close-delimited bodies hand sm_rx over as the response body through sm_take_body(). A Content-Length body grows sm_rx once to the whole response; a close-delimited body is capped at max_response_bytes. - With WOLFSSL_STATIC_MEMORY and a pool in use, sm_rx grows to the full allowance in one step for a chunked or close-delimited body. - docs/EMBEDDED.md and README.md describe the header budget, the client response buffer and pool sizing. - Tests cover a header block over the budget and a close-delimited body over the cap, and the cap tests pad the body instead of the headers. Based on the patch from the wolfSSL#59 review, without its separate Content-Length buffer and in-place header parsing.

- sm_read_head() reads at most WOLFCERT_HTTP_HEADER_BUDGET bytes of header block and fails with WOLFCERT_ERR_PROTOCOL and a message once the budget is reached; nb_read_some() takes the limit sm_rx may fill to and fails when sm_rx already holds more. - chunked_decode_into() decodes a chunked body in place; wolfcert_http_chunked_decode() wraps it, and DynBuf and dyn_append() are removed. - Chunked, Content-Length and close-delimited bodies hand sm_rx over as the response body through sm_take_body(). A Content-Length body grows sm_rx once to the whole response; a close-delimited body is capped at max_response_bytes. - With WOLFSSL_STATIC_MEMORY and a pool in use, sm_rx first tries to grow to the full allowance in one step for a chunked or close-delimited body, and grows as bytes arrive when that fails. - docs/EMBEDDED.md describes the header budget, the client response buffer, pool sizing and how to fit the one-step block into the pool's largest bucket. - Tests cover a header block over the budget, a close-delimited body over the cap and a chunked reply under a cap larger than any pool block, and the cap tests pad the body instead of the headers. Based on the patch from the wolfSSL#59 review, without its separate Content-Length buffer and in-place header parsing.

- sm_read_head() reads at most WOLFCERT_HTTP_HEADER_BUDGET bytes of header block and fails with WOLFCERT_ERR_PROTOCOL and a message once the budget is reached; nb_read_some() takes the limit sm_rx may fill to and fails when sm_rx already holds more. - chunked_decode_into() decodes a chunked body in place; wolfcert_http_chunked_decode() wraps it, and DynBuf and dyn_append() are removed. - Chunked, Content-Length and close-delimited bodies hand sm_rx over as the response body through sm_take_body(). A Content-Length body grows sm_rx once to the whole response; a close-delimited body is capped at max_response_bytes. - With WOLFSSL_STATIC_MEMORY and a pool in use, sm_rx grows to the full allowance in one step for a chunked or close-delimited body, and fails with a message naming the settings when no pool block is that large. - docs/EMBEDDED.md describes the header budget, the client response buffer, pool sizing and how to fit the one-step block into the pool's largest bucket. - Tests cover a header block over the budget, a close-delimited body over the cap and a chunked reply whose cap no pool block fits, and the cap tests pad the body instead of the headers. Based on the patch from the wolfSSL#59 review, without its separate Content-Length buffer and in-place header parsing.

Frauschi

Thanks for the thorough updates! All is fixed except one small static-memory issue, see #59 (comment)

yosuke-wolfssl self-assigned this Oct 19, 2026

Copilot AI balanced review requested due to automatic review settings October 19, 2026 03:01

Copilot AI reviewed Oct 19, 2026

yosuke-wolfssl requested a review from wolfSSL-Fenrir-bot October 19, 2026 03:03

wolfSSL-Fenrir-bot reviewed Oct 19, 2026

View reviewed changes

yosuke-wolfssl force-pushed the fix/http-engine branch 4 times, most recently from b37906b to d464770 Compare October 19, 2026 06:15

yosuke-wolfssl assigned Frauschi and unassigned yosuke-wolfssl Oct 19, 2026

yosuke-wolfssl requested a review from Frauschi October 19, 2026 06:16

Frauschi reviewed Oct 19, 2026

View reviewed changes

Comment thread src/http.c

Comment thread src/http.c Outdated

Comment thread src/http.c

Comment thread wolfcert/http.h

Comment thread src/http.c

Comment thread src/http.c Outdated

Comment thread src/http.c

Comment thread src/http.c Outdated

Frauschi assigned yosuke-wolfssl and unassigned Frauschi Oct 19, 2026

yosuke-wolfssl force-pushed the fix/http-engine branch from d464770 to debe4b5 Compare October 20, 2026 01:32

yosuke-wolfssl and others added 4 commits October 20, 2026 10:47

yosuke-wolfssl force-pushed the fix/http-engine branch from debe4b5 to 91253a8 Compare October 20, 2026 01:50

yosuke-wolfssl force-pushed the fix/http-engine branch from 91253a8 to 3839b30 Compare October 20, 2026 02:43

yosuke-wolfssl force-pushed the fix/http-engine branch 2 times, most recently from 354e3ca to 87f7cb8 Compare October 20, 2026 05:50

yosuke-wolfssl requested a review from Frauschi October 20, 2026 06:12

yosuke-wolfssl assigned Frauschi and unassigned yosuke-wolfssl Oct 20, 2026

Frauschi requested changes Oct 20, 2026

View reviewed changes

Frauschi assigned yosuke-wolfssl and unassigned Frauschi Oct 20, 2026

yosuke-wolfssl force-pushed the fix/http-engine branch from 87f7cb8 to 68ae328 Compare October 20, 2026 07:39

yosuke-wolfssl assigned Frauschi and unassigned yosuke-wolfssl Oct 20, 2026

Frauschi approved these changes Oct 20, 2026

View reviewed changes

Frauschi merged commit 9f8686a into wolfSSL:main Oct 20, 2026
30 checks passed

yosuke-wolfssl deleted the fix/http-engine branch October 20, 2026 11:02

yosuke-wolfssl mentioned this pull request Oct 21, 2026

Refactor EST and SCEP protocol engine #62

Merged

Navigation Menu

Repository navigation

Refactor the HTTP Engine - #59

Refactor the HTTP Engine#59
Frauschi merged 4 commits into
wolfSSL:mainfrom
yosuke-wolfssl:fix/http-engine

yosuke-wolfssl commented Oct 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

wolfSSL-Fenrir-bot left a comment

Uh oh!

Frauschi left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Frauschi left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

yosuke-wolfssl commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix (src/http.c)

Verification

Not in this PR

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

wolfSSL-Fenrir-bot left a comment

Choose a reason for hiding this comment

Fenrir Automated Review — PR #59

Uh oh!

Frauschi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Frauschi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yosuke-wolfssl commented Oct 19, 2026 •

edited

Loading

Fix (`src/http.c`)