Skip to content

feat: support timeouts in the send_request context helper - #2174

Open
Battleplus wants to merge 1 commit into
apify:masterfrom
Battleplus:feat/send-request-timeout
Open

feat: support timeouts in the send_request context helper#2174
Battleplus wants to merge 1 commit into
apify:masterfrom
Battleplus:feat/send-request-timeout

Conversation

@Battleplus

Copy link
Copy Markdown

Closes #2138

Summary

SendRequestFunction (used via context.send_request() in request handlers) only accepted url, method, payload and headers, so a handler could not bound how long an extra HTTP call may take. All four HTTP client implementations (HttpxHttpClient, CurlImpersonateHttpClient, ImpitHttpClient, PlaywrightHttpClient) already support a per-request timeout: timedelta | None — this PR just exposes it through the public contract.

Changes

  • src/crawlee/_types.py: add timeout: timedelta | None = None to the SendRequestFunction protocol + docstring.
  • src/crawlee/crawlers/_basic/_basic_crawler.py: thread timeout through the _prepare_send_request_function closure into HttpClient.send_request.
  • src/crawlee/crawlers/_playwright/_playwright_http_client.py: fixPlaywrightHttpClient passed timeout.total_seconds() to Playwright's APIRequestContext.fetch, which expects milliseconds. Without this fix, any timeout passed via context.send_request() on a Playwright crawler would be ~1000x too short (e.g. timedelta(seconds=12) → 12ms).
  • tests/unit/crawlers/_basic/test_basic_crawler.py: regression test — context.send_request('/slow?delay=2', timeout=timedelta(milliseconds=100)) raises asyncio.TimeoutError; plus a unit test asserting timeout is forwarded to HttpClient.send_request.
  • tests/unit/crawlers/_playwright/test_playwright_http_client.py: regression test asserting the seconds→milliseconds conversion.

Validation

  • pytest tests/unit/crawlers/_basic/test_basic_crawler.py → 112 passed, 1 skipped
  • New tests (forwarding, real timeout, Playwright ms conversion) all pass
  • ruff check + ruff format --check clean on all touched files

Note: no new dependency; timeout uses the existing timedelta convention across the Python HTTP clients (the JS implementation names it timeoutMillis, but the Python clients are timedelta-based).

Expose the HTTP clients' existing per-request `timeout` support through
`SendRequestFunction` so request handlers can bound how long an extra
HTTP call may take. Threads the value through `context.send_request()`
into `HttpClient.send_request`.

Also fixes `PlaywrightHttpClient` which passed `timedelta.total_seconds()`
to Playwright's `APIRequestContext.fetch`, which expects milliseconds —
previously any timeout would be ~1000x too short. Adds regression tests
for both the forwarding and the unit conversion.

Closes apify#2138

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support timeouts in send_request

2 participants