Repository navigation
normalize_url lowercases the entire URL, silently deduplicating case-distinct pages #2008
Description
Description
normalize_url lowercases the entire URL, including the path and query, even though its docstring says it only converts the scheme and netloc to lower case:
crawlee-python/src/crawlee/_utils/requests.py
Lines 41 to 47 in 8cc09f2
Since compute_unique_key uses the normalized URL as the default unique_key, any two URLs that differ only in path or query casing collide:
https://example.com/Product/ABCandhttps://example.com/product/abcproduce the same unique key.https://example.com/?token=SeCrEtandhttps://example.com/?token=secretcollide as well.
Per RFC 3986, only the scheme and host are case-insensitive. The path and query are case-sensitive.
Impact
On sites with case-sensitive paths (base64 or hashid identifiers, usernames, ...), case-distinct pages are silently deduplicated. The crawl finishes successfully with pages quietly missing. There's no log message and no statistic that would reveal it.
Proposed fix
Lowercase only the scheme and host, which matches browser behavior. Keep the path, query, and fragment casing intact.
This changes how default unique keys are computed: crawls that relied on the case-insensitive dedup will now visit more pages, and keys stored in persisted queues won't match newly computed ones. It should therefore land in 2.0 as a breaking change:
- Document the change loudly in the upgrading guide.
- Users who need the old behavior can pass an explicit
unique_keyor usetransform_request_functionto lowercase URLs before enqueuing.
Activity
ayushcodes13 commentedon Aug 10, 2026
I can take a pass at this with a focused regression test.
I’ll first pin the current behavior around
normalize_url/ defaultunique_keygeneration, then update the normalization so only scheme and host are lowercased while path/query/fragment casing is preserved.vdusek commentedon Aug 17, 2026
Hi @ayushcodes13, this is planned for v2 because it will introduce a breaking change.
enriquealonso01 commentedon Sep 21, 2026
Submitted a PR for the v2 milestone: #2241 — drops the blanket
.lower()innormalize_url(path/query keep casing per RFC 3986 §6.2.2.1), includes the unique-key migration note for the 2.0 upgrade guide, plus regression tests. All 17 tests intests/unit/_utils/test_requests.pypass.enriquealonso01 commentedon Sep 28, 2026
Fix proposed in #2259 (rebased onto current master, all unit tests in tests/unit/_utils/test_requests.py passing).
vdusek commentedon Sep 29, 2026
Hi @enriquealonso01, as mentioned above and in your previous PR, this is something intended to be implemented in v2. Since v2 is not currently under development, this issue is just a placeholder for a future implementation.
fix!: keep URL path, query and fragment casing in the default `unique…
vdusek commentedon Oct 31, 2026
resolved by #2371
fix!: keep URL path, query and fragment casing in the default `unique…