Skip to content

fix(nodes): keep the scheme of absolute URLs extracted by ParseNode - #1172

Open
MohammadHijjawi97 wants to merge 1 commit into
ScrapeGraphAI:pre/betafrom
MohammadHijjawi97:fix/parse-node-absolute-urls
Open

MohammadHijjawi97 wants to merge 1 commit into
ScrapeGraphAI:pre/betafrom
MohammadHijjawi97:fix/parse-node-absolute-urls

Conversation

@MohammadHijjawi97

Copy link
Copy Markdown
Contributor

Summary

ParseNode.url_pattern begins with [http[s]?:\/\/]?, which is a character class rather than an optional http(s):// group. The scheme is therefore never part of the match. When parse_urls=True (OmniScraperGraph), every absolute link or image URL loses its scheme and is then joined onto the page URL as if it were relative:

node = ParseNode(input="doc & (url | local_dir)", output=["parsed_doc", "link_urls", "img_urls"],
                 node_config={"chunk_size": 4096, "parse_urls": True, "llm_model": None})
text = "[guide](https://docs.example.org/guide/start) ![logo](https://cdn.example.net/img/logo.png)"
node._extract_urls(text, "https://example.com/blog/post")
# before: (['https://example.com/blog/docs.example.org/guide/start'], ['https://example.com/blog/cdn.example.net/img/logo.png'])
# after:  (['https://docs.example.org/guide/start'], ['https://cdn.example.net/img/logo.png'])

With a local HTML source the same URLs were dropped entirely, because only results starting with http are kept. The broken img_urls are the ones OmniScraperGraph sends to ImageToTextNode.

Changes

  • Capture the scheme with (https?:\/\/). The pattern already required ://, so the set of matched URLs is unchanged; they now keep their scheme. Relative URLs are still resolved against the page URL.
  • Added tests/nodes/parse_node_test.py covering absolute and relative links and images, a local source, and execute() with parse_urls=True.

Testing

  • pytest tests/nodes/parse_node_test.py: 3 failed on pre/beta, 3 passed with the fix.
  • pytest tests/test_error_page_detection.py: passes.
  • ruff check, black --check and isort --check-only on the changed files, at the versions pinned in .pre-commit-config.yaml.

The scheme part of ParseNode.url_pattern was written as a character
class, so the match never captured "http(s)://". Absolute links and
image URLs were then joined onto the page URL as if they were relative
(https://docs.example.org/guide on https://example.com/blog/post became
https://example.com/blog/docs.example.org/guide), and dropped entirely
when the source was local HTML. This affects the link_urls and img_urls
used by OmniScraperGraph.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant