Repository navigation
[CI only] feat(web-fetch): AI/ML API Toolbox scraping backend - #2
Closed
hugoaimlapi wants to merge 5 commits into
Closed
hugoaimlapi wants to merge 5 commits into
hugoaimlapi wants to merge 5 commits into
Conversation
WebFetch can already replace its raw HTTP fetch with Firecrawl's scrape endpoint. This adds the AI/ML API Toolbox as a second scraping backend, selected with WEB_FETCH_PROVIDER=aimlapi and authenticated with the existing AIMLAPI_API_KEY. Toolbox runs are billed per page, so the backend is opt-in only: an AI/ML API key on its own never changes how WebFetch fetches pages. When selected it takes precedence over a Firecrawl key. - Runs the search1api/crawl tool, chosen by measurement against the other page-to-Markdown tools ($0.0013 per run, main-content Markdown with links, unbilled is_error runs for missing pages), with max_cost_usd capped at $0.005. - Polls queued (202) runs, reports is_error runs and over-cost refusals as WebFetch errors, and retries 429/502/503/504 with the same idempotency key so a retry is never charged twice. - Sends the key and the X-AIMLAPI-Source / X-AIMLAPI-Partner-ID attribution headers only to the fixed tools.aimlapi.com endpoint. - Like the Firecrawl path, a failed scrape is reported rather than retried over plain HTTP.
An aimlapi provider profile that points at the user's own proxy exports the proxy's credential as AIMLAPI_API_KEY. The WebFetch Toolbox backend now treats that key as not configured when the route is aimlapi and OPENAI_BASE_URL (or OPENAI_API_BASE) is not the canonical https://api.aimlapi.com/v1, so it is never sent to tools.aimlapi.com.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…ndling Apply the same hardening Twigpine#2465 gave the AI/ML API web search backend: - send only the first usable key of a comma-separated AIMLAPI_API_KEY pool (firstUsableCredential), instead of the whole list as one Bearer token; - treat the OPENAI_API_BASE alias like OPENAI_BASE_URL when deciding that an aimlapi profile points at a proxy, so its credential is withheld; - send both Toolbox requests with redirect: 'error', so the key and the attribution headers never reach another host; - redact the active key from Toolbox error text before it is shown. Tests cover each case.
… use Word the $0.005 max_cost_usd as a limit each run requests, as the merged search docs do, and note that a pooled AIMLAPI_API_KEY contributes only its first usable key.
…etup guide Describe the requested $0.005 cost limit, the OPENAI_API_BASE alias in the proxy check, first-key use of a pooled AIMLAPI_API_KEY, and that Toolbox requests never follow redirects.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI validation run for the upstream PR to Twigpine/openclaude. Not for merge; will be closed once PR Checks finish.