Repository navigation
When a site blocks the server, end the import error with a yt-dlp command to run at home - #75
Merged
Merged
Conversation
Impersonating a browser was not enough on the production server: Rumble's watch page answers its IP with Cloudflare's JavaScript challenge (403, cf-mitigated: challenge), which nothing but a real browser can pass. The same server reaches Rumble's oEmbed API and embed player without a challenge. So when a rumble.com watch page still answers 403 as a browser, ask oEmbed for the video's embed id and import rumble.com/embed/<id>/, recording the watch page as the asset's source URL so the already-imported check still matches. The embed lacks the description and tags; the title, channel, date, duration and thumbnail come through. The 403 message now names the likely cause, a Cloudflare challenge bound to the server's IP, instead of suggesting a cookies file, which cannot carry that clearance from another machine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V1geD4inqJxe1F6HuWV82R
The production server proved the embed route useless: Cloudflare challenges its IP on Rumble's embed data endpoint (embedJS/u3) as well as the watch page, with or without a Referer, leaving only oEmbed open, which carries no video. No request from that IP can import from Rumble. So remove the fallback and add URL_IMPORT_PROXY, an http(s) or SOCKS proxy for imports, with URL_IMPORT_PROXY_HOSTS limiting it to the sites that need it (a name covers its subdomains; empty means every import). It is chosen by the pasted link and set as yt-dlp's proxy for the whole run, so the video files on the site's CDN go the same way. A value that is not a proxy URL stops startup. docs/url-import.md walks through a free proxy: an SSH reverse SOCKS tunnel from a computer at home, bound to Docker's bridge address. The 403 message now points there instead of at a cookies file, which cannot carry Cloudflare's clearance from another machine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V1geD4inqJxe1F6HuWV82R
davior
changed the title
…home Decided against the proxy: an SSH tunnel home exposes the home network to anything on the server, and a residential proxy is a subscription for an occasional video. Rumble videos will be downloaded locally and uploaded. So the 403 that survives the browser retry now says what it usually is, Cloudflare blocking this server's IP, and ends with the yt-dlp command that fetches the same link the way an import would: --embed-metadata so the upload reads back the channel, date and link, the title as filename, the same height cap and MP4 container. The link is percent-encoded for anything a shell would act on inside double quotes, so a pasted URL cannot run anything when the command is pasted into a terminal. The activity panel renders a line starting `yt-dlp ` as code with a copy button, since a command is only useful whole and exact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V1geD4inqJxe1F6HuWV82R
davior
changed the title
davior
marked this pull request as ready for review
November 7, 2026 04:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #74. On the production server, Cloudflare challenges the server's IP on every Rumble path that leads to a video (
403,cf-mitigated: challenge), even with browser impersonation. That leaves only oEmbed open, which carries no video.This PR's earlier commits tried two ways round it, and both were set aside:
Rumble videos will be downloaded locally and uploaded instead. The net diff against main is only what that needs.
Changes
app/ingest/ytdlp.py:local_download_command. The command uses:--embed-metadata, so the upload reads back channel, date and link (embedded_metadata.py),",$, backtick,!,\, space). A pasted URL can't run anything when the command is pasted into a terminal, and it still names the same page.ActivityIndicator.tsx: an error line startingyt-dlprenders as a code block with a Copy button. If the clipboard is unavailable, it falls back towindow.prompt, asAttributionPaneldoes.docs/url-import.md: the Cloudflare case, the check to run on a server, the local-download route, and why the proxy was set aside.Testing
pytest -q: 1602 passed. New tests:npm test: 523 passed.npm run lintandnpm run buildare clean. New tests: the command renders as<code>apart from the prose and the Copy button copies it exactly; an error without a command has no button.🤖 Generated with Claude Code
https://claude.ai/code/session_01V1geD4inqJxe1F6HuWV82R