Standalone background worker and reusable Media Core for downloading, validating, processing, archiving, and storing media. It has no Discord, HTTP server, frontend, or Fastify dependency.
Serializable job
-> WorkerRuntime
-> Media Core
-> engine planner and fallback
-> artifact and FFprobe validation
-> optional processing
-> storage adapter
-> structured result
WorkerRuntime.execute() is the queue seam. The BullMQ entrypoint validates API contract version 1, passes a plain runtime job through this seam, relays progress, and returns the structured result. Queue transport does not belong inside Media Core.
import { createMediaCore } from "mediafilez-worker";
const core = createMediaCore();
const resolution = await core.resolveMedia(url, { outputType: "video", quality: "720p" });
const downloaded = await core.downloadMedia(url, jobDirectory, {
outputType: "video",
quality: "720p",
maxBytes: 5 * 1024 ** 3,
signal,
onProgress,
});
const inspected = await core.inspectMedia(downloaded.filePath, downloaded.fileName);
const output = await core.processMedia(inspected, {
outputType: "video",
format: "mp4",
maxOutputBytes: 500 * 1024 ** 2,
allowCompression: true,
tempDir: jobDirectory,
signal,
onProgress,
});The public quality choices are controlled variant IDs: best, 1080p, 720p, 480p, 360p, and audio-only. Raw yt-dlp format selectors are not accepted as job input.
best uses codec-neutral video and audio selection, with no forced MP4 merge container. format: "original" preserves the selected source; original audio extracted from a combined video is copied without re-encoding. Request mp4 or mp3 explicitly for conversion. Platform variants and fallback engines can expose different quality ceilings. The API and partner contract is documented in the companion API's docs/partners.md.
No output size limit is applied unless the job supplies output.maxBytes. MAX_DOWNLOAD_SIZE is an operator-controlled ceiling for every source transfer and defaults to 5 GB. Batch jobs also share the MAX_BATCH_TOTAL_SIZE aggregate ceiling, which defaults to 10 GB.
Instagram tries VideoDropper first using its encrypted URL protocol. Its resolver has a 20-second deadline, and media transfers keep the existing DNS, redirect, byte-limit and artifact checks. Provider failures continue through the existing Instagram engine order. INSTAGRAM_PROXY_FIRST puts that proxy after VideoDropper; DISABLED_ENGINES=videodropper omits the new engine. This uses public media URLs and sends no Instagram cookies or worker credentials to VideoDropper.
Download job:
{
"type": "download",
"jobId": "job_123",
"url": "https://example.com/video",
"output": {
"type": "video",
"format": "mp4",
"quality": "720p",
"maxBytes": 524288000,
"allowCompression": true
}
}Batch jobs accept 1–100 download items and create a streamed ZIP archive. Each item is bounded by MAX_DOWNLOAD_SIZE. Processed batch outputs and the ZIP stream are each bounded by MAX_BATCH_TOTAL_SIZE, including ZIP overhead. Duplicate entry names are disambiguated, and missing inputs fail the archive instead of silently omitting files. Storage keys are generated by the storage adapter; jobs cannot overwrite arbitrary objects by supplying a key. Cleanup jobs remove only abandoned temporary directories whose process ownership can be proven inactive.
JOB_TIMEOUT_MS covers one runtime execution, including its retries and backoff. Failed attempts release their temporary directories before retrying. Progress callbacks are best effort: a reporting failure is logged and cannot repeat a completed upload. Engines and storage adapters must honor the provided signal; the deadline cannot forcibly interrupt an adapter that ignores it. BullMQ deliveries each start their own runtime execution.
audio-only quality applies to both downloading and processing, including extraction from direct video files. Its format must be original or mp3; incompatible mp4 and jpg requests fail validation. Explicit MP3 conversion does not require allowCompression unless an oversized source must be fitted to an output limit.
Run a job locally:
pnpm install
pnpm run build
node dist/worker.js ./job.jsonRun the connected queue worker with REDIS_URL, QUEUE_PREFIX, and shared R2 credentials configured:
pnpm startThe API and Worker must use the same Redis URL, queue prefix, R2 bucket, and contract version. Active cancellation is delivered through a short-lived Redis control key and becomes an AbortSignal inside the runtime.
Public CDN jobs use delivery: "public" in the version 1 job. Set R2_PUBLIC_BUCKET to a separate R2 bucket attached to a public custom domain. The Worker stores public files there with their detected media type and inline disposition; private jobs continue to use R2_BUCKET. For a public video, it also stores a small JPEG thumbnail and returns its object key when extraction succeeds. Thumbnail failure does not fail the original download. Public jobs do not apply a Discord upload size limit unless the API request explicitly asks for processing. Keep retention and cache policy for the public bucket aligned with your deployment needs.
Progress is written as JSON Lines to stderr. The final result is written to stdout.
MediaStorage is the storage seam. Runtime jobs use a stable identity derived from job ID and artifact role. The included adapters recover existing results and reject reuse of a job ID with different input. Local storage atomically commits a directory containing the file and result metadata. R2 uses conditional create-only writes and object metadata. Legacy calls to put without identity retain UUID keys. Custom adapters need to implement find and create-only identity semantics for recovery.
Download engines and FFmpeg must never upload to storage directly. Retention is an operator policy. delete(key) targets private storage; use delete(key, { public: true }) for public originals and previews. Delete both key and thumbnailKey when a preview exists. See reliability and isolation for recovery boundaries and deployment controls.
- HTTP/HTTPS-only input URLs
- private, loopback, link-local, documentation, multicast, and reserved address blocking
- DNS checks before requests and socket-level lookup checks against rebinding
- redirect validation and bounded response bodies
- streaming transfer byte limits
- required content signatures, media kind checks, and FFprobe validation for audio/video
- sanitized file names and storage traversal protection
- extractor child processes with
shell: false, bounded output capture, timeouts, process-group termination, and cancellation; FFmpeg/FFprobe also use bounded, shell-free execution - owned temporary directories and safe stale-directory cleanup
- bounded image dimensions and pixel count before every FFmpeg decode
Guarded HTTP and YouTube.js fetches validate redirects and socket DNS. Cobalt directory discovery uses the guarded fetch transport. Native yt-dlp and gallery-dl receive a job-scoped authenticated proxy that checks destinations and connects to a verified address. Their child environments omit worker credentials and clear proxy exclusions. This does not prevent native tools or auxiliary programs from bypassing the proxy: deployment must enforce egress separately. Configured Cobalt origins, including their exact ports, are explicit trusted exceptions for private API and tunnel access. Cobalt POST redirects are refused. External engine relays receive the source URL and require their own egress controls.
MAX_JOB_TEMP_SIZE bounds aggregate temporary bytes through serial monitoring and checks before local storage commits. MAX_JOB_PROCESSES bounds worker-launched tools per job. Linux additionally applies prlimit CPU and per-file limits; the Docker image includes util-linux. Hard per-job disk quotas, memory limits and complete descendant isolation require deployment support. Production Compose adds container-wide memory, CPU and PID limits.
pnpm run checkThis builds TypeScript, verifies the Discord/Fastify independence rule, runs the migrated MediaFilez core tests and Worker tests, and checks formatting.
For real Redis integration, point TEST_REDIS_URL at a disposable test server and run pnpm run test:redis. Tests use a random queue prefix and never default to REDIS_URL. Without that variable the suite reports a skip.
The mature downloader implementation remains JavaScript during the first extraction phase and is compiled through allowJs. New Worker/Core interfaces are TypeScript. Modules can be converted incrementally while the interface tests continue to prove behavior.
The Worker reuses HTTP keep-alive connections for direct media transfers and gives page metadata probes a shorter, independent timeout so blocked pages do not delay the next engine. Set PAGE_METADATA_TIMEOUT_MS (5–60 seconds) when a deployment needs a different probe budget. HTTP_RESPONSE_TIMEOUT_MS and HTTP_IDLE_TIMEOUT_MS still control actual media transfers.
Remote transfer time remains bounded by the source server, network path, and file size. Cookies can significantly reduce social-site challenge and login delays; set MEDIA_COOKIES_FILE to a private Netscape-format cookie file and keep it outside the repository.
Startup rejects missing, directory or malformed cookie mounts before claiming queue jobs. yt-dlp and gallery-dl use private writable copies per attempt; the host mount stays read-only. Compose refuses to create a host directory for a missing cookie file. A worker heartbeat lets the API distinguish its own liveness from queue readiness. Instagram posts return all resolved photos and videos through result.items, while result.output remains the first file. Other native playlists stay restricted to the first item; use explicit batch jobs for them. See collection storage and retry behavior.
Instagram authentication specifically requires a non-expired sessionid cookie scoped to instagram.com. A file without it can still be useful for other sites, but cannot authenticate Instagram. INSTAGRAM_PROXY_HOSTS is empty by default: set it only to a service you operate or explicitly trust that returns media files rather than HTML pages.
For a split API/Worker deployment, use compose.production.yaml. It joins the API queue network and the separate Cobalt engine network, so the configured http://cobalt:9000 endpoints stay reachable after a Worker redeploy. Set MEDIA_COOKIES_HOST_PATH to the private host path before running docker compose -f compose.production.yaml up -d --build.
YOUTUBE_JS_ENABLED is disabled by default because youtubei.js needs a separately configured JavaScript evaluator. Leave it disabled unless that evaluator is installed. When the cookie file contains a valid YouTube session, set YTDLP_COOKIES_FOR_YOUTUBE=true to avoid avoidable login challenges.
The command-line fallbacks use one retry and a 15-second connection timeout by default. gallery-dl also has a 30-second total process guard so an extractor that hangs cannot hold a job indefinitely. These settings keep blocked or unavailable sources moving to the next engine quickly while still allowing a transient retry. Tune YTDLP_RETRIES, YTDLP_SOCKET_TIMEOUT_SECONDS, GALLERY_DL_RETRIES, GALLERY_DL_HTTP_TIMEOUT_SECONDS, and GALLERY_DL_TIMEOUT_MS for slower sources.