[pull] master from apify:master - #293
Merged
Merged
Conversation
Adds `@crawlee/otel`, an OpenTelemetry instrumentation for Crawlee v4. It patches the crawler classes as they are imported, so an existing crawler produces traces and logs without any change to its own code. Closes #2955 ## What it does **Automatic instrumentation.** `CrawleeInstrumentation` plugs into the OpenTelemetry SDK like any other instrumentation and patches these methods: | Module | Method | Span name | Kind | | --- | --- | --- | --- | | `@crawlee/basic` | `BasicCrawler.run` | `crawlee.crawler.run` | internal | | `@crawlee/basic` | `BasicCrawler.handleRequest` | `crawlee.crawler.handleRequest` | internal | | `@crawlee/basic` | `BasicCrawler.runRequestHandler` | `crawlee.crawler.runRequestHandler` | internal | | `@crawlee/basic` | `BasicCrawler.requestFunctionErrorHandler` | `crawlee.crawler.requestFunctionErrorHandler` | internal | | `@crawlee/basic` | `BasicCrawler.handleFailedRequestHandler` | `crawlee.crawler.handleFailedRequestHandler` | internal | | `@crawlee/http` | `HttpCrawler.makeHttpRequest` | `crawlee.http.makeHttpRequest` | client | | `@crawlee/browser` | `BrowserCrawler.navigate` | `crawlee.browser.navigate` | client | Every span carries `code.function.name`, the `run` span carries `crawlee.crawler.type`, and the methods that receive a crawling context also carry `url.full`, `http.request.method`, `crawlee.request.id`, and `crawlee.request.retry_count`. `url.full` and `http.request.method` are the stable semantic conventions, so the traces stay comparable with other instrumented HTTP clients; Crawlee data with no convention keeps the `crawlee.` prefix. A method that is missing from the installed Crawlee version is skipped with a warning instead of breaking the module load. **Log forwarding.** Patches the logging methods Crawlee derives in `BaseCrawleeLogger`, so `log.info()`, `log.exception()`, and the rest are emitted as OpenTelemetry log records with the Crawlee level mapped onto the OpenTelemetry severity. Because every v4 logger derives from that base, a Winston, Pino, or hand-written adapter is forwarded just like the default one. The structured `data` of a log call becomes the record attributes, and an `Error` in it becomes `exception.type`, `exception.message`, and `exception.stacktrace`. **Manual spans.** `wrapWithSpan()` wraps a request handler, a hook, or any other function. The span name and options are either static or computed from the call arguments. A synchronous function stays synchronous. **Custom instrumentation.** `customInstrumentation` takes module, class, and method triples, so a class the package does not know about can be patched the same way. `requestHandlingInstrumentation` and `logInstrumentation` switch the two built-in groups off. Errors set `SpanStatusCode.ERROR` and record an exception event. Success is left `UNSET`, following the OpenTelemetry guidance that an instrumentation should not claim a span succeeded. ## Setup Automatic instrumentation needs the OpenTelemetry module hook installed before anything imports Crawlee, and ESM imports are hoisted, so the hook and the SDK setup have to sit in preloaded files rather than at the top of the crawler: ```js // otel-hook.js import { register } from 'node:module'; import { pathToFileURL } from 'node:url'; register('@opentelemetry/instrumentation/hook.mjs', pathToFileURL('./')); ``` ```bash node --import ./otel-hook.js --import ./otel-setup.js ./main.js ``` `otel-setup.js` builds the `NodeSDK` with `CrawleeInstrumentation` and owns the shutdown flush. The guide has both files in full. The example in this repo runs with: ```bash pnpm exec tsx \ --import ./docs/guides/trace_and_monitor_register_hook.ts \ --import ./docs/guides/trace_and_monitor_setup.ts \ ./docs/guides/trace_and_monitor_basic.ts ``` Verified end to end against Jaeger over OTLP/gRPC. One thing worth knowing: Jaeger is a tracing backend and rejects OTLP logs, so the forwarded logs need a collector or a backend that ingests them. The guide calls that out. ## Docs New guide at `docs/guides/trace-and-monitor-crawlers.mdx`, with runnable snippets for the setup, `wrapWithSpan`, and custom instrumentation, the configuration reference, and the full list of automatically instrumented methods and attributes. ## Dependencies `@opentelemetry/instrumentation` and `@opentelemetry/semantic-conventions` are dependencies. `@opentelemetry/api` and `@opentelemetry/api-logs` are peer dependencies, so the package shares the host application's API instance. --------- Co-authored-by: Josef Procházka <jos.prochazka@post.cz> Co-authored-by: Josef Prochazka <josef.prochazka@apify.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Jan Buchar <jan@buchar.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )