Repository navigation
Corosio only - #1
Merged
Merged
Conversation
The crawler looks up a host's HTTPS records with c-ares-async and follows RFC 9460 to get from them to an ordered list of routes: aliases, compatible records, priority, port prefixes, and treating "http" as "https" where there is a record, without going back to cleartext. routes.cpp does this without any I/O. Only what anyhttp can do becomes a route, which is HTTP/3 to the origin's own name, and HTTP/1.1 in the clear to addresses from c-ares where there is no record. Anything else is rejected with its reason, so most real sites can't be crawled until anyhttp has TLS over TCP and can connect to one name while verifying another. It refuses to run without an allowlist of exact host names, which every seed, link and redirect is checked against. It reads robots.txt first, stops on an origin whose robots.txt can't be read, and spaces its requests per host. The tests run on loopback only, against a real anyhttp server and the DNS server from c-ares-async's tests. Two of them stop the context early, as anyhttp does not close a session whose h3 connect was cancelled and it goes on sending for 30 seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README had it not crawling anything yet. It now says what happens for each kind of host, and why, given what anyhttp's client can do so far. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What is wanted is DNS and a single request: https://<host>/ for each host, with the status and the rest of what comes back. So the links, robots.txt, the pacing and the allowlist go, and so do http:// and port prefixes, as the port is always 443. The HTTPS records still decide how to connect. Redirects are reported and not followed, so the only names ever connected to are the ones given. Those can now come from a file, with --file, and c-ares-async's lists of hosts are fine for that, which CLAUDE.md now says. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It does not crawl any more, so the name says the opposite of what it does. http-probe for the program and the repository, like c-ares-async, http_probe in code. The User-Agent and the README now point at github.com/pgit/http-probe, which takes renaming the repository on GitHub to be right. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anyhttp's client has TLS on HTTP/3 only, so that is what there is to connect to. For a list like c-ares-async's 999 names, each host now has an outcome: answered, failed, HTTP/3 out of reach on a target that can't be used, without HTTP/3, or lookup failed, which is told apart as it says nothing about the host. Hosts without HTTP/3 are only printed with --verbose, and the counts come at the end. Hosts are worked on 16 at a time by default, with --concurrency, so that one that does not answer does not hold up the others. The resolver in the devcontainer answers with no data for HTTPS records, which looks just like no host having any, so a run that finds none says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Colours as in c-ares-async's example: host names in yellow, IPv4 in magenta and IPv6 in blue, for terminals only unless there is --color. On top of that, the status by its class and the outcomes that went wrong. The formatting is in output.cpp, where it can be tested without a terminal. -v logs to stderr through spdlog's default logger, which is where anyhttp logs already go: its requests and responses, the queries of c-ares, and the steps of the program in between. -vv adds all of QUIC. So --verbose no longer lists the hosts without HTTP/3, and --all does that now. The test site tries again when it finds a port taken. Its servers need the same number on TCP and UDP, which failed about once in 30 runs of the tests in parallel, with another test holding the other half. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
To have something to run the program on, as it is in c-ares-async's example/resolve. The tests don't use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As resolve in c-ares-async and server in anyhttp do. This adds --help, and with it --file=x, "--" and shortened option names. Giving an option with a single value twice is now an error, and hosts from --file come after those on the command line. The numbers are read as signed, because lexical_cast takes -1 for the largest value of an unsigned type. Boost is linked statically under clang, for the reason anyhttp does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anyhttp's client has TLS over TCP now, so a host no longer has to offer HTTP/3. Within a record, HTTP/3 goes first and TCP on the same port next, and a host without a usable record gets TCP on port 443. Names that do not exist, failed lookups, targets other than the host and records saying that the service is not available are still not connected to. This lifts the HTTP/3-only rule in CLAUDE.md, as decided by Peter. The next route is now only for a connection that failed. Before, a timeout during a request moved on to the next route as well, which could send the host a second request. Sessions now end with shutdown(), outside of the timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the dev preset and the mounts of the sibling checkouts, so that everything builds the way any other user gets it. Whatever other agents do in the libraries now arrives here with a pin move and not before. With a single preset left, its tree moves to build/, where clangd looks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g changed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onse no longer crashes the client Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The address came from anyhttp's own resolver, so --dns-server did not apply to it and nothing could tell which one had been contacted. Now c-ares looks it up and each attempt gets one address through client::Config::endpoints. The output has the address that answered, or the last one tried on a failure. Addresses are tried in the order c-ares sorts them, within the timeout of the route, until one can be connected to. Once a request has gone out, that ends it for the host, on this address as on any other route. A host without an address now counts as a failed lookup and is not connected to. Address hints get their colour per address, leaving the commas without. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c-ares goes by RFC 6724, which puts IPv4 first wherever the source address is not a global one, so IPv6 did not get a turn on such a machine. The addresses are still tried one at a time and not with Happy Eyeballs, so an IPv6 address that does not answer at all takes the whole timeout of the route. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The libraries take either runtime and need a layer for that. This is a program, so it decides on one and has no layer: no switches, one preset, and capy::task, capy::any_executor, corosio::endpoint and std::error_code directly instead of the aliases of anyhttp. CMakeLists.txt selects the runtime for both libraries and declares the pins of Capy and Corosio, the same commits as theirs. when_all() and corosio::timeout() replace the parallel group and when_either(). A failure now comes back as capy::results<Response>, so Response has no error code of its own. If a worker throws, when_all() stops the others, where the parallel group let them finish. Asio stays in the tests, for the DNS server of c-ares-async, which is on Asio whatever the runtime. Site polls its context from a coroutine, which keeps everything on one thread. The other use is anyhttp's write_eof(), which takes nothing but an Asio buffer. What a run_async() handler throws ends the process, so main() and the fixture hold on to it until run() returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abandon_quic_sessions was there because anyhttp went on sending for 30 seconds after a cancelled connect, so the context did not run dry. The pin has ef20614, which ends that connection, and the tests now end without stopping the context. So the README has no such wait to warn about, and main() prints the counts after run() rather than from a handler. Only tested on Corosio: the Asio code on http-probe still has the workaround. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…capy to the dictionary Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It filters hosts and not protocols: a host with such a record is treated as it would be without the option, so it can still go over TCP or be out of reach. The others are left out after their HTTPS lookup, printed only with --all and counted as "without h3". A failed lookup is not taken for having no HTTP/3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The same way as the server of anyhttp, except for looking at stdout as well as stderr, as --help goes to one and a usage error to the other. Without a terminal it is the 80 columns of Boost.ProgramOptions, in place of a fixed 100. Boost only wraps the table of options, so the paragraphs around it are wrapped here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Along with --tls-hostname and --tls-no-verify, and host names in --dns-server, as in the resolve example of c-ares-async. The transport is the library's. The connection for DNS over TLS stays open and keeps the event loop going, so the resolver now lives for the duration of run(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…edit fix Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.