Corosio only

.cspell.json

@@ -1,12 +1,15 @@
 {
     "words": [
+        "alpn",
         "anyhttp",
         "ASIO",
+        "capy",
         "cares",
         "corosio",
         "Eisenlohr",
         "psedoc",
-        "spdlog"
+        "spdlog",
+        "SVCB"
     ],
     "ignorePaths": [
         ".devcontainer/*",

.devcontainer/devcontainer.json

@@ -38,7 +38,7 @@
 				"testMate.cpp.test.parallelExecutionLimit": 20,
 				"testMate.cpp.test.advancedExecutables": [
 					{
-						"pattern": "${command:cmake.buildDirectory}/test/test_*",
+						"pattern": "${command:cmake.buildDirectory}/test/*_tests",
 						"cwd": "${workspaceDirectory}",
 						"runTask": {
 							"before": [
@@ -75,8 +75,6 @@
 
 	"mounts": [
 		"source=cpp-devcontainer-bashhistory,target=/commandhistory,type=volume",
-		"source=${localEnv:HOME}/src/anyhttp,target=/workspaces/anyhttp,type=bind,consistency=cached",
-		"source=${localEnv:HOME}/src/c-ares-async,target=/workspaces/c-ares-async,type=bind,consistency=cached",
 		"source=${localEnv:HOME}/.claude,target=/home/vscode/.claude,type=bind,consistency=cached",
 		"source=${localEnv:HOME}/.claude.json,target=/home/vscode/.claude.json,type=bind,consistency=cached"
 	],

CLAUDE.md

@@ -1,29 +1,42 @@
-# Working on the crawler
+# Working on http-probe
 
-A web crawler on two libraries of the same author, each in a repository of its own: anyhttp for HTTP
-and c-ares-async for DNS. [README.md](README.md) has the presets and the build commands.
+It requests `https://<host>/` once per host and reports what came back, connecting the way the
+host's HTTPS records say. It is built on two libraries of the same author, each in a repository of
+its own: anyhttp for HTTP and c-ares-async for DNS.
+[README.md](README.md) has the build commands.
 
 ## Safety rules
 
 These hold for all code in this repository, including tests and experiments.
 
-- **Never open, fetch or connect to anything in c-ares-async's `url_list.txt`, and never use it as
-  input.** It is for DNS lookups only: 1000 mixed sites, some adult or spam-looking. The crawler is
-  the first program here that could turn those names into connections.
-- **Tests crawl local servers on loopback only.**
-- **Once the program crawls, it refuses to run without an explicit host allowlist.** robots.txt and
-  per-host rate limits are built in from the start, not added later.
+- **One request per host, and nothing more.** c-ares-async's lists of hosts are fine as input here,
+  which is what makes this rule matter: they have 1000 mixed sites, some adult or spam-looking. So
+  nothing is followed: no links, no redirects and no Alt-Svc. Another route is for a connection that
+  failed: once a request has gone out, that was it for the host, however it ended.
+- **Tests connect to local servers on loopback only.**
+- **If the program ever crawls, it refuses to run without an explicit host allowlist,** with
+  robots.txt and per-host rate limits built in from the start, not added later.
 - **Run tests and programs under `timeout -s KILL`**, 5 seconds per test. Never let a suite sit on a
   hang.
 
+## The code
+
+[routes.cpp](src/routes.cpp) says what the HTTPS records of a name mean for how to connect, without
+any I/O, and [prober.cpp](src/prober.cpp) does the rest. The tests in [test/](test) run against
+[Site](test/site.hpp), a real anyhttp server and a DNS server with a zone of its own, on loopback.
+
 ## The libraries
 
 - Changes to a library belong in that library's repository and session, with a test in its suite.
-  Do not work around a library bug in crawler code: get it fixed there, then move the pin.
-- Moving a pin in [CMakeLists.txt](CMakeLists.txt) is a commit of its own. CI builds from the pins
-  only, so a crawler change that needs a library change has to wait for that change to be pushed.
-- The `dev` preset builds the sibling checkouts as they are, on whatever branch they are on, and can
-  differ from the pins. Check with the `pinned` preset before committing.
+  Do not work around a library bug in this code: get it fixed there, then move the pin.
+- Ask for a library change with an issue in that library's repository, using `gh`. Peter reviews the
+  full text first, and the same goes for comments on issues and for closing them. The repositories
+  are public, so reproduce on loopback and name no hosts from the lists.
+- Moving a pin in [CMakeLists.txt](CMakeLists.txt) is a commit of its own. Everything builds from
+  the pins, so a change here that needs a library change has to wait for that change to be pushed.
+- The libraries come from GitHub, as for any other user, and there are no checkouts of them to build
+  against. Other agents working on them do not interfere with this repository: nothing they do
+  arrives here until a pin moves. To read a library at its pin, look in `build/_deps/`.
 - Everything in one process has to agree on the standard library, the TLS library and the I/O
   runtime. [CMakeLists.txt](CMakeLists.txt) decides these for both libraries, so change them there.
   Do not configure a tree by hand: `cc` and `c++` are GCC on this host, and a tree without clang
@@ -35,5 +48,4 @@ As in anyhttp: clang-format with 3-space indent and 100 columns, members `foo_`,
 `snake_case`, types `CamelCase`. Commit subjects are prefixed `fix:`, `refactor:`, `docs:`, `chore:`
 or `test:`.
 
-Commit or push only when asked. `gh`'s token is read-only, so push over SSH to git@github.com; Peter
-opens the pull requests.
+Commit or push only when asked. Push over SSH to git@github.com; Peter opens the pull requests.

CMakeLists.txt

@@ -1,5 +1,5 @@
 cmake_minimum_required(VERSION 3.30)
-project(crawler VERSION 0.0.1 DESCRIPTION "A web crawler on anyhttp and c-ares-async" LANGUAGES C CXX)
+project(http_probe VERSION 0.0.1 DESCRIPTION "Requests hosts the way their HTTPS records say, on anyhttp and c-ares-async" LANGUAGES C CXX)
 
 set(CMAKE_CXX_STANDARD 23)
 set(CMAKE_CXX_STANDARD_REQUIRED On)
@@ -20,10 +20,16 @@ if (CMAKE_CXX_COMPILER_ID MATCHES "Clang")
     add_compile_options($<$<COMPILE_LANGUAGE:CXX>:-stdlib=libc++>)
     add_link_options(-stdlib=libc++)
     list(PREPEND CMAKE_PREFIX_PATH /opt/libc++)
+    # As in anyhttp: the shared ones share their names with the libstdc++ ones in /usr/local/lib.
+    set(Boost_USE_STATIC_LIBS ON)
 endif()
 
-# I/O runtime: ASIO or COROSIO. COROSIO would have to declare the capy and corosio pins here, too.
-set(ANYHTTP_API ASIO)
+#
+# I/O runtime: Capy and Corosio, and only those. The libraries take either, which is their business:
+# this is a program, and it gets to decide. These shadow the cache entries of the libraries, which
+# go on saying ASIO in CMakeCache.txt.
+#
+set(ANYHTTP_API COROSIO)
 set(C_ARES_ASYNC_API ${ANYHTTP_API})
 
 #
@@ -36,28 +42,55 @@ set(C_ARES_ASYNC_TLS ${TLS_LIBRARY})
 # --------------------------------------------------------------------------------------------------
 
 #
-# Pinned to a commit, and moving a pin is a commit of its own. To build against a checkout instead
-# there is the 'dev' preset, which sets FETCHCONTENT_SOURCE_DIR_*. SYSTEM, so that the warnings of
-# this project do not apply to the headers of the libraries.
-#
-# anyhttp: the head of branch 'packaging', move this to master once that is merged
+# Pinned to a commit, and moving a pin is a commit of its own. SYSTEM, so that the warnings of this
+# project do not apply to the headers of the libraries.
 #
 include(FetchContent)
 FetchContent_Declare(anyhttp
     GIT_REPOSITORY https://github.com/pgit/anyhttp.git
-    GIT_TAG af39b5c71490978d545c1b47ee8e149f9a7082c9
+    GIT_TAG cc3faee4fb351a56bbca48db7cfcac7585e59e8b
     SYSTEM)
 FetchContent_Declare(c_ares_async
     GIT_REPOSITORY https://github.com/pgit/c-ares-async.git
-    GIT_TAG 2fb55f687f210d3da816e24aab9cefb61d6b4673
+    GIT_TAG 6d46c6c8e65231f23a250bcec5dd00b3a1d9ba34
+    SYSTEM)
+
+#
+# Both libraries pin these as well, to the same commits for now. The first declaration wins, so these
+# are the ones both of them get, and here they are SYSTEM, too.
+#
+FetchContent_Declare(capy
+    GIT_REPOSITORY https://github.com/cppalliance/capy.git
+    GIT_TAG 24c54af2cf967ccbf8d3809af4bcd9c8f61f553b
+    EXCLUDE_FROM_ALL
     SYSTEM)
+FetchContent_Declare(corosio
+    GIT_REPOSITORY https://github.com/cppalliance/corosio.git
+    GIT_TAG 41d0de18b6e4fa1b397c38a866f2daf52c6c36b5
+    EXCLUDE_FROM_ALL
+    SYSTEM)
+
 FetchContent_MakeAvailable(anyhttp c_ares_async)
 
+# Only for the command line. Neither library needs this component, so neither asks for it here.
+find_package(Boost 1.88 REQUIRED COMPONENTS program_options)
+
 # --------------------------------------------------------------------------------------------------
 
-add_executable(crawler src/main.cpp)
-target_link_libraries(crawler PRIVATE anyhttp::anyhttp c_ares_async::c_ares_async)
-target_compile_options(crawler PRIVATE -Wall -Wextra -Werror)
+#
+# The program in a library of its own, for the tests to link. The warnings are not for the sources
+# from c-ares-async that the tests borrow, see test/CMakeLists.txt.
+#
+set(http_probe_warnings -Wall -Wextra -Werror -Wno-missing-designated-field-initializers)
+
+add_library(http_probe_core STATIC src/output.cpp src/prober.cpp src/routes.cpp)
+target_include_directories(http_probe_core PUBLIC src)
+target_link_libraries(http_probe_core PUBLIC anyhttp::anyhttp c_ares_async::c_ares_async)
+target_compile_options(http_probe_core PRIVATE ${http_probe_warnings})
+
+add_executable(http-probe src/main.cpp)
+target_link_libraries(http-probe PRIVATE http_probe_core Boost::program_options)
+target_compile_options(http-probe PRIVATE ${http_probe_warnings})
 
 enable_testing()
-add_test(NAME crawler_starts COMMAND crawler)
+add_subdirectory(test)

CMakePresets.json

@@ -18,30 +18,15 @@
                 "CMAKE_EXPORT_COMPILE_COMMANDS": "ON"
             }
         },
-        {
-            "name": "dev",
-            "displayName": "Debug, against sibling checkouts",
-            "description": "What the crawler is developed with: the two libraries as they are in their checkouts next to this one, whatever the pins say. Keep this tree in build/, clangd reads its compile_commands.json",
-            "inherits": "base",
-            "binaryDir": "${sourceDir}/build",
-            "cacheVariables": {
-                "FETCHCONTENT_SOURCE_DIR_ANYHTTP": "${sourceDir}/../anyhttp",
-                "FETCHCONTENT_SOURCE_DIR_C_ARES_ASYNC": "${sourceDir}/../c-ares-async"
-            }
-        },
         {
             "name": "pinned",
             "displayName": "Debug, from the pins",
-            "description": "The libraries at the commits CMakeLists.txt pins, fetched from GitHub: what CI builds",
+            "description": "The libraries at the commits CMakeLists.txt pins, fetched from GitHub. Keep this tree in build/, clangd reads its compile_commands.json",
             "inherits": "base",
-            "binaryDir": "${sourceDir}/build-pinned"
+            "binaryDir": "${sourceDir}/build"
         }
     ],
     "buildPresets": [
-        {
-            "name": "dev",
-            "configurePreset": "dev"
-        },
         {
             "name": "pinned",
             "configurePreset": "pinned"
@@ -60,11 +45,6 @@
                 "noTestsAction": "error"
             }
         },
-        {
-            "name": "dev",
-            "configurePreset": "dev",
-            "inherits": "base"
-        },
         {
             "name": "pinned",
             "configurePreset": "pinned",

README.md

@@ -1,30 +1,102 @@
-# crawler
+# http-probe
 
-A simple web crawler on [anyhttp](https://github.com/pgit/anyhttp) (HTTP/1.1, HTTP/2 and HTTP/3) and
+Requests hosts the way their HTTPS records say, on [anyhttp](https://github.com/pgit/anyhttp) (HTTP) and
 [c-ares-async](https://github.com/pgit/c-ares-async) (DNS).
 
-**It does not crawl anything yet.** The `crawler` program sets up both libraries on one event loop
-and exits, which shows that they build and link into one process.
-
 > [!WARNING]
 > This repository is purely experimental. It is not meant for production use, and anything in it
 > can change or break at any time, without notice.
 
 
+## What it does
+
+```sh
+http-probe --dns-server 1.1.1.1 cloudflare.com
+http-probe --dns-server 1.1.1.1 --file hostnames.txt
+http-probe --dot --dns-server one.one.one.one cloudflare.com
+```
+
+It looks up the HTTPS records of each host in DNS ([RFC 9460](https://www.rfc-editor.org/rfc/rfc9460)),
+and requests `https://<host>/` once, over TLS, in the way that the records say. It prints the status,
+the route, the IP address it connected to, the time it took and the header fields of the response,
+along with the host's records. A redirect is reported and not followed, and nothing else on the host
+is requested.
+
+There is output for every host but those that do not exist, which `--all` adds. The counts at the end
+look like this, on stderr:
+
+```
+190 answered, 1 failed, 3 out of reach, 12 not found, 2 lookups failed
+```
+
+| The host                                      | Outcome                                     |
+|-----------------------------------------------|---------------------------------------------|
+| has a record on its own name                  | answered or failed, by the protocols and the port of the record |
+| has no record, or none that is compatible     | answered or failed, over TCP on port 443, HTTP/2 or HTTP/1.1 as the server likes |
+| has records for other targets only, or an alias to another name | out of reach, not connected to: anyhttp can't connect to one name and verify another |
+| has a record that says the service is not available | out of reach, not connected to        |
+| does not exist                                | not found, not connected to                 |
+| gets no answer from DNS, or has no address    | lookup failed, so there is no telling, not connected to |
+| has no record that offers `h3`, with `--h3`   | without h3, not connected to                |
+
+Aliases are followed, a record has to be compatible, and the lowest priority goes first. Within a
+record, HTTP/3 goes first, and then TCP on the same port: HTTP/2 if the record has `h2`, otherwise
+HTTP/1.1, which is in every record that does not say `no-default-alpn`. The route in the output is
+what the server went for, which may be HTTP/1.1 where HTTP/2 was offered.
+
+**With `--h3`,** only hosts with a compatible record that has `h3` in its `alpn` go ahead, on whichever
+target, and they are treated as above, so that one can still end up out of reach, or going over TCP
+when HTTP/3 can not be connected to. The others are left out after their HTTPS lookup, and printed
+only with `--all`. The counts at the end then add how many there were, as `5 without h3`.
+
+**One request means one.** The next route is for when connecting failed. Once a request has gone out,
+the host has been asked, so that a timeout or a response cut short ends it there, as a failure.
+
+The address comes from c-ares, so `--dns-server` applies to it, and `/etc/hosts` as well. The address
+hints in a record do not. Addresses are tried one at a time, IPv6 first and otherwise in the order
+c-ares sorts them, within the timeout of the route, until one of them can be connected to. So an IPv6
+address that does not answer at all takes the whole timeout of the route, and IPv4 does not get a turn.
+A failure says which address it was on. Certificates are always verified, against the host's name.
+
+**To see what is going on,** `-v` logs to stderr: the requests and responses from anyhttp, the queries
+from c-ares and the steps in between. `-vv` adds all of QUIC. Hosts run at the same time mix in the
+log, so `--concurrency 1` helps. Colours are for terminals only unless there is `--color`, except that
+anyhttp puts them into some of its log messages regardless.
+
+**Use `--dns-server`.** The resolver in the devcontainer answers with no data for HTTPS records, even
+where they exist, which looks just like no host having any. It takes addresses and host names, with a
+port behind either. A name is looked up in the clear with the servers of the system, whatever the
+queries proper go out as, and only its first address becomes a server.
+
+**`--dot` asks over DNS over TLS** ([RFC 7858](https://www.rfc-editor.org/rfc/rfc7858)), on port 853
+unless the server says otherwise, for the records and the addresses alike. The certificate of the
+server needs a name to be checked against. That is the host name in `--dns-server`, and with an
+address, or several names, `--tls-hostname` has to give it. `--tls-no-verify` skips the check, which
+is for testing and has nothing to do with the certificates of the hosts. A failed handshake makes the
+hosts lookup failures, and says why in the log. `--ca-file` is for the hosts as well: the certificate
+of the DNS server goes by the trust store of OpenSSL, which `SSL_CERT_FILE` can change.
+
+**A response goes to memory as a whole,** as anyhttp has no way of getting at it piece by piece with a
+GET yet. So only `--timeout-ms` limits how large one can get.
+
+To see it work without going anywhere, `demo_site` runs a site and a DNS server on loopback, and
+prints the command to request it with:
+
+```sh
+./build/test/demo_site
+```
+
+
 ## Build
 
 C++23 and CMake 3.30 or later. The devcontainer in [.devcontainer](.devcontainer) has the compilers
-and every dependency that the two libraries need. There are two presets:
-
-| Preset   | Tree            | Libraries come from                                             |
-|----------|-----------------|-----------------------------------------------------------------|
-| `dev`    | `build/`        | the checkouts next to this one, as they are                     |
-| `pinned` | `build-pinned/` | GitHub, at the commits pinned in [CMakeLists.txt](CMakeLists.txt); what CI builds |
+and every dependency that the two libraries need. The libraries themselves are fetched from GitHub,
+at the commits pinned in [CMakeLists.txt](CMakeLists.txt):
 
 ```sh
 cmake --preset pinned
 cmake --build --preset pinned
 ctest --preset pinned
 ```
 
-Both presets use clang, libc++, OpenSSL and the Asio runtime.
+The tree is `build/`, with clang, libc++, OpenSSL and Capy and Corosio as the runtime.