Dns only option for disabling http probe

.devcontainer/.bash_aliases

@@ -3,3 +3,9 @@ alias gs="git status"
 alias gd="git diff"
 alias gds="git diff --staged"
 alias tma="tmux attach"
+
+mkdir -p ~/.config/{btop,htop}
+[[ -f ~/.config/btop/btop.conf ]] || ln -sf /commandhistory/btop.conf ~/.config/btop/
+[[ -f ~/.config/htop/htoprc    ]] || ln -sf /commandhistory/htoprc    ~/.config/htop/
+
+[[ -f /commandhistory/.bashrc-pgit-bot ]] && source /commandhistory/.bashrc-pgit-bot

.devcontainer/Dockerfile

@@ -2,7 +2,7 @@
 # The dependencies of this project have been moved into another Dockerfile that can be built
 # separately. This image can be pushed to the HUB, speeding up codespace creation considerably.
 #
-FROM docker.io/psedoc/anyhttp:0.31
+FROM docker.io/psedoc/anyhttp:0.32
 
 # ==================================================================================================
 

CLAUDE.md

@@ -9,10 +9,11 @@ its own: anyhttp for HTTP and c-ares-async for DNS.
 
 These hold for all code in this repository, including tests and experiments.
 
-- **One request per host, and nothing more.** c-ares-async's lists of hosts are fine as input here,
-  which is what makes this rule matter: they have 1000 mixed sites, some adult or spam-looking. So
-  nothing is followed: no links, no redirects and no Alt-Svc. Another route is for a connection that
-  failed: once a request has gone out, that was it for the host, however it ended.
+- **One request per hostname, but if explicitly enabled, also one request per IP, and nothing
+  more.** c-ares-async's lists of hosts are fine as input here, which is what makes this rule
+  matter: they have 1000 mixed sites, some adult or spam-looking. So nothing is followed: no links,
+  no redirects and no Alt-Svc. Another route is for a connection that failed: once a request has
+  gone out, that was it for the hostname, or for the IP if enabled, however it ended.
 - **Tests connect to local servers on loopback only.**
 - **If the program ever crawls, it refuses to run without an explicit host allowlist,** with
   robots.txt and per-host rate limits built in from the start, not added later.
@@ -45,7 +46,7 @@ any I/O, and [prober.cpp](src/prober.cpp) does the rest. The tests in [test/](te
 ## Style
 
 As in anyhttp: clang-format with 3-space indent and 100 columns, members `foo_`, functions
-`snake_case`, types `CamelCase`. Commit subjects are prefixed `fix:`, `refactor:`, `docs:`, `chore:`
-or `test:`.
+`snake_case`, types `CamelCase`. Commit subjects are prefixed `feat:`, `fix:`, `docs:`, `style:`,
+`refactor:`, `perf:`, `test:`, `build:`, `ci:`, `chore:` or `revert:`.
 
 Commit or push only when asked. Push over SSH to git@github.com; Peter opens the pull requests.

README.md

@@ -38,6 +38,8 @@ look like this, on stderr:
 | does not exist                                | not found, not connected to                 |
 | gets no answer from DNS, or has no address    | lookup failed, so there is no telling, not connected to |
 | has no record that offers `h3`, with `--h3`   | without h3, not connected to                |
+| could be connected to, with `--dns-only`      | resolved, not connected to                  |
+| could be connected to, with `--each-address`  | answered if all of its addresses did, otherwise failed |
 
 Aliases are followed, a record has to be compatible, and the lowest priority goes first. Within a
 record, HTTP/3 goes first, and then TCP on the same port: HTTP/2 if the record has `h2`, otherwise
@@ -49,6 +51,38 @@ target, and they are treated as above, so that one can still end up out of reach
 when HTTP/3 can not be connected to. The others are left out after their HTTPS lookup, and printed
 only with `--all`. The counts at the end then add how many there were, as `5 without h3`.
 
+**With `--dns-only`,** it stops at the point where it would connect, so that no host is connected to
+or requested. A host that would have been goes by `resolved` and comes with its records and all of
+its addresses, in the order they would be tried:
+
+```
+example.com: resolved
+  record: HTTPS 1 . alpn=h3,h2
+  address: 2001:db8::1
+  address: 192.0.2.1
+```
+
+All other hosts get the outcome they always get, which leaves `answered` and `failed` at 0. The addresses
+are only looked up for a host that would have been connected to, so an `out of reach` one has none.
+The counts at the end then add how many there were, as `5 resolved`, and `--h3` works as above.
+
+**With `--each-address`,** every address of a host gets a request of its own instead of only the first
+that can be connected to, one after the other, with a line per address for the status line of the
+response or what went wrong:
+
+```
+example.com: answered
+  record: HTTPS 1 . alpn=h3,h2
+  address: 2001:db8::1 200 OK
+  address: 192.0.2.1 200 OK
+```
+
+All other hosts get the outcome they always get. A host counts as `answered` only if all of its
+addresses did, whatever the status, and otherwise as `failed`. This is the one exception to one request
+per host, and it can't go with `--dns-only`. `--timeout-ms` then is per address on each route. Beyond
+that, the next paragraph holds for each address, and nothing is followed here either. HTTP/2 and HTTP/3
+have no reason phrase, so there the `OK` is what that status is called.
+
 **One request means one.** The next route is for when connecting failed. Once a request has gone out,
 the host has been asked, so that a timeout or a response cut short ends it there, as a failure.
 

src/main.cpp

@@ -160,6 +160,12 @@ po::options_description describe_options(Arguments& arguments)
         "how long one request may take");
    opts("h3", po::bool_switch(&config.h3_only),
         "connect only to the hosts with an HTTPS record that offers HTTP/3");
+   opts("dns-only", po::bool_switch(&config.dns_only),
+        "connect to nothing: stop after the lookups, and print the records and the addresses of "
+        "each host");
+   opts("each-address", po::bool_switch(&config.each_address),
+        "send a request to each address of a host and print the status line of each, instead of "
+        "one request per host");
    opts("all", po::bool_switch(&arguments.all),
         "also print the hosts that do not exist, and those that --h3 leaves out");
    opts("color", po::bool_switch(&arguments.color),

src/output.cpp

@@ -43,6 +43,19 @@ std::string address_text(std::string_view address, bool color)
    return colored(address, v6 ? color_of::v6 : color_of::v4, color);
 }
 
+//
+// What came of the request to one address: the status line, as far as it goes without the version,
+// which the routes say more about, or what went wrong.
+//
+std::string reply_text(const AddressReply& reply, bool color)
+{
+   if (!reply.error.empty())
+      return std::format("{}: {}", colored("failed", color_of::bad, color), reply.error);
+   return reply.reason.empty()
+             ? status_text(reply.status, color)
+             : std::format("{} {}", status_text(reply.status, color), reply.reason);
+}
+
 //
 // The value of an address hint: the addresses have a colour, the commas between them do not.
 //
@@ -95,15 +108,21 @@ std::string format_result(const Result& result, bool color)
    switch (result.outcome)
    {
    case Outcome::answered:
-      text = std::format("{}: {} [{}] at {}, {} bytes in {}\n", host,
-                         status_text(result.status, color), result.route,
-                         address_text(result.address, color), result.body_size, result.duration);
+      text =
+         !result.replies.empty()
+            ? std::format("{}: {}\n", host, colored("answered", color_of::good, color))
+            : std::format("{}: {} [{}] at {}, {} bytes in {}\n", host,
+                          status_text(result.status, color), result.route,
+                          address_text(result.address, color), result.body_size, result.duration);
       break;
    case Outcome::failed:
-      text = std::format(
-         "{}: {}{}: {}\n", host, colored("failed", color_of::bad, color),
-         result.address.empty() ? "" : std::format(" at {}", address_text(result.address, color)),
-         result.error);
+      text = !result.replies.empty()
+                ? std::format("{}: {}\n", host, colored("failed", color_of::bad, color))
+                : std::format("{}: {}{}: {}\n", host, colored("failed", color_of::bad, color),
+                              result.address.empty()
+                                 ? ""
+                                 : std::format(" at {}", address_text(result.address, color)),
+                              result.error);
       break;
    case Outcome::out_of_reach:
       text = std::format("{}: {}\n", host, colored("out of reach", color_of::moved, color));
@@ -117,10 +136,18 @@ std::string format_result(const Result& result, bool color)
    case Outcome::no_h3:
       text = std::format("{}: no h3 record\n", host);
       break;
+   case Outcome::resolved:
+      text = std::format("{}: {}\n", host, colored("resolved", color_of::good, color));
+      break;
    }
 
    for (const auto& record : result.records)
       text += std::format("  record: {}\n", record_text(record, color));
+   for (const auto& address : result.addresses)
+      text += std::format("  address: {}\n", address_text(address, color));
+   for (const auto& reply : result.replies)
+      text += std::format("  address: {} {}\n", address_text(reply.address, color),
+                          reply_text(reply, color));
    for (const auto& note : result.notes)
       text += colored(std::format("  note: {}", note), color_of::note, color) + "\n";
    for (const auto& [name, value] : result.fields)
@@ -137,11 +164,14 @@ std::string format_summary(Counts counts, bool color)
       return counts[outcome] ? colored(line, code, color) : line;
    };
    return std::format(
-      "{}, {}, {}, {} not found, {}{}", count(Outcome::answered, "answered", color_of::good),
+      "{}, {}, {}, {} not found, {}{}{}", count(Outcome::answered, "answered", color_of::good),
       count(Outcome::failed, "failed", color_of::bad),
       count(Outcome::out_of_reach, "out of reach", color_of::moved), counts[Outcome::not_found],
       count(Outcome::lookup_failed, "lookups failed", color_of::bad),
-      counts[Outcome::no_h3] ? std::format(", {} without h3", counts[Outcome::no_h3]) : "");
+      counts[Outcome::no_h3] ? std::format(", {} without h3", counts[Outcome::no_h3]) : "",
+      counts[Outcome::resolved]
+         ? std::format(", {}", count(Outcome::resolved, "resolved", color_of::good))
+         : "");
 }
 
 // =================================================================================================

src/prober.cpp

@@ -188,6 +188,8 @@ Prober::Prober(capy::any_executor executor, Config config, Observer observer)
       throw std::invalid_argument("nothing to do: no host names given");
    if (config_.concurrency == 0)
       throw std::invalid_argument("a concurrency of 0 would never get anywhere");
+   if (config_.dns_only && config_.each_address)
+      throw std::invalid_argument("--dns-only connects to nothing, which rules out --each-address");
 
    std::set<std::string> seen;
    for (const auto& host : config_.hosts)
@@ -393,6 +395,27 @@ capy::task<Result> Prober::probe(const std::string& host)
    //
    prefer_ipv6(addresses);
 
+   if (config_.dns_only)
+   {
+      result.outcome = Result::Outcome::resolved;
+      for (const auto& address : addresses)
+         result.addresses.push_back(address.address().to_string());
+      co_return result;
+   }
+
+   if (config_.each_address)
+   {
+      result.outcome = Result::Outcome::answered;
+      for (const auto& address : addresses)
+      {
+         auto reply = co_await ask_address(plan.routes, host, address, result.notes);
+         if (!reply.error.empty())
+            result.outcome = Result::Outcome::failed;
+         result.replies.push_back(std::move(reply));
+      }
+      co_return result;
+   }
+
    result.outcome = Result::Outcome::failed;
    for (const auto& route : plan.routes)
    {
@@ -432,6 +455,40 @@ capy::task<Result> Prober::probe(const std::string& host)
    co_return result;
 }
 
+//
+// One request to one address: the next route is for a connection that did not come about, and once
+// a request has gone out, that was it for this address, however it ended.
+//
+capy::task<AddressReply> Prober::ask_address(const std::vector<Route>& routes,
+                                             const std::string& host,
+                                             const corosio::endpoint& address,
+                                             std::vector<std::string>& notes)
+{
+   AddressReply reply{.address = address.address().to_string()};
+   for (const auto& route : routes)
+   {
+      config_.logger->debug("{}: connecting to {} by {}, for {}", host, reply.address,
+                            route.to_string(), route.source);
+      const auto deadline = std::chrono::steady_clock::now() + config_.timeout;
+      auto [ec, response] =
+         co_await attempt(route, host, corosio::endpoint(address, route.port), deadline);
+      if (!ec)
+      {
+         reply.status = static_cast<int>(response.message.result_int());
+         reply.reason = response.message.reason();
+         reply.error.clear();
+         co_return reply;
+      }
+
+      reply.error = std::format("[{}] {}", route.to_string(), response.error);
+      notes.push_back(std::format("{} at {}", reply.error, reply.address));
+      config_.logger->debug("{}: {} at {}", host, reply.error, reply.address);
+      if (response.request_sent)
+         co_return reply;
+   }
+   co_return reply;
+}
+
 //
 // Connects as the route says, to this one address so that there is no doubt about which it was, and
 // asks for the host's root, by the deadline. After that it ends the connection in an orderly way,

src/prober.hpp

@@ -70,6 +70,14 @@ struct Config
    /// treated as usual, which includes going over TCP if HTTP/3 can not be connected to.
    bool h3_only = false;
 
+   /// Stop after the lookups, with the addresses that it would have gone on to connect to, and
+   /// connect to nothing. See Result::Outcome::resolved.
+   bool dns_only = false;
+
+   /// Send a request to each address of a host, one at a time, rather than to the first that can be
+   /// connected to. Config::timeout then is per address on each route. See Result::replies.
+   bool each_address = false;
+
    /// Where the messages of this program and of c-ares go. anyhttp can't be told: it always uses
    /// the default logger of spdlog, so a logger other than that one can have a level of its own.
    std::shared_ptr<spdlog::logger> logger = spdlog::default_logger();
@@ -89,17 +97,27 @@ std::vector<std::string> read_hosts(std::istream& list, std::vector<std::string>
 /// Moves the IPv6 addresses to the front, each family keeping the order it was in.
 void prefer_ipv6(std::vector<corosio::endpoint>& addresses);
 
+/// What came of the request to one address of a host, see Config::each_address.
+struct AddressReply
+{
+   std::string address;
+   int status = 0; ///< without \c error
+   std::string reason; ///< without \c error: "OK", which HTTP/2 and HTTP/3 only know by the status
+   std::string error; ///< what went wrong on the last route, if no response could be had
+};
+
 /// What came of one host.
 struct Result
 {
    enum class Outcome
    {
-      answered, ///< a response arrived, whatever its status
-      failed, ///< it was connected to, and no response could be had
+      answered, ///< a response arrived, whatever its status, with Config::each_address from all
+      failed, ///< it was connected to, and no response could be had, or not from all addresses
       out_of_reach, ///< not connected to, its records say so or lead elsewhere, see plan_routes()
       not_found, ///< not connected to, the name does not exist
       lookup_failed, ///< not connected to, DNS gave no answer, so there is no telling
       no_h3, ///< not connected to, it has no record that offers HTTP/3, see Config::h3_only
+      resolved, ///< not connected to, but it could have been tried, see Config::dns_only
    };
 
    std::string host;
@@ -115,7 +133,9 @@ struct Result
 
    /// The address that answered or, with \c failed, the last one contacted. Empty without one.
    std::string address;
+   std::vector<std::string> addresses; ///< with \c resolved: all of them, in the order of trying
    std::string error; ///< with \c failed: what went wrong on the last route
+   std::vector<AddressReply> replies; ///< with Config::each_address, instead of the above
    std::vector<std::string> notes; ///< records that could not be used, routes that did not work
 };
 
@@ -125,7 +145,9 @@ struct Result
  * trying its addresses one at a time on each route, IPv6 first. A response is what comes back,
  * whatever its status: a redirect is reported and not followed, nothing in the body is looked at,
  * and nothing else on the host is requested. The next route is for a connection that did not come
- * about: once a request has gone out, that was it for this host, however it ended.
+ * about: once a request has gone out, that was it for this host, however it ended. With
+ * Config::dns_only, it is the lookups and nothing else. With Config::each_address, all of this goes
+ * for each address of the host instead of for the host: one request per address, in order.
  *
  * Hosts are worked on Config::concurrency at a time, so results arrive as they are ready. All of
  * this is on the one executor, which is also what calls the observer.
@@ -156,6 +178,9 @@ class Prober
    capy::task<capy::results<>> work();
    capy::task<ServiceRecords> resolve_service(const std::string& host);
    capy::task<Result> probe(const std::string& host);
+   capy::task<AddressReply> ask_address(const std::vector<Route>& routes, const std::string& host,
+                                        const corosio::endpoint& address,
+                                        std::vector<std::string>& notes);
    capy::task<capy::results<Response>> attempt(const Route& route, const std::string& host,
                                                const corosio::endpoint& endpoint,
                                                std::chrono::steady_clock::time_point deadline);

test/CMakeLists.txt

@@ -89,6 +89,22 @@ add_test(NAME http_probe_refuses_tls_options_without_dot
 set_tests_properties(http_probe_refuses_tls_options_without_dot PROPERTIES
     PASS_REGULAR_EXPRESSION "only apply with --dot")
 
+add_test(NAME http_probe_takes_dns_only
+         COMMAND http-probe --dns-only --dns-server 127.0.0.1:1 localhost)
+set_tests_properties(http_probe_takes_dns_only PROPERTIES
+    PASS_REGULAR_EXPRESSION "1 lookups failed")
+
+add_test(NAME http_probe_refuses_each_address_with_dns_only
+         COMMAND http-probe --each-address --dns-only localhost)
+set_tests_properties(http_probe_refuses_each_address_with_dns_only PROPERTIES
+    PASS_REGULAR_EXPRESSION "rules out --each-address")
+
+# An empty list gets as far as having no hosts, and goes nowhere.
+add_test(NAME http_probe_takes_each_address_with_a_list
+         COMMAND http-probe --each-address --file /dev/null)
+set_tests_properties(http_probe_takes_each_address_with_a_list PROPERTIES
+    PASS_REGULAR_EXPRESSION "no host names given")
+
 #
 # The levels of logging, as far as they show without a connection: the lookups fail on a port of
 # loopback that nothing listens on, so this needs nothing but the program.