Skip to content

Let a Logger report on a full queue, and close the client - #26

Merged
dmccoystephenson merged 2 commits into
mainfrom
feature/logger-reentrancy
Nov 5, 2026
Merged

dmccoystephenson merged 2 commits into
mainfrom
feature/logger-reentrancy

Conversation

@dmccoystephenson

@dmccoystephenson dmccoystephenson commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Summary

A Logger that calls back into the client no longer hangs or aborts the program.

  • Cause: Shared::log holds logMutex while the logger runs, so a call that reached log or forgetLogger again on the same thread locked a std::mutex that thread already owned.
  • report() from a Logger on a full queue: the message that report would raise is dropped without a mention, on either thread the Logger runs on. Shared now records which thread is running the logger, under the existing mutex. A re-entrancy check is used instead of a recursive mutex, as the issue requires, and no std::atomic is added, so the header cannot pick up a libatomic link requirement.
  • close() from a Logger: it now waits for nothing, whatever bound it is given, because the sending thread may be running the logger or waiting its turn to, so the wait might never end. The thread is detached under sharedMutex_ and what is queued is abandoned. log() clears the logger after it returns, since a running std::function cannot be destroyed. Beyond the hang the issue predicted, the old path also aborted: the stashed run below ended in terminate called without an active exception, consistent with the destructor finding thread_ still joinable while close() was blocked in forgetLogger.
  • Docs: the README promise table, the Logger typedef and close()'s comment state both behaviors.

The change is confined to Shared and TraceClient::close, so all transports and platforms share it. The curl argv, config quoting, process-spawn code, wire format and version are unchanged.

Closes #24

Deferred issues

Test plan

  • make clean && make test (g++, C++11): 65 tests, 0 failed checks, up from 61
  • make test STD=c++2a (g++): 65 tests, 0 failed checks
  • Header stashed, tests kept: aLoggerMayReportWhileTheQueueIsFull and aLoggerMayFillTheQueueFromTheSendingThread failed after their 30 s bound, and aLoggerMayCloseFromTheSendingThread failed on returned and then aborted the run, so aLoggerMayCloseFromTheReportingThread was not reached
  • Mutation, reverted before commit: with log()'s if (forgotten) logger = Logger(); disabled, aLoggerMayCloseFromTheReportingThread failed with expected [1] got [2]
  • CI Build green on the PR head, including both linux-sanitizers legs, macos and windows: all 11 jobs printed 65 tests, 0 failed checks, and no sanitizer report appears in the logs

The TSan runtime is not installed locally, and clang++ and make test-libcurl were not run locally; all three were covered by CI.

This PR description was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).

🤖 Generated with Claude Code


drafted by Claude on behalf of Daniel Stephenson

dmccoystephenson and others added 2 commits November 4, 2026 21:59
Shared::log held logMutex while the logger ran, so a logger whose report()
found the queue full locked that mutex a second time on the same thread and
hung, on the reporting thread or the sending one. A logger calling close()
hung in forgetLogger the same way, or aborted the program: the destructor
could find close() returning at once while thread_ was still joinable.

Shared now records which thread is running the logger. A message raised on
that thread is dropped, and close() called from it waits for nothing: it
detaches the thread under sharedMutex_ and abandons what is queued, leaving
log() to clear the logger once it has returned.

Closes #24

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the reporting thread the sending thread is not always stuck, only
possibly so; "cannot finish" overstated it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dmccoystephenson

Copy link
Copy Markdown
Member Author

Self-review rubric (performed inline in a headless dispatch; not an independent review):

  • Scope: PASS — three files, all needed for report() hangs forever when a Logger reports while the queue is full #24: the fix, its tests, and the README promises it changes. The abandon() extraction exists because both close() paths now use it.
  • Tests-new: PASS — setLoggerThread, inLogger and abandon are internal to Shared and are exercised through the four new tests.
  • Tests-fix: PASS — with the header stashed, aLoggerMayReportWhileTheQueueIsFull and aLoggerMayFillTheQueueFromTheSendingThread failed on the 30 s bound, and aLoggerMayCloseFromTheSendingThread failed on returned and then aborted with terminate called without an active exception. aLoggerMayCloseFromTheReportingThread was not reached in that run, so it was proved by a mutation instead: with if (forgotten) logger = Logger(); disabled it failed with expected [1] got [2]. All four pass with the fix.
  • Sibling structure / renames: PASS (n/a) — no new files, no renamed identifiers.
  • Docs: PASS — the README "Never throws" and "close() drains" rows, the Logger typedef and the close() comment state the new behavior. The other Phase 7 rows are unaffected: no constant, opt-out, transport or wire-format change.
  • Issue resolution: PASS — the reported hang and both related paths are covered. As the issue requires, a re-entrancy check is used instead of std::recursive_mutex, the logger stays under logMutex, and the regression tests use a time-bounded wait.
  • CI: PASS — all 11 Build jobs succeeded on head 318611d, each printed 65 tests, 0 failed checks, and the logs contain no sanitizer report.
  • Registered: PASS — each new name appears in its definition and in tests[]; the count rose from 61 to 65.
  • C++11: PASS — make test (g++, -std=c++11 -Werror) built clean, and both CI c++11 legs are green.
  • No new dependency: PASS — no #include was added to the header; std::thread::id comes from <thread>, and std::atomic was deliberately avoided. Makefile and CMakeLists.txt are untouched.
  • Never throws: PASS — no new throw. inLogger catches its own failures, and setLoggerThread and thread_.detach() run inside the existing try blocks.
  • Curl argv fixed / Version triple: PASS (n/a) — neither is touched.
  • Sanitizers: PASS — both linux-sanitizers legs are green. Lock order is one way (logMutex, then sharedMutex_, then mutex); mutex is never held while taking either of the others.

Findings:

  • trace_client.hpp:1107 — the first version of this comment said the sending thread "cannot finish while the logger runs". That holds on the sending thread but overstates the reporting-thread path, where it only may be stuck. It was corrected in 318611d, along with the PR description.
  • close() from inside the Logger drops what is queued even when the sending thread could have drained it, on the reporting-thread path. This is accepted: a wait there could deadlock, and the alternative was a hang. The README states it.
  • Not addressed, and unchanged by this PR: a report that finds the queue full still waits on logMutex while the Logger runs on another thread, so a slow Logger can delay that report. close() waits on the same lock. The "Returns immediately" row speaks of the network, so this is not a broken promise, but it is a limit on it.
  • The test bodies run on a detached thread in finishesWithin, so a real regression leaves that thread behind for the rest of the run. That is intended, since a hang then fails a check rather than the suite.

Do-not-auto-merge paths: none touched (no workflow, Makefile, CMakeLists, curl argv, quoting, spawn code, proto line, version or wire-format change).

This comment was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).


drafted by Claude on behalf of Daniel Stephenson

@dmccoystephenson
dmccoystephenson merged commit 7b2173b into main Nov 5, 2026
11 checks passed
@dmccoystephenson
dmccoystephenson deleted the feature/logger-reentrancy branch November 5, 2026 05:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

report() hangs forever when a Logger reports while the queue is full

1 participant