Skip to content

test(cloudxr): poll the run-flags cmdline read in test_background.py - #1376

Open
ivany-nv wants to merge 2 commits into
mainfrom
ivany-nv/fix-flaky-cmdline-test
Open

ivany-nv wants to merge 2 commits into
mainfrom
ivany-nv/fix-flaky-cmdline-test

Conversation

@ivany-nv

@ivany-nv ivany-nv commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

cloudxr_test_background fails intermittently on main. TestReadRunFlags::test_reads_real_cmdline_without_a_trailing_empty failed with assert [] == ['--host-client'] on build-ubuntu (Debug, 3.12, arm64) for #1372, which does not touch this code (run 40632105330). Fail-fast then cancelled the other build jobs.

Popen() returns once the child has started exec, but its /proc/<pid>/cmdline reads empty until the kernel has set up the new argv. The test read the run flags straight away, so in that window read_pid() did not recognise the child and read_run_flags() returned []. #1209 and #1327 fixed the same race for other tests in this file by polling read_pid(); this one was missed.

  • Replace _poll_read_pid() with _await_exec(pid), which waits for the child's cmdline itself. It gives up after 5 s with pytest.fail("/proc/<pid>/cmdline is still empty 5s after the spawn").
  • Call it first in every test here that reads a just-spawned child's cmdline. That includes the two that expect no match (test_none_when_pid_belongs_to_something_else, test_a_process_that_merely_names_the_module_is_not_the_service): an empty cmdline made them pass without exercising the matching at all, which polling read_pid() cannot fix.
  • Tighten the matching rule in tests/AGENTS.md to say the wait comes before any check, negative ones included.

No product code changes. status, stop and the launcher read the cmdline of a service that finished exec long ago. start_and_wait() does call read_pid() right after spawn() and shares this window, but this test does not cover it; that is product code and left for a separate change.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Testing

All on a 48-core x86_64 host, against a Release build of main.

  • CTest's command in the leaf: uv run --python 3.12 --extra dev pytest -v --tb=short test_background.py → 20 passed.
  • 16 parallel workers, each pytest --count 300 (pytest-repeat) on the failing test, under stress-ng --cpu 96 --fork 16:
    • before: 32 of 4800 failed, all with CI's assert [] == ['--host-client']
    • after: 0 of 4800 failed
  • Whole file, 16 parallel workers × --count 100 under the same load: 32,000 passed, 0 failed.
  • Without parallel load the race did not show (300 serial runs passed before the fix). A bare Popen() followed by an immediate cmdline read came back empty 1,824 of 16,000 times under the load above.
  • Fail path: _await_exec() on an unreaped zombie fails after 5 s with the message above, and the test's finally still reaps the child.
  • posix_spawn path (CPython 3.13's default where os.POSIX_SPAWN_CLOSEFROM exists; forced here with close_fds=False): the child can briefly show the parent's argv, so _await_exec also waits past that. Before: 7 of 10,000 cmdline-test runs failed; after: 0 of 10,000, and 0 of 12,000 under stress-ng on both paths.
  • SKIP=check-copyright-year pre-commit run --all-files passed.

This PR is itself the test fix, so it adds no new test.

Not covered yet: arm64, where CI failed; this PR's CI run covers it.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the linter and formatter with SKIP=check-copyright-year pre-commit run --all-files
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix/feature works (or explained why not)
  • I have signed off all my commits (git commit -s) per the DCO

Summary by CodeRabbit

  • Tests
    • Background-process checks now wait for a process to start before verifying its state, improving the reliability of those checks.
  • Documentation
    • Clarified guidance for keeping stand-in child processes running and waiting for startup before checking their command line.

Popen() can return while the child is still mid-exec, when its
/proc/<pid>/cmdline reads empty. test_reads_real_cmdline_without_a_trailing_empty
read the run flags at once and failed with `assert [] == ['--host-client']`.

Replace the per-reader read_pid() poll with _await_exec(), which waits for
the child's cmdline itself and fails clearly after 5s. Every test that reads
a just-spawned child's cmdline now calls it first, including the two that
expect no match: an empty cmdline made those pass without testing anything.

Signed-off-by: Ivan Yang <yifanyang@nvidia.com>
@ivany-nv ivany-nv self-assigned this Oct 28, 2026
@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between a74babb and 036e95e.

Files selected for processing: 2
  • tests/AGENTS.md
  • tests/python/core/cloudxr/test_background.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.

Walkthrough

The background tests replace PID polling with a helper that waits for /proc/<pid>/cmdline to become non-empty. Tests use the helper before checking process recognition, PID reuse, run flags, or termination. The stand-in child guidance also requires waiting for command-line readiness before checks.

Priority: Low

Estimated code review effort: 2 (Simple) | ~8 minutes

Possibly related PRs

Merge Risk: Minimal · previous review 036e9

The test-only change adds bounded process-readiness waiting without an identified merge-blocking risk.

Pre-merge checks: Passed 5
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly identifies the CloudXR test change and the command-line polling behavior addressed by the pull request.
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 1 files. (1 skipped: 1 unsupported.)

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

…dline

On the posix_spawn path the child shares our memory until exec swaps it,
so /proc/<pid>/cmdline can briefly read pytest's own argv. _await_exec
then returned at once, and the next read could still miss the service.
CPython 3.13 takes that path by default where os.POSIX_SPAWN_CLOSEFROM
exists; uv's managed builds lack it, so CI is unaffected.

Forced onto posix_spawn, the previous helper failed 7 of 10,000 runs of
the cmdline tests; this one failed none.

Signed-off-by: Ivan Yang <yifanyang@nvidia.com>

This branch was successfully deployed

1 active deployment
dev — 7ed7d58e Deployed Oct 28, 2026 by ivany-nv via publish-wheel #6137
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants