Skip to content

Flake: nightly poetry hosted rollback fails on one transient PyPI request (UpstreamClient has no retry) #1650

Open
Open
Flake: nightly poetry hosted rollback fails on one transient PyPI request (UpstreamClient has no retry)#1650
Labels
agent:triagedci-janitorOpened by the CI janitor routine (flakes, redundant tests, CI perf)priority:p3

Description

@mikolalysenko
Mikola Lysenko (mikolalysenko)
opened on Oct 21, 2026

Nightly e2e-extended (ubuntu-latest, e2e_vex_build, poetry:: --ignored, 1.1.15) failed once on a single failed PyPI request during hosted rollback.

Evidence

  • Run https://github.com/SocketDev/socket-patch/actions/runs/39101574188 (schedule, main, 2026-10-18), job 117599202945
  • poetry::poetry_hosted_fresh_install_then_manifestless_vex panicked at crates/socket-patch-cli/tests/e2e_vex_build/poetry.rs:898: rollback exited 1 with hosted_restore_refused:
    cannot restore pkg:pypi/six@1.16.0 to its upstream registry entry: six==1.16.0: GET https://pypi.org/pypi/six/1.16.0/json: error sending request for url (...)
  • The other 3 poetry tests in the same job reached PyPI fine seconds earlier. The 10-19 nightly passed on the same code.
  • Frequency: 1 of the last 12 nightlies. It has not been seen on PR or merge_group runs (this leg is nightly-only).

Root cause (partly known)
UpstreamClient (crates/socket-patch-core/src/patch/redirect/upstream/client.rs) sends each registry lookup once. get_json_authorized and download turn any transport error into a final GET {url}: {e} with no retry. The patch API client does retry dropped connections and 429/503 (api::retry, send_json_request). Because the message keeps only reqwest's top-level text, the log can't tell us the transport error class (DNS, reset, timeout).

Proposed fix (not started: the evidence was too thin this run)

  1. Format registry-lookup errors with the cause chain, the same way api::client::network_error_detail does, so the next occurrence is classifiable.
  2. If it recurs as a dropped connection or a 429/503, give UpstreamClient's GETs the same bounded ApiRetry policy that send_json_request uses (is_retryable_transport plus is_retryable_status, the shared retry window). A user's hosted rollback hits the same failure mode.

Filed by the hourly CI janitor; the job log expires, so the signature is quoted above.

Generated by Claude Code

Activity

  1. Mikola Lysenko (mikolalysenko)
    added
    ci-janitorOpened by the CI janitor routine (flakes, redundant tests, CI perf)
    agent:triaged
    priority:p3
    on Oct 21, 2026

  2. mikolalysenko commented on Oct 21, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 21, 2026
    CollaboratorAuthor

    [agent] Triaged (CI lane): p3, nightly-only flake (1 in 12 nightlies, not seen on PR or merge-queue runs), so it costs no PR/queue runner time. No open PR covers it yet. The proposed fix is in UpstreamClient (core), so it also affects users doing a hosted rollback. Not claimed this run because the CI-lane fixer already has 3 open PRs (#1641, #1642, #1647) waiting on review.

    Generated by Claude Code

  3. mikolalysenko commented on Oct 22, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 22, 2026
    CollaboratorAuthor

    [agent] Re-verified (CI lane, 2026-10-22). Step 1 of the proposed fix is on main now: #1652 (bc307ce) formats registry-lookup errors with the transport cause chain (network_error_detail in crates/socket-patch-core/src/patch/redirect/upstream/client.rs, get_json_authorized and download). The regression test a_failed_lookup_keeps_the_transport_cause covers it.

    Step 2, the bounded retry, still depends on the next occurrence's error class, as the issue says. I'm not claiming it until a nightly shows the flake again with the new message. If it shows a dropped connection or a 429/503, the fix is to give UpstreamClient's GETs the api::retry policy.

    Generated by Claude Code

  4. mikolalysenko commented on Oct 26, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 26, 2026
    CollaboratorAuthor

    [agent] Re-verified (CI lane, 2026-10-26 21:0x UTC). No recurrence. The 4 nightlies since the last check (39907645034, 40067511756, 40166694387, 40276353917) had no failed poetry hosted-rollback job. Their only red legs were test (windows-latest, *) (the yarnrc expectation, fixed by #1710, merged today) and, on 10-23, two e2e-extended gem legs (Ruby 3.1, Bundler 2.1.4 and 1.17.3). Still waiting on a new occurrence with the #1652 error detail before choosing the retry policy, so I'm not claiming it.

    Generated by Claude Code

  5. mikolalysenko commented on Oct 28, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 28, 2026
    CollaboratorAuthor

    [agent] Re-verified (CI lane, 2026-10-28 08:1x UTC). Still no recurrence. Nightly 40639505834 (10-28) had no failed poetry job. Its red legs were e2e_bun_lockb (#1770), mode_migration_vlt (#1774) and a deno install blip. Still waiting on an occurrence with the #1652 error detail.

    Generated by Claude Code

  6. mikolalysenko commented on Oct 29, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 29, 2026
    CollaboratorAuthor

    [agent] Re-verified (CI lane, 2026-10-29 07:5x UTC). Still no recurrence. Nightly 40824880386 (10-29) had no failed poetry job. Its only red legs were test (windows-latest, 1) and test (windows-latest, 2): the Go workspace fixtures (#1815, fixed by #1816) and the cargo-home tests (#1814). Still waiting on an occurrence with the #1652 error detail before picking a retry policy, so not claiming it.

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent:triagedci-janitorOpened by the CI janitor routine (flakes, redundant tests, CI perf)priority:p3

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions