Skip to content

fix Add opt-in reference lenses with lazy resolution #2194 - #5676

Closed
asukaminato0721 wants to merge 2 commits into
pyrefly-org:mainfrom
asukaminato0721:2194-reference-lenses
Closed

asukaminato0721 wants to merge 2 commits into
pyrefly-org:mainfrom
asukaminato0721:2194-reference-lenses

Conversation

@asukaminato0721

Copy link
Copy Markdown
Collaborator

Summary

Fixes part of #2194

Resolve visible reference counts through the existing search engine on a separate worker. Require client support and an explicit setting, validate document versions, and use cached function metadata to omit overload signatures.

  • I am a human, and I fully understand the code I am submitting.

Test Plan

add test

Resolve visible reference counts through the existing search engine on a separate worker. Require client support and an explicit setting, validate document versions, and use cached function metadata to omit overload signatures.
@meta-codesync

meta-codesync Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

This pull request has been imported. If you are a Meta employee, you can view this in D132645470. (Because this pull request was imported automatically, there will not be any future comments.)

@github-actions

Copy link
Copy Markdown

✅ This PR meets all requirements from our pull request checklist.

Optional recommendations:

  • This PR is very large, which makes it slow and difficult to review. We strongly recommend splitting it into smaller PRs that can be merged independently. If that isn't possible, please structure it as a series of small commits. If you've already split it as far as is practical, no action is needed.

Note: this comment updates automatically after changes to the PR. Updates may take up to half an hour.

@github-actions

This comment has been minimized.

@stroxler stroxler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! This review is still pretty big; AI-assisted review thinks it could potentially be split. Up to you, but this is out of my wheelhouse and it's very big so you're pretty much getting an AI review here.

AI review notes two major issues related to a possible stall, and one minor concern. The major issues are by far the most important, I'm mostly not too fussed about IDE functionality being perfect especially with potentially expensive logic being opt-in, but stalling the session would be bad!

A count search can stall edit processing

Changed scheduling path: pyrefly/lib/lsp/non_wasm/server.rs:5721–5759.

Each lens search opens a cancellable transaction at line 5726. That transaction retains the state read guard (pyrefly/lib/state/state.rs:4082, 4117–4118). Processing an edit can synchronously commit on the main LSP thread (server.rs:4486 → 3539 → 3678), and committing waits for the state write lock (state.rs:4299). This commit path does not cancel long-running readers first; the background recheck path does (server.rs:3879–3884). A separate worker/queue therefore does not isolate editing from the search's transaction lifetime. The transaction also remains alive while the external-reference thread is joined (server.rs:5844–5865).

Controlled reproducer: block the first lens search in a supported external-reference provider; after the provider acknowledges the search, send didChange for a different open file, then a CodeLens request as a dispatch barrier. The barrier did not finish dispatch during a one-second observation window. After releasing the search, both requests completed. The source-level lock chain explains the dependency; this was not merely a large-project timing observation.

Contract and impact: #2194 explicitly requires counting not to block other features. A main-thread commit waiting behind a proactive lens search stalls subsequent hover, completion and other request processing. The read-lock mechanism pre-exists for interactive searches, but this change adds an automatic, per-visible-definition workload to it.

Required here: make the new lens workload yield to editor commits, including avoiding retention of the state guard across external waits. Cancellation and lock-lifetime handling must actually unblock commits; queue separation alone is insufficient. Add an edit-during-search regression test. Documentation or a later optimization cannot make the feature meet its responsiveness requirement.

Notebook exclusion disables existing Run/Test lenses

pyrefly/lib/lsp/non_wasm/server/code_lens.rs:125–138, plus the deleted cell filter in server/code_lens/runnable.rs.

The new notebook-cell early return runs before the runnable provider and before checking whether reference lenses are enabled.

Same-input base/head reproducer: initialize with runnableCodeLens=true, referencesCodeLens=false, no reference-client opt-in and indexing off. Open a .py-backed notebook with one code cell:

def test_example():
    pass

Request CodeLens for the cell URI. The base returns one pyrefly.runTest “Test” lens; the head returns []. Ordinary .ipynb notebooks were already excluded by the runnable provider, but .py-backed cells previously passed its per-cell filter.

Contract and impact: the new setting is documented as separate from Run/Test actions and not affecting them. This removes a previously supported action with the new feature completely disabled. It is not a reference-count precision boundary.

Required here: restrict notebook exclusion to the reference provider and preserve runnable per-cell behavior. This is a narrow regression introduced by this change; redefining notebook support or deferring it would unnecessarily remove unrelated existing functionality.

What an incomplete index means to a displayed count

With default background indexing, early counts can reflect an incomplete index. Completion triggers refresh; configured indexing limits can still leave counts incomplete, as they do for Find References. Both base-only reviewers preferred pending/provisional treatment. The issue discusses hiding lenses while indexing, but does not establish a complete-count contract. Decide whether numeric labels should be delayed or explicitly best-effort. This is a support-contract question, not a proven additional correctness blocker.

Cancel speculative lens readers before commits and release reference transactions before external waits. Preserve runnable notebook lenses independently of reference-count settings.
@github-actions github-actions Bot added size/xl and removed size/xl labels Nov 19, 2026
@github-actions

Copy link
Copy Markdown

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@stroxler stroxler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.

@meta-codesync

meta-codesync Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

This pull request has been merged in 6ac29d8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants