Skip to content

fix: --sharpit-app reports an uninspectable app as unknown, not as an empty pass - #207

Draft
michaelegner wants to merge 10 commits into
ffroliva:mainfrom
michaelegner:fix/sharpit-app-uninspectable
Draft

michaelegner wants to merge 10 commits into
ffroliva:mainfrom
michaelegner:fix/sharpit-app-uninspectable

Conversation

@michaelegner

Copy link
Copy Markdown

Found while scoping #75 (Flask adapter). Independent of Flask: no Flask code and no new dependency.

Problem

pytest --sharpit --sharpit-app <spec> on main (47753e64):

--sharpit-app names Before After
a factory returning an object with no route list (a Flask app, an async factory's coroutine) rest-api PASSED, no route inspected ("usable": false) UNKNOWN
an object with no route list, named directly (a WSGI app, CORSMiddleware(app)) UNKNOWN: application could not load (TypeError): the loader called it UNKNOWN, not called
an APIRouter / Starlette Router UNKNOWN: application could not load (TypeError): the loader called it assessed, not called
a FastAPI app with no routes PASSED PASSED (confirmed empty, 0.3.0 spec §6)

The new reason is application routes cannot be inspected (<module.Class>); expected an app, a router or a zero-argument factory.

Cause

  • _route_list in fastapi/introspector.py returned [] both for "no routes" and for "no route list to read". run_advisors skips the "no routes seen" guard when --sharpit-app is given, so the empty model went on to the confirmed-empty PASSED.
  • load_app called every callable without a router attribute. A WSGI app and a router are callable, so both were called with no arguments.

Fix

  • _stored_routes returns None when there is no route list. inspectable(app) and _route_list share it, so the two cannot drift. A class is never inspectable.
  • load_app calls the named object only when it accepts no arguments. It is used as it is when it is inspectable, has a router attribute (inspect.getattr_static), needs arguments, or has an unreadable signature. On Python 3.14 the signature is read with annotation_format=FORWARDREF, so a factory annotated with a TYPE_CHECKING-only name is not broken.
  • _configured_models reports UNKNOWN for an object that is not inspectable. The reason goes through the report masker, and the class name is never formatted from a non-string __module__.

Behaviour change

A suite that named a non-app object with --sharpit-app was green and is now unknown (exit 1) unless --sharpit-allow-unknown is passed. This changes one existing test, test_live_engine_driver_counts_as_covered. It named a SQLAlchemy engine to get its module imported. Branch review also showed it did not bite: driver detection runs before advisors import the app. It now imports the module from the test file. With the engine it exits 0; without the engine it exits 1 with untraced database driver in use: aiosqlite.

Finding → test → red output

All reds were checked with scripts/dev/red_check.py: RED (assertion under tests/).

Finding Test Red before the fix
False pass for a factory of an unreadable object test_sharpit_app_factory_of_an_uninspectable_app_is_unknown assert 'PASSED' == 'UNKNOWN'
Same, default gate test_sharpit_app_uninspectable_app_fails_the_run_by_default assert <ExitCode.OK: 0> == 1
WSGI-shaped instance called test_sharpit_app_instance_without_a_route_list_is_uninspectable reason application could not load (TypeError)
Router refused test_sharpit_app_router_is_assessed assert 'UNKNOWN' in ('PASSED', 'FAILED')
Inspectable callable called test_load_app_never_calls_an_inspectable_object assert 1 == 0 (call count)
Unreadable signature called test_load_app_never_calls_a_callable_with_an_unavailable_signature assert 1 == 0 (call count)
Callable needing arguments called test_load_app_returns_a_callable_that_needs_arguments assert None is <function needs_arguments>
A class holding routes taken for an app test_inspectable_is_false_for_a_class_that_holds_a_route_list assert True is False (wrong-value stub, as in docs/development/workflow.md)
Reason not masked test_uninspectable_reason_passes_the_report_masker stub: no run reported
Host __str__ formats the reason test_uninspectable_reason_never_formats_a_host_module_object stub: assert 1 == 0
Python 3.14: annotation evaluated (found in branch review) test_load_app_calls_a_factory_annotated_with_a_type_checking_only_name NameError caught, assert None is <FastAPI> (red on 3.14.6; skipped below 3.14)

Controls that pass before and after:

  • --sharpit-allow-unknown tolerates the new UNKNOWN
  • an empty FastAPI app stays PASSED
  • a FastAPI factory is assessed
  • a raising factory keeps application could not load (<Type>)
  • an object with a router attribute is not called (removing the rule turns it red)
  • a factory with optional arguments is called

Limitations

  • A successful signature.bind() shows a callable accepts no arguments, not that it is a factory. A callable app whose __call__(*args) accepts zero arguments is still called; the result is UNKNOWN.
  • inspect.signature reads __wrapped__/__signature__, so loading a named app is not free of host code. The old hasattr was not either.
  • A proxy that forwards router through __getattr__ was returned as it was before; it is now called. Before it gave a false PASSED, now UNKNOWN.
  • A router's assessment covers its routes only, not the handlers an app adds around them: RAPI-ERR-001 (INFO) can appear for a router where the app would not show it.
  • Out of scope: the neighbouring reason application could not load (...) is not masked either; that predates this change.

Process

  • /sharpit:predict: five personas, CAUTION 5.6/10, no stop condition; mitigations are in the plan.
  • /sharpit:scenario: docs/superpowers/plans/2026-11-16-sharpit-app-uninspectable-scenario.md.
  • /sharpit:plan: docs/superpowers/plans/2026-11-16-sharpit-app-uninspectable.md.
  • /sharpit:branch-review: seven lenses. One High and one Medium were fixed test-first; the Lows are listed above.

Gates (local)

  • ruff check, ruff format --check and pyright are clean. Pyright is also clean on --pythonversion 3.14 for apps.py.
  • Full suite pytest -q -n 8 (idle machine): 15,186 passed, 6 failed, none from this change.
    • test_dotenv and test_nested_secret_json_is_masked_get_and_mcp[__] fail the same way on untouched main on this machine (local environment).
    • Four overhead-budget tests fail only under -n 8. Run alone (-n 0) all 11 tests in those files pass, on this branch and on untouched main. The change touches neither the capture middleware nor the SQL listener.
  • Run on the changed test files only, not as full jobs:
    • lowest-direct (Python 3.12, fastapi 0.115.7, starlette 0.44.0)
    • without SQLAlchemy
    • under coverage (pytest_plugin/apps.py 94%)
    • Python 3.14.6 (47 passed)
  • End to end with a real Flask + Flask-SQLAlchemy suite: --sharpit-app pkg:app and pkg:create_app both give UNKNOWN with the new reason, and the N+1 finding still appears.

@michaelegner michaelegner mentioned this pull request Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant