Skip to content

fix(harness): answer the framework comments on the subfolder PR - #248

Merged
khushalsonawat merged 2 commits into
feat/TH-8804-source-subfolderfrom
fix/TH-8805-dileep-comments
Nov 6, 2026
Merged

khushalsonawat merged 2 commits into
feat/TH-8804-source-subfolderfrom
fix/TH-8805-dileep-comments

Conversation

@khushalsonawat

@khushalsonawat khushalsonawat commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Stacked on #225 (TH-8804). This PR's base is feat/TH-8804-source-subfolder, so its diff shows only these fixes. Two commits: the fixes, then the repeated tests removed at review.

Summary

This answers the review's three code comments on src/fi/alk/harness/framework_detection.py, which reached #225 from TH-8805 (#229). Each comment was reproduced at 3e4e3ec5 before it was fixed.

  • JavaScript counts as other source. .js, .jsx, .mjs and .cjs now count, as do .mts and .cts, like .ts. A Python folder beside them reads unknown instead of none. Files under test, example and similar folders still don't count, and a named framework still wins. File names are matched case-sensitively, as before.
  • Any blocker comes before the chat-model rule. A chat model names langchain only when nothing the check couldn't read might be building the agent. Before, a folder with a chat model and, for example, an unread file beside it read langchain. It now reads unknown.
  • A repository-origin process is scanned in the source. framework_folders_for takes the folder's base from source_origin, the same base the runtime builds the process from (process_runtime.py). Bundle-origin processes stay in the bundle. Every process the authoring step writes is bundle-origin, so only hand-written manifests change folder.

Ticket: TH-8805.

Known limits

  • A LangChain agent that only uses a chat model, with no agent or graph builder, reads unknown whenever anything blocks. Triggers include test or eval files that import a framework, the app's own agents package, and JavaScript or TypeScript outside test and example folders. That includes config and built files such as tailwind.config.js, static/js/ and web/dist/. A StateGraph app still reads langgraph. This is the rule the review asked for. Narrowing it is possible later.
  • .JS and .TS in upper case aren't matched, as before.

Verification

  • tests/harness/test_framework_detection.py: 262 passed. tests/harness/test_hosted_entrypoint.py: 136 passed. tests/harness/test_scenario_source_provision_agent.py: 23 passed. All on Python 3.12.14, each file in its own run, on 9c187fe2 plus this commit. On Python 3.12.3, the sandbox's version, the detector and scenario-source files give 262 + 23.
  • 10 planted bugs, one per changed rule and test, each caught by its named test.
  • Over 640 synthetic folder findings, the decision changes in 14 rows. Every one goes from langchain to unknown, with a chat model and one blocker. No answer changes in the 101 design-review fixtures, the 19 examples/frameworks or the 2 harness fixtures.
  • One cold change review, clear (High 0, Medium 2, Low 4). Both Mediums were about feat(harness): build an agent from a repo's subfolder #225's description text. Applied after that review, not reviewed: two test rows (.jsx under tests/, .mjs under examples/), one test comment, the description text, and the removal of the repeated tests asked for in this PR's review.
  • ruff check passes on the three files.

🤖 Generated with Claude Code

- Count .js, .jsx, .mjs, .cjs, .mts and .cts files as other source, so a
  Python folder beside them reads unknown, not none. Test and example
  folders still don't count.
- Answer unknown on any blocker before the chat-model rule, so a chat
  model names LangChain only when nothing unseen could build the agent.
- Scan a repository-origin process in the source, the same base the
  runtime builds it from; bundle-origin processes stay in the bundle.

TH-8805

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
@khushalsonawat khushalsonawat self-assigned this Nov 6, 2026

@cdileep23 cdileep23 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the comments.

if process.name == control_service:
folder = bundle_dir / process.working_directory
# The same base the runtime builds the process from.
base = bundle_dir if process.source_origin == "bundle" else source

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now the same rule as build_process_trees in process_runtime.py (context.bundle_dir if process.source_origin == "bundle" else source_root). The comment above it exists because the two copies have to stay in step, and nothing enforces that. If a third origin is added, or the rule changes in the runtime, this copy will drift without a failing test.

Optional, since two copies is still within the rule of three: a small method on SourceProcess, for example def root(self, *, source: Path, bundle_dir: Path) -> Path, would let both call sites use one rule, and the comment could go.

@khushalsonawat khushalsonawat Nov 6, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that the two copies can drift. I'd do this as a follow-up together with the typing comment: the method needs the tests' SimpleNamespace processes replaced by real SourceProcess models, which is that change. I'm leaving this thread open until that's settled.

@@ -1004,7 +1017,9 @@ def framework_folders_for(
elif control_service:
for process in manifest.processes:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not caused by this PR, but it now depends on it: framework_folders_for takes manifest: Any (and job: Any), so process.source_origin and process.working_directory are read with no type to check them against. A typo or a renamed field would only show up at runtime.

The manifest is an EnvironmentBundleV2, so typing the parameter as that would let a type checker see these reads. The tests pass SimpleNamespace fakes, so they would need the real model or a cast. Fine as a follow-up.

@khushalsonawat khushalsonawat Nov 6, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I'd do this as a follow-up, together with the shared SourceProcess method from the other comment, since both need the tests' fakes replaced by the real model. No change in this PR. I'm leaving the thread open until that's settled.

- The chat-model-with-a-blocker test repeated test_blockers_stop_none,
  which puts a chat model beside every blocker; its check that the chat
  model is read as a model moves there.
- The JavaScript test keeps only the test and example folder rows; the
  .js and .mjs rows repeat BLOCKER_CASES.
- The standalone repository-process test goes; the folder-order row and
  the end-to-end hosted test cover the rule.

TH-8805

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
@khushalsonawat
khushalsonawat merged commit 7d8ad86 into feat/TH-8804-source-subfolder Nov 6, 2026
11 checks passed
@khushalsonawat
khushalsonawat deleted the fix/TH-8805-dileep-comments branch November 6, 2026 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants