Repository navigation
fix(harness): a settings template adds only SDK setting names (TH-9048) - #298
Open
khushalsonawat wants to merge 4 commits into
Open
khushalsonawat wants to merge 4 commits into
khushalsonawat wants to merge 4 commits into
Conversation
A template is free text, and each review of the shape rules found a new shape that put a fragment of a pasted value into env_names. _env_names now adds a template name only when it is one of 46 names SDKs read for themselves (_SDK_SETTINGS); the code's names, their order and the cap are unchanged. Whatever a template holds, it can add only names from that list. The left-out names are explained beside it. Ten tests that expected a template's other names are rewritten to the rule; new tests cover each route, every value shape in four encodings, 200 seeded values, an exact match, shared code, and the list itself, typed out rather than taken from the constant. TH-9048 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…ate-names-from-code test_scan_walk.py: both sides appended tests at the end (this branch's template-name tests, Signed-off-by: khushalsonawat <khushalsonawat@gmail.com> #293's folder-charge tests); both kept.
cdileep23
requested changes
Nov 12, 2026
…it's names from credentials (TH-9048) FI_API_KEY, FI_SECRET_KEY and FI_BASE_URL are the platform's own tracing account, which the runner and the connect form refuse, so a template no longer adds them: the list is 43 names. A test checks it shares no name with the runner's reserved set. The LiveKit names and its plugins' keys now come from credentials.py, so the two lists can't drift. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…LK_ name is listed (TH-9048) From the fix-only check of a740fbb: a literal copy of credentials.py's lists passed every test, and the reserved-name test didn't cover the runner's ALK_ prefix. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
cdileep23
requested changes
Nov 12, 2026
| "AWS_REGION", | ||
| "AWS_DEFAULT_REGION", | ||
| } | ||
| | frozenset(_CONNECTOR_REQUIREMENTS["livekit"]) |
Member
There was a problem hiding this comment.
P3. The code is good now. Two lines in the PR body are stale: the Known limits bullet still says _SDK_SETTINGS copies credentials.py's list, but it imports it since a740fbb (here); and the Verification box "Fix-only check pending ... the hand-merged test file in 9617660" is still unticked. Update both.
cdileep23
approved these changes
Nov 13, 2026
| "GEMINI_API_KEY", | ||
| "GOOGLE_GENAI_USE_VERTEXAI", | ||
| "GOOGLE_CLOUD_PROJECT", | ||
| "GOOGLE_CLOUD_LOCATION", |
Member
There was a problem hiding this comment.
P3. Two names an SDK reads for itself are not on the list, so a template that names them now adds nothing.
GOOGLE_GENAI_USE_ENTERPRISE. The lockedgoogle-genai2.25.0 reads it besideGOOGLE_GENAI_USE_VERTEXAI(_api_client.pyandlive.py), and google-adk warns thatGOOGLE_GENAI_USE_VERTEXAIis deprecated in its favour. Repro:agent.pywithgenai.Client()and a.env.exampleofGOOGLE_GENAI_USE_ENTERPRISE=true,GOOGLE_CLOUD_PROJECT=,GOOGLE_CLOUD_LOCATION=gives only the last two. Without the flag the client runs in API-key mode and fails.TAVILY_API_KEY.TavilySearch()in a LangGraph agent reads it for itself, so the code never names it. Repro:create_react_agent(ChatAnthropic(...), [TavilySearch()])withANTHROPIC_API_KEY=andTAVILY_API_KEY=gave both before this PR and onlyANTHROPIC_API_KEYnow. The run then fails at the first search, mid-call, not at start.
Fix: add both names (and GOOGLE_GEMINI_BASE_URL / GOOGLE_VERTEX_BASE_URL if base URLs stay listed), and add them to SDK_NAMES in the test.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Into
feat/TH-8804-source-subfolder.Summary
The agent scan lists, for each agent, the settings names it needs (
env_names), and the platform shows them as the connect form's fields. The list is the names the agent's own code reads, then the names in the repository's settings template (.env.example,.env.sample,.env.template) that the code doesn't read.A template is free text a person wrote, and it often holds pasted values: keys wrapped over lines, open quotes, key blocks.
_template_namesguesses which lines are values. In each of TH-8950's three change reviews a new shape put a fragment of a secret value intoenv_names, and each fix closed only the shapes that review found. A fuzzer over the shapes the contract names (1.1 million templates) still got value fragments through by the tens of thousands.Now a template adds a name only when it is one of 43 names SDKs read for themselves (
scan_walk._SDK_SETTINGS): the model providers' keys and base URLs, the tracing names, LiveKit's names and the per-provider keys its plugins read. The filter sits in_env_names, the one function the folder walk,describe_entryand upload mode all call:env_names = _env_names(code, templates & _SDK_SETTINGS)What it promises: whatever a template holds, it can add only names from that fixed list. A line inside a value that spells one of them adds that name; no other text reaches
env_names. The class is closed without guessing at shapes. The code's names, their order, the early return and the cap of 50 are unchanged, and_template_namesand its shape rules stay as they are.Left out of the list on purpose, each explained beside it:
FI_API_KEY,FI_SECRET_KEYandFI_BASE_URL(the platform's own tracing account, which the runner and the connect form refuse; after Dileep's review),GOOGLE_APPLICATION_CREDENTIALS(a key file, which the platform takes through its credential-file control),OTEL_EXPORTER_OTLP_ENDPOINTand_HEADERS(on hosted runs the platform owns trace export), LiteLLM's and LiveKit's OpenAI-plugin provider names (too many to list), and the platform's ownALK_*names.Ticket: TH-9048. Contract:
scan-parts.mdP1.11 and P1.12 (1.3l, agreed with Cedar).Measured
Run over the repositories holding the 25 distinct real templates on this machine, on both routes:
traceAI/python/e2e_tests,livekit-traceai-voice-agent,core/docs-cookbooks,growth/avaandlanggraph-otel-agent.future-agi, 0-17 shown template-only names a row become 0-5. In simulation-acceptance, the names read through a helper (_env(name)) drop from 10 to 5.Known limits
scripts/, a deeper agent's folder) is no longer vouched for by the template. An upload counts the root, so it keeps them.import.meta.env, a pydantic-settings field, or built at run time aren't collected, unless they're SDK names._NAMES_PER_FILE, applied before the filter), as today.credentials.pylistsELEVENLABS_API_KEYfor LiveKit's ElevenLabs plugin, but the installed plugin readsELEVEN_API_KEY, so an agent usingelevenlabs.TTS()is told the wrong required name._SDK_SETTINGSimportscredentials.py's LiveKit names (sincea740fbb2) and listsELEVEN_API_KEYitself too, so a template that names either is listed.Tests
test_a_template_adds_only_sdk_names(the root agent, three routes);test_a_template_value_never_becomes_a_name(each value shape in four encodings, 42 cases, each also checking the template was read);test_a_template_value_never_becomes_a_name_in_random_values(200 seeded values, with a guard that the shapes still let value lines past the reader alone);test_a_value_line_that_spells_an_sdk_name_adds_it;test_a_template_name_matches_exactly;test_a_shared_module_does_not_vouch_for_a_template_name;test_the_sdk_names, whose 43 names are typed out rather than taken from the constant;test_no_sdk_name_is_one_the_runner_refuses,test_the_livekit_names_are_credentials_ownandtest_a_tracing_template_lists_no_platform_account_name(after Dileep's review).Review
Merges with
#293 (Cedar's TH-9449) also adds tests at the end of
test_scan_walk.py. The reviewer merged the two:scan_walk.pymerges clean, the test file's one conflict is resolved by keeping both, and the merged tree passes.Verification
test_scan_walk.py,test_agent_scan.py,test_scan_document.py,test_scan_ai.pyandtest_authoring_entrypoint_find_agents.pyon every push.a740fbb2(Dileep's P2 and P3): clear, no High or Medium (one reviewer, 572 passed on 3.12.3). Its two Lows are fixed inb9da87ac: a test that the LiveKit names are imported fromcredentials.py, not copied, and one that no listed name starts withALK_(the runner refuses that prefix). Fix-only check ofb9da87ac: clear, no findings;test_scan_walk.py212 passed.9617660b: clear, no findings. The merge keeps every test from both sides with none duplicated; 212 passed on 3.12.3.a740fbb2, Python 3.12.3: the five files above, 377 passed;ruff checkclean.9617660b:test_scan_walk.py209,test_agent_scan.py84,test_scan_document.py17,test_scan_ai.py59,test_authoring_entrypoint_find_agents.py5 passed.ruff checkclean on both files; changed lines wrapped at 99;compileallonsrc/.