diff --git a/README.md b/README.md index 313793b..52bbcbc 100644 --- a/README.md +++ b/README.md @@ -177,6 +177,9 @@ optional task subset; or explicitly not supplied): Work scope (implementation/local tests, also MEDS-DEV integration, or preparation for full evaluation): +Scientific ambiguity mode (confirmation required by default; exploratory only if explicitly enabled): + + Additional implementation constraints: @@ -186,6 +189,32 @@ one always overrides the other. If only a paper is provided, distinguish source- necessary assumptions. If only code is provided, inspect its revision, configuration and checkpoint requirements; do not claim fidelity to an unavailable paper. +Resolve scientific ambiguities before implementing the affected behavior. Check the paper, supplements, +official code and available configurations first; record sources and material conflicts. Ask whether two +reasonable interpretations could produce materially different algorithms or change what evaluation measures. +A missing learning rate or batch size can usually use a plausible, configurable default with its rationale +recorded, unless that choice is central to the scientific claim. Missing feature construction, temporal +aggregation, architecture components or training objectives are material ambiguities, not routine defaults. + +In confirmation-required mode (also the default if no mode is supplied), pause dependent implementation +and request confirmation for a material unresolved ambiguity. Explain what is missing, the evidence +consulted, plausible interpretations and their consequences; recommend an option when justified. Continue +independent work while waiting. An earlier explicit decision covering the same choice is sufficient unless +new evidence or a scope change invalidates it; silence is not confirmation. + +Only when exploratory mode is explicitly enabled may you implement provisional interpretations of material +algorithm-design ambiguities without first obtaining a decision. Record each as provisional before using +it, expose the choice in configuration where appropriate, and label affected code, tests and results as an +exploratory adaptation. Require review of these decisions before claiming faithful reproduction. Do not +present an assumption as source-backed behavior or silently substitute a familiar method for a missing one. + +In BOTH modes, ambiguities about leakage, permitted label access, cohort/target definitions or evaluation +validity remain blocking for dependent work until explicitly resolved. Exploratory mode does not authorize +changing supplied tasks, weakening acceptance tests, accessing data or running demo/full-data/cluster jobs +without the existing approvals. Record the mode, routine defaults, unresolved questions, provisional choices +and explicit decisions in IMPLEMENTATION_REPORT.md under "Scientific assumptions and decisions"; update the +existing report rather than creating another specification file. + For automatic profile selection, choose one or more profiles supported by documented source behavior: supervised, finetune, probe, zero_shot_direct, zero_shot_materialized or predict_only. Record evidence for each choice and proceed when the evidence is clear. Do not add speculative capabilities merely because @@ -240,6 +269,13 @@ Write minimalist, auditable, object-oriented algorithm code. Put source-defined in the algorithm-owned frozen Config class in config.py; validate values and prove serialization round trips. Keep the implementation in .py and expose the named algorithm and configuration classes from the package, so callers can import them meaningfully. +Keep the public API intentional and small. Classes, functions and modules used only inside the implementation +should use a single leading underscore (for example _FeatureEncoder or _features.py) and should not be +re-exported from the package __init__.py. List only intended public exports in __all__. Keep contract-required +methods and integration hooks publicly named where required. A leading underscore signals an internal, +unstable interface in Python; it does not enforce access control. Prefer it over double-underscore name +mangling unless avoiding a specific inheritance collision. Do not create extra classes just to organize +private helpers, and do not hide the algorithm/configuration classes callers are expected to import. For neural algorithms with matching semantics, extend core NNConfig rather than duplicating shared capacity, learning_rate, batch_size, weight_decay and dropout settings. Define source-backed defaults and the capacity-multiplier mapping (reference architecture, dimensions, rounding @@ -419,6 +455,9 @@ and [shared-package assessment](https://github.com/McDermottHealthAI/MEDS_algori ### Unreleased +- Require confirmation for material scientific ambiguities by default; offer explicitly enabled exploratory + assumptions in the prompt builder while preserving evaluation and execution approval gates. + - Generate an algorithm release guide, Trusted Publishing workflow, and installed-distribution checks. - Configure PyPI metadata separately from Python names and bundle portable run/profile YAMLs once at build time. diff --git a/TESTING.md b/TESTING.md index 4c8d81e..85d9e31 100644 --- a/TESTING.md +++ b/TESTING.md @@ -39,7 +39,9 @@ builder uses plain JavaScript without runtime dependencies, network calls or per | Test | What it checks | Why it exists | |---|---|---| -| All choice combinations | Exercises all 72 source/destination/profile/task/scope combinations and common instructions. | Preserve the shared scientific requirements and review gates for every configuration. | +| All choice combinations | Exercises all 144 source/destination/profile/task/scope/ambiguity-mode combinations and common instructions. | Preserve scientific review and execution gates in both confirmation-required and exploratory modes. | +| Ambiguity-mode default and validation | Confirms omitted/false mode requires confirmation and rejects non-boolean opt-ins. | Avoid enabling exploratory assumptions through an absent or truthy string value. | +| Exploratory checkbox lifecycle | Checks unchecked defaults, explicit opt-in, stale-output invalidation, toggling back and reset in a DOM. | Require an intentional selection and restore confirmation mode on reset. | | Invalid plans | Rejects missing sources, destination, manual profiles or prepared-bundle metadata. | Avoid an apparently ready prompt with incomplete required inputs. | | Task subsets and literal text | Retains existing task identifiers and arbitrary input text without recursive replacement. | Keep the user's supplied scope and constraints intact. | | Conditional form fields | Exercises visibility, required fields and multi-profile generation in a DOM. | Prevent hidden fields or single-profile defaults from overriding the selected options. | diff --git a/docs/assets/prompt-builder.js b/docs/assets/prompt-builder.js index 48df29c..11eca1d 100644 --- a/docs/assets/prompt-builder.js +++ b/docs/assets/prompt-builder.js @@ -23,6 +23,9 @@ const mode = choice(values.profile_mode, ["automatic", "manual"], "profile selection"); const tasks = choice(values.tasks_mode, ["none", "ready"], "task-input option"); const scope = choice(values.scope, Object.keys(scopes), "work scope"); + if (values.exploratory_mode !== undefined && typeof values.exploratory_mode !== "boolean") { + throw new Error("Exploratory mode must be an explicit checkbox selection."); + } let repository = "New local Git repository only; do not create a GitHub repository."; if (destination === "github") { const name = required(values.github_repo, "the new GitHub repository") @@ -61,6 +64,9 @@ DATASET_CONTEXT_OR_NOT_SUPPLIED: clean(values.dataset) || "Not supplied; use synthetic data for local implementation tests.", PREPARED_TASK_BUNDLE_OR_NOT_SUPPLIED: bundle, WORK_SCOPE: scopes[scope], + SCIENTIFIC_AMBIGUITY_MODE: values.exploratory_mode === true + ? "Exploratory mode explicitly enabled: provisional algorithm-design assumptions are permitted under the policy below." + : "Confirmation required (default): ask before implementing material unresolved scientific choices.", ADDITIONAL_CONSTRAINTS_OR_NONE: clean(values.constraints) || "None." }; return template.replace(/<([A-Z][A-Z0-9_]*)>/g, (token, key) => { @@ -112,6 +118,7 @@ const data = new FormData(form); const values = Object.fromEntries(data.entries()); values.profiles = data.getAll("profiles"); + values.exploratory_mode = field("exploratory_mode").checked; try { output.value = renderPrompt(template, values); output.readOnly = false; diff --git a/docs/prompt-builder.md b/docs/prompt-builder.md index 940b677..b035ce8 100644 --- a/docs/prompt-builder.md +++ b/docs/prompt-builder.md @@ -74,6 +74,8 @@ You can edit the result before copying. Demo and full-data execution retain thei

Preparing for full evaluation does not authorize a demo, dataset download or cluster run. The existing review gates still apply.

+ +

Off by default. The agent normally asks before implementing a major unresolved scientific choice, such as unspecified feature construction; routine defaults may proceed with documentation. When enabled, provisional design choices must be recorded and reviewed before claiming faithful reproduction. Leakage, cohort/target definitions, evaluation validity and execution approvals still require resolution.

diff --git a/docs/workflow.md b/docs/workflow.md index f1dc548..38c3304 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -47,6 +47,13 @@ local repository or a new GitHub repository with an explicit local checkout and Automatic profile selection must follow source evidence; manual selection may include several profiles. The agent records the choice before generation, then implements every selected profile. +By default, the agent asks for confirmation before implementing a material unresolved scientific choice +(for example, unspecified featurization). Routine defaults such as a missing learning rate may proceed +with a documented rationale unless central to the paper's claim. The builder's **Exploratory mode** +checkbox is unchecked by default: enabling it permits provisional design assumptions, which must be +recorded and reviewed before claiming faithful reproduction. In both modes, unresolved leakage, +cohort/target definitions or evaluation validity block dependent work. Independent work can continue. + Evaluation tasks, when supplied, are already ready: labels plus task configs, bundle identity and dataset context. The builder does not ask the agent to invent tasks or extract new labels. Missing tasks defer only the corresponding evaluation. Choose the intended work scope explicitly; demo and full-data runs diff --git a/template/AGENTS.md.jinja b/template/AGENTS.md.jinja index ebba3d6..942be25 100644 --- a/template/AGENTS.md.jinja +++ b/template/AGENTS.md.jinja @@ -3,6 +3,10 @@ Before editing, read `SPEC.md`, `README.md`, and `docs/IMPLEMENTATION_GUIDE.md` completely. `SPEC.md` points to the pinned core contract: run `meds-algorithm-spec` and read that specification too. +Follow the scientific-ambiguity policy in `docs/IMPLEMENTATION_GUIDE.md`: confirmation is the default +for material unresolved choices; exploratory assumptions require explicit opt-in. Track decisions in +`IMPLEMENTATION_REPORT.md`, and preserve the blocking evaluation and execution gates in either mode. + - `meds-algorithm` is an installed dependency. Do not copy its implementations into this repository. Keep framework, batching, preprocessing-format and algorithm assumptions in the algorithm package. - Depend on the source's own package or executable, pinned to the reviewed revision, and write only contract @@ -18,6 +22,9 @@ Before editing, read `SPEC.md`, `README.md`, and `docs/IMPLEMENTATION_GUIDE.md` `PayloadMetadata`; core owns publication, measurements and reserved manifest/config/request files. - Keep object-oriented code minimalist and auditable: cohesive components, explicit dependencies, no unnecessary mutable globals/singletons, import-time computation or duplicated contract definitions. +- Keep the public API small: export the intended algorithm/configuration classes, use a single leading + underscore for implementation-only helpers/modules, and omit internals from package re-exports and `__all__`. + Preserve required public contract methods/hooks; follow the API guidance in `docs/IMPLEMENTATION_GUIDE.md`. - Test configuration/request serialization and step execution in a fresh process. Reopen fitted artifacts; do not rely on training having run on the same Python object. - Preserve the `{{ profiles | join(', ') }}` algorithm profile and established command argument names. diff --git a/template/IMPLEMENTATION_REPORT.md.jinja b/template/IMPLEMENTATION_REPORT.md.jinja index c8bb442..0f20dfc 100644 --- a/template/IMPLEMENTATION_REPORT.md.jinja +++ b/template/IMPLEMENTATION_REPORT.md.jinja @@ -5,6 +5,20 @@ - Paper/repository: {{ implementation_source if implementation_source else 'TODO' }} - Exact revision/version: TODO +## Scientific assumptions and decisions + +- Mode: confirmation required (default). If exploratory mode was explicitly enabled, record that instruction here. +- Unresolved blocking questions: TODO (or explicitly none). +- Provisional choices still requiring review before a faithful-reproduction claim: TODO (or explicitly none). + +Record choices before implementing dependent behavior. Statuses: routine default, awaiting confirmation, +provisional, confirmed, or resolved from source. Link evidence and explicit decisions; do not treat silence +as approval. Revisit affected code, tests and results when a decision changes. + +| Missing/conflicting detail | Sources consulted | Options and scientific consequences | Choice/rationale and status | User decision or source resolution | Affected code/tests/results | +|---|---|---|---|---|---| +| TODO | TODO | TODO | TODO | TODO | TODO | + ## Profile choices {% for profile in profiles %}### {{ profile }} diff --git a/template/docs/IMPLEMENTATION_GUIDE.md.jinja b/template/docs/IMPLEMENTATION_GUIDE.md.jinja index 292de3e..87015c4 100644 --- a/template/docs/IMPLEMENTATION_GUIDE.md.jinja +++ b/template/docs/IMPLEMENTATION_GUIDE.md.jinja @@ -8,6 +8,33 @@ Read the paper, supplement, repository, configs, data preparation, checkpoints, Build a source-element inventory before implementation; do not infer the algorithm solely from its headline architecture. +### Scientific ambiguities and decisions + +Default to confirmation-required mode unless the user explicitly enabled exploratory mode in the +implementation prompt or a later instruction. Consult the paper, supplements, official code and configs +before deciding a detail is missing. Record conflicting sources; do not silently choose one. + +A routine default (such as learning rate or batch size) may be chosen, made configurable where appropriate, +and documented unless it is central to the scientific claim. If reasonable interpretations would produce +materially different algorithms or change what evaluation measures, treat the choice as material. Missing +featurization, temporal aggregation, an architecture component or an objective usually meets this threshold. + +In confirmation-required mode, pause the dependent implementation and ask for a decision: state what is +missing, sources consulted, options, consequences and a recommendation if justified. Continue independent +work. Reuse an earlier explicit decision unless new evidence or scope invalidates it; silence is not approval. + +Explicit exploratory mode permits provisional algorithm-design choices. Record them before use, identify +affected code/tests/results as exploratory adaptations, and require review before claiming faithful +reproduction. It does not make assumptions source-backed facts. In both modes, unresolved leakage, +permitted label access, cohort/target definitions or evaluation validity still block dependent work. +Existing task constraints, acceptance tests and data/demo/full-data/cluster approval gates still apply. + +Maintain the mode and decisions in the existing `IMPLEMENTATION_REPORT.md` section **Scientific assumptions +and decisions**. Record evidence and rationale for routine defaults, open questions and provisional choices, +then append the user's decision or source resolution and update the affected implementation/results. + +### Reuse the source implementation + Prefer reuse over reimplementation. If the source ships an installable package or executable, add it as a dependency pinned to the exact reviewed revision and call it; write only contract glue (label-free preprocessing, cohort/key adaptation, explicit key association, coverage). Resolve dependencies in this order: @@ -73,6 +100,19 @@ Keep code minimalist and auditable: small cohesive classes, typed immutable conf dependencies and persisted fitted state. Avoid speculative abstractions, mutable global runtime state, hidden singletons, import-time computation and duplicated core behavior. Use the same runner from the CLI and `tests/e2e.py`; demonstrate independent execution after reopening serialized inputs. + +### Public API and internal helpers + +Expose the named algorithm and configuration classes for callers. Give implementation-only classes, +functions and modules a single leading underscore, such as `_FeatureEncoder` or `_features.py`, and keep +them out of package-level re-exports in `__init__.py`. Keep `__all__` limited to intentional public exports. +Preserve the required names of public profile methods and integration hooks. The underscore marks an +internal, unstable interface; Python does not enforce private access, and `__all__` does not prevent explicit +imports. Use double-underscore name mangling only for a specific inheritance collision, not as the default +privacy mechanism. Introduce helper classes only when their responsibilities justify a class. + +### Payloads and lifecycle independence + Keep shared preprocessing/pretraining task-independent. Prediction must receive its prepared data or embeddings explicitly and check compatibility with the fitted algorithm. Probe prediction applies the head; it does not rerun the encoder. Preserve standard external arguments. Choose and version intermediate payload formats, reject incompatible inputs before expensive work, let the runner publish with rollback, and diff --git a/tests/prompt_builder.test.cjs b/tests/prompt_builder.test.cjs index be855bc..09c2b13 100644 --- a/tests/prompt_builder.test.cjs +++ b/tests/prompt_builder.test.cjs @@ -27,17 +27,22 @@ function browser() { const output = doc.querySelector("[data-prompt-output]"); return {dom, doc, form, field, set, output}; } -test("all source, destination, profile, task and scope combinations preserve the shared contract", () => { +test("all source, destination, profile, task, scope and ambiguity modes preserve the shared contract", () => { for (const source_kind of ["paper", "code", "both"]) for (const destination_kind of ["local", "github"]) for (const profile_mode of ["automatic", "manual"]) for (const tasks_mode of ["none", "ready"]) - for (const scope of ["local", "integration", "full"]) { - const prompt = renderPrompt(template, {...base, source_kind, destination_kind, profile_mode, tasks_mode, scope}); + for (const scope of ["local", "integration", "full"]) + for (const exploratory_mode of [false, true]) { + const prompt = renderPrompt(template, {...base, source_kind, destination_kind, profile_mode, tasks_mode, scope, exploratory_mode}); assert(!/<[A-Z][A-Z0-9_]*>/.test(prompt)); assert(prompt.includes("Only after approval, run:")); assert(prompt.includes("minimalist, auditable, object-oriented")); assert(prompt.includes("Do not generate new tasks,")); + assert(prompt.includes("In BOTH modes, ambiguities about leakage")); + assert(prompt.includes("Require review of these decisions before claiming faithful reproduction.")); + assert.equal(prompt.includes("Exploratory mode explicitly enabled:"), exploratory_mode); + assert.equal(prompt.includes("Confirmation required (default):"), !exploratory_mode); assert.equal(prompt.includes("paper.pdf"), source_kind !== "code"); assert.equal(prompt.includes(base.code), source_kind !== "paper"); assert.equal(prompt.includes("New GitHub repository:"), destination_kind === "github"); @@ -45,6 +50,42 @@ test("all source, destination, profile, task and scope combinations preserve the assert.equal(prompt.includes("Prepared labels: /data/tasks/labels"), tasks_mode === "ready"); } }); +test("omitted ambiguity mode defaults to confirmation and non-boolean opt-ins are rejected", () => { + assert.equal(renderPrompt(template, base), renderPrompt(template, {...base, exploratory_mode: false})); + assert(renderPrompt(template, base).includes("Confirmation required (default):")); + for (const exploratory_mode of ["false", "true", "on", null, 0, 1]) { + assert.throws(() => renderPrompt(template, {...base, exploratory_mode}), /explicit checkbox/); + } +}); +test("exploratory checkbox is opt-in, invalidates the previous prompt and resets to confirmation", () => { + const {dom, doc, form, field, set, output} = browser(); + try { + const checkbox = field("exploratory_mode"); + assert.equal(checkbox.checked, false); + assert.equal(checkbox.defaultChecked, false); + assert(doc.getElementById(checkbox.getAttribute("aria-describedby"))); + set("papers", base.papers); set("code", base.code); set("local_path", base.local_path); + form.requestSubmit(); + assert(output.value.includes("Confirmation required (default):")); + const original = output.value; + checkbox.click(); + assert.equal(checkbox.checked, true); + assert(doc.querySelector("[data-copy]").disabled && doc.querySelector("[data-download]").disabled); + assert.equal(output.value, original); + form.requestSubmit(); + assert(output.value.includes("Exploratory mode explicitly enabled:")); + assert(!output.value.includes("Confirmation required (default):")); + assert(output.value.includes("Only after approval, run:")); + checkbox.click(); + form.requestSubmit(); + assert.equal(output.value, original); + checkbox.click(); + form.requestSubmit(); + doc.querySelector("[data-reset]").click(); + assert.equal(checkbox.checked, false); + assert.equal(output.value, ""); + } finally { dom.window.close(); } +}); test("incomplete or invalid plans fail instead of yielding an executable-looking prompt", () => { for (const bad of [ {papers: ""}, {code: ""}, {local_path: ""}, {source_kind: "unknown"},