Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,9 @@ optional task subset; or explicitly not supplied):
Work scope (implementation/local tests, also MEDS-DEV integration, or preparation for full evaluation):
<WORK_SCOPE>

Scientific ambiguity mode (confirmation required by default; exploratory only if explicitly enabled):
<SCIENTIFIC_AMBIGUITY_MODE>

Additional implementation constraints:
<ADDITIONAL_CONSTRAINTS_OR_NONE>

Expand All @@ -186,6 +189,32 @@ one always overrides the other. If only a paper is provided, distinguish source-
necessary assumptions. If only code is provided, inspect its revision, configuration and checkpoint
requirements; do not claim fidelity to an unavailable paper.

Resolve scientific ambiguities before implementing the affected behavior. Check the paper, supplements,
official code and available configurations first; record sources and material conflicts. Ask whether two
reasonable interpretations could produce materially different algorithms or change what evaluation measures.
A missing learning rate or batch size can usually use a plausible, configurable default with its rationale
recorded, unless that choice is central to the scientific claim. Missing feature construction, temporal
aggregation, architecture components or training objectives are material ambiguities, not routine defaults.

In confirmation-required mode (also the default if no mode is supplied), pause dependent implementation
and request confirmation for a material unresolved ambiguity. Explain what is missing, the evidence
consulted, plausible interpretations and their consequences; recommend an option when justified. Continue
independent work while waiting. An earlier explicit decision covering the same choice is sufficient unless
new evidence or a scope change invalidates it; silence is not confirmation.

Only when exploratory mode is explicitly enabled may you implement provisional interpretations of material
algorithm-design ambiguities without first obtaining a decision. Record each as provisional before using
it, expose the choice in configuration where appropriate, and label affected code, tests and results as an
exploratory adaptation. Require review of these decisions before claiming faithful reproduction. Do not
present an assumption as source-backed behavior or silently substitute a familiar method for a missing one.

In BOTH modes, ambiguities about leakage, permitted label access, cohort/target definitions or evaluation
validity remain blocking for dependent work until explicitly resolved. Exploratory mode does not authorize
changing supplied tasks, weakening acceptance tests, accessing data or running demo/full-data/cluster jobs
without the existing approvals. Record the mode, routine defaults, unresolved questions, provisional choices
and explicit decisions in IMPLEMENTATION_REPORT.md under "Scientific assumptions and decisions"; update the
existing report rather than creating another specification file.

For automatic profile selection, choose one or more profiles supported by documented source behavior:
supervised, finetune, probe, zero_shot_direct, zero_shot_materialized or predict_only. Record evidence for
each choice and proceed when the evidence is clear. Do not add speculative capabilities merely because
Expand Down Expand Up @@ -240,6 +269,13 @@ Write minimalist, auditable, object-oriented algorithm code. Put source-defined
in the algorithm-owned frozen <AlgorithmClass>Config class in config.py; validate values and prove
serialization round trips. Keep the implementation in <algorithm_slug>.py and expose the named algorithm
and configuration classes from the package, so callers can import them meaningfully.
Keep the public API intentional and small. Classes, functions and modules used only inside the implementation
should use a single leading underscore (for example _FeatureEncoder or _features.py) and should not be
re-exported from the package __init__.py. List only intended public exports in __all__. Keep contract-required
methods and integration hooks publicly named where required. A leading underscore signals an internal,
unstable interface in Python; it does not enforce access control. Prefer it over double-underscore name
mangling unless avoiding a specific inheritance collision. Do not create extra classes just to organize
private helpers, and do not hide the algorithm/configuration classes callers are expected to import.
For neural algorithms with matching semantics, extend core NNConfig rather than
duplicating shared capacity, learning_rate, batch_size, weight_decay and dropout settings. Define
source-backed defaults and the capacity-multiplier mapping (reference architecture, dimensions, rounding
Expand Down Expand Up @@ -419,6 +455,9 @@ and [shared-package assessment](https://github.com/McDermottHealthAI/MEDS_algori

### Unreleased

- Require confirmation for material scientific ambiguities by default; offer explicitly enabled exploratory
assumptions in the prompt builder while preserving evaluation and execution approval gates.

- Generate an algorithm release guide, Trusted Publishing workflow, and installed-distribution checks.
- Configure PyPI metadata separately from Python names and bundle portable run/profile YAMLs once at build time.

Expand Down
4 changes: 3 additions & 1 deletion TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,9 @@ builder uses plain JavaScript without runtime dependencies, network calls or per

| Test | What it checks | Why it exists |
|---|---|---|
| All choice combinations | Exercises all 72 source/destination/profile/task/scope combinations and common instructions. | Preserve the shared scientific requirements and review gates for every configuration. |
| All choice combinations | Exercises all 144 source/destination/profile/task/scope/ambiguity-mode combinations and common instructions. | Preserve scientific review and execution gates in both confirmation-required and exploratory modes. |
| Ambiguity-mode default and validation | Confirms omitted/false mode requires confirmation and rejects non-boolean opt-ins. | Avoid enabling exploratory assumptions through an absent or truthy string value. |
| Exploratory checkbox lifecycle | Checks unchecked defaults, explicit opt-in, stale-output invalidation, toggling back and reset in a DOM. | Require an intentional selection and restore confirmation mode on reset. |
| Invalid plans | Rejects missing sources, destination, manual profiles or prepared-bundle metadata. | Avoid an apparently ready prompt with incomplete required inputs. |
| Task subsets and literal text | Retains existing task identifiers and arbitrary input text without recursive replacement. | Keep the user's supplied scope and constraints intact. |
| Conditional form fields | Exercises visibility, required fields and multi-profile generation in a DOM. | Prevent hidden fields or single-profile defaults from overriding the selected options. |
Expand Down
7 changes: 7 additions & 0 deletions docs/assets/prompt-builder.js
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@
const mode = choice(values.profile_mode, ["automatic", "manual"], "profile selection");
const tasks = choice(values.tasks_mode, ["none", "ready"], "task-input option");
const scope = choice(values.scope, Object.keys(scopes), "work scope");
if (values.exploratory_mode !== undefined && typeof values.exploratory_mode !== "boolean") {
throw new Error("Exploratory mode must be an explicit checkbox selection.");
}
let repository = "New local Git repository only; do not create a GitHub repository.";
if (destination === "github") {
const name = required(values.github_repo, "the new GitHub repository")
Expand Down Expand Up @@ -61,6 +64,9 @@
DATASET_CONTEXT_OR_NOT_SUPPLIED: clean(values.dataset) || "Not supplied; use synthetic data for local implementation tests.",
PREPARED_TASK_BUNDLE_OR_NOT_SUPPLIED: bundle,
WORK_SCOPE: scopes[scope],
SCIENTIFIC_AMBIGUITY_MODE: values.exploratory_mode === true
? "Exploratory mode explicitly enabled: provisional algorithm-design assumptions are permitted under the policy below."
: "Confirmation required (default): ask before implementing material unresolved scientific choices.",
ADDITIONAL_CONSTRAINTS_OR_NONE: clean(values.constraints) || "None."
};
return template.replace(/<([A-Z][A-Z0-9_]*)>/g, (token, key) => {
Expand Down Expand Up @@ -112,6 +118,7 @@
const data = new FormData(form);
const values = Object.fromEntries(data.entries());
values.profiles = data.getAll("profiles");
values.exploratory_mode = field("exploratory_mode").checked;
try {
output.value = renderPrompt(template, values);
output.readOnly = false;
Expand Down
2 changes: 2 additions & 0 deletions docs/prompt-builder.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ You can edit the result before copying. Demo and full-data execution retain thei
<label>Requested endpoint
<select name="scope"><option value="local">Implementation and local tests</option><option value="integration">Also run MEDS-DEV integration</option><option value="full">Prepare for demo and full-data evaluation</option></select></label>
<p class="pb-hint">Preparing for full evaluation does not authorize a demo, dataset download or cluster run. The existing review gates still apply.</p>
<label class="pb-check"><input type="checkbox" name="exploratory_mode" aria-describedby="exploratory-mode-help"> Exploratory mode — allow provisional scientific assumptions</label>
<p class="pb-hint" id="exploratory-mode-help">Off by default. The agent normally asks before implementing a major unresolved scientific choice, such as unspecified feature construction; routine defaults may proceed with documentation. When enabled, provisional design choices must be recorded and reviewed before claiming faithful reproduction. Leakage, cohort/target definitions, evaluation validity and execution approvals still require resolution.</p>
<label>Additional constraints (optional) <span class="pb-hint">For example: algorithm name, hardware limits, checkpoint requirements or implementation priorities.</span>
<textarea name="constraints" rows="3"></textarea></label>
</fieldset>
Expand Down
7 changes: 7 additions & 0 deletions docs/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,13 @@ local repository or a new GitHub repository with an explicit local checkout and
Automatic profile selection must follow source evidence; manual selection may include several profiles.
The agent records the choice before generation, then implements every selected profile.

By default, the agent asks for confirmation before implementing a material unresolved scientific choice
(for example, unspecified featurization). Routine defaults such as a missing learning rate may proceed
with a documented rationale unless central to the paper's claim. The builder's **Exploratory mode**
checkbox is unchecked by default: enabling it permits provisional design assumptions, which must be
recorded and reviewed before claiming faithful reproduction. In both modes, unresolved leakage,
cohort/target definitions or evaluation validity block dependent work. Independent work can continue.

Evaluation tasks, when supplied, are already ready: labels plus task configs, bundle identity and dataset
context. The builder does not ask the agent to invent tasks or extract new labels. Missing tasks defer
only the corresponding evaluation. Choose the intended work scope explicitly; demo and full-data runs
Expand Down
7 changes: 7 additions & 0 deletions template/AGENTS.md.jinja
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
Before editing, read `SPEC.md`, `README.md`, and `docs/IMPLEMENTATION_GUIDE.md` completely.
`SPEC.md` points to the pinned core contract: run `meds-algorithm-spec` and read that specification too.

Follow the scientific-ambiguity policy in `docs/IMPLEMENTATION_GUIDE.md`: confirmation is the default
for material unresolved choices; exploratory assumptions require explicit opt-in. Track decisions in
`IMPLEMENTATION_REPORT.md`, and preserve the blocking evaluation and execution gates in either mode.

- `meds-algorithm` is an installed dependency. Do not copy its implementations into this repository.
Keep framework, batching, preprocessing-format and algorithm assumptions in the algorithm package.
- Depend on the source's own package or executable, pinned to the reviewed revision, and write only contract
Expand All @@ -18,6 +22,9 @@ Before editing, read `SPEC.md`, `README.md`, and `docs/IMPLEMENTATION_GUIDE.md`
`PayloadMetadata`; core owns publication, measurements and reserved manifest/config/request files.
- Keep object-oriented code minimalist and auditable: cohesive components, explicit dependencies,
no unnecessary mutable globals/singletons, import-time computation or duplicated contract definitions.
- Keep the public API small: export the intended algorithm/configuration classes, use a single leading
underscore for implementation-only helpers/modules, and omit internals from package re-exports and `__all__`.
Preserve required public contract methods/hooks; follow the API guidance in `docs/IMPLEMENTATION_GUIDE.md`.
- Test configuration/request serialization and step execution in a fresh process. Reopen fitted artifacts;
do not rely on training having run on the same Python object.
- Preserve the `{{ profiles | join(', ') }}` algorithm profile and established command argument names.
Expand Down
14 changes: 14 additions & 0 deletions template/IMPLEMENTATION_REPORT.md.jinja
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,20 @@
- Paper/repository: {{ implementation_source if implementation_source else 'TODO' }}
- Exact revision/version: TODO

## Scientific assumptions and decisions

- Mode: confirmation required (default). If exploratory mode was explicitly enabled, record that instruction here.
- Unresolved blocking questions: TODO (or explicitly none).
- Provisional choices still requiring review before a faithful-reproduction claim: TODO (or explicitly none).

Record choices before implementing dependent behavior. Statuses: routine default, awaiting confirmation,
provisional, confirmed, or resolved from source. Link evidence and explicit decisions; do not treat silence
as approval. Revisit affected code, tests and results when a decision changes.

| Missing/conflicting detail | Sources consulted | Options and scientific consequences | Choice/rationale and status | User decision or source resolution | Affected code/tests/results |
|---|---|---|---|---|---|
| TODO | TODO | TODO | TODO | TODO | TODO |

## Profile choices

{% for profile in profiles %}### {{ profile }}
Expand Down
40 changes: 40 additions & 0 deletions template/docs/IMPLEMENTATION_GUIDE.md.jinja
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,33 @@ Read the paper, supplement, repository, configs, data preparation, checkpoints,
Build a source-element inventory before implementation; do not infer the algorithm solely from its headline
architecture.

### Scientific ambiguities and decisions

Default to confirmation-required mode unless the user explicitly enabled exploratory mode in the
implementation prompt or a later instruction. Consult the paper, supplements, official code and configs
before deciding a detail is missing. Record conflicting sources; do not silently choose one.

A routine default (such as learning rate or batch size) may be chosen, made configurable where appropriate,
and documented unless it is central to the scientific claim. If reasonable interpretations would produce
materially different algorithms or change what evaluation measures, treat the choice as material. Missing
featurization, temporal aggregation, an architecture component or an objective usually meets this threshold.

In confirmation-required mode, pause the dependent implementation and ask for a decision: state what is
missing, sources consulted, options, consequences and a recommendation if justified. Continue independent
work. Reuse an earlier explicit decision unless new evidence or scope invalidates it; silence is not approval.

Explicit exploratory mode permits provisional algorithm-design choices. Record them before use, identify
affected code/tests/results as exploratory adaptations, and require review before claiming faithful
reproduction. It does not make assumptions source-backed facts. In both modes, unresolved leakage,
permitted label access, cohort/target definitions or evaluation validity still block dependent work.
Existing task constraints, acceptance tests and data/demo/full-data/cluster approval gates still apply.

Maintain the mode and decisions in the existing `IMPLEMENTATION_REPORT.md` section **Scientific assumptions
and decisions**. Record evidence and rationale for routine defaults, open questions and provisional choices,
then append the user's decision or source resolution and update the affected implementation/results.

### Reuse the source implementation

Prefer reuse over reimplementation. If the source ships an installable package or executable, add it as a
dependency pinned to the exact reviewed revision and call it; write only contract glue (label-free
preprocessing, cohort/key adaptation, explicit key association, coverage). Resolve dependencies in this order:
Expand Down Expand Up @@ -73,6 +100,19 @@ Keep code minimalist and auditable: small cohesive classes, typed immutable conf
dependencies and persisted fitted state. Avoid speculative abstractions, mutable global runtime state,
hidden singletons, import-time computation and duplicated core behavior. Use the same runner from the
CLI and `tests/e2e.py`; demonstrate independent execution after reopening serialized inputs.

### Public API and internal helpers

Expose the named algorithm and configuration classes for callers. Give implementation-only classes,
functions and modules a single leading underscore, such as `_FeatureEncoder` or `_features.py`, and keep
them out of package-level re-exports in `__init__.py`. Keep `__all__` limited to intentional public exports.
Preserve the required names of public profile methods and integration hooks. The underscore marks an
internal, unstable interface; Python does not enforce private access, and `__all__` does not prevent explicit
imports. Use double-underscore name mangling only for a specific inheritance collision, not as the default
privacy mechanism. Introduce helper classes only when their responsibilities justify a class.

### Payloads and lifecycle independence

Keep shared preprocessing/pretraining task-independent. Prediction must receive its prepared data or embeddings explicitly and check compatibility with
the fitted algorithm. Probe prediction applies the head; it does not rerun the encoder. Preserve standard external arguments. Choose and version
intermediate payload formats, reject incompatible inputs before expensive work, let the runner publish with rollback, and
Expand Down
Loading
Loading