feat(orchestrator): tell headless runs that no user is present

[rockymadden]

Problem

Task YAMLs carry hand-copied lines like "Do NOT ask for approval, confirmation, or feedback" to get autonomous behavior. That phrasing forbids asking. It does not say nobody is there to ask.

Those are different instructions, and an agent can honor the first while still stopping at a consent gate to wait for a reply that never arrives. That is not hypothetical: in the 2026-09-04 skill-flow-* nightly, 5 of 8 tasks built and validated a flow, reported success, and never executed it. Every one of those prompts contained the "Do NOT ask" line.

Measured across the uipath-maestro-flow suite (128 tasks):

	count
Tasks stating the run is headless	0
Non-simulated tasks	119
...saying nothing about autonomy	51
...saying something, across 8 wording variants	68
Silent and graded on a real run of the artifact	9

The instruction is hand-maintained, inconsistent, and missing from nearly half the suite.

Fix

The orchestrator already knows which case it is in. Simulated tasks are routed to _simulation_dialog_loop at orchestrator.py:1925 and never reach the prompt built in _evaluation_loop. The two paths are separate functions, so stating it once at that construction site covers every headless task, cannot reach a task that has a live user, and needs no conditional.

HEADLESS_PREAMBLE = (
    "This run is headless. No user is present, and nobody will answer a question "
    "or grant an approval. Do not wait for input: choose the best available option "
    "and record every decision and assumption in your final response. Stop only "
    "when proceeding would be unsafe or irreversible, and say what you stopped on. "
    "Instructions in the task below take precedence over this paragraph."
)

Ordering is cwd → preamble → task prompt.

Why no opt-out flag

The last sentence defers to the task. A task that deliberately asserts a refuse-and-surface still overrides it. skill-flow-eval-no-auto-upload is the live example: it instructs the agent to refuse an upload and surface the choice, and it keeps working without anyone remembering to set a flag.

I would rather find out that is too soft from a real run than add a config field on speculation.

Tests

Two, both in tests/test_orchestrator.py:

• test_evaluation_loop_prepends_headless_preamble — the preamble reaches the agent, and the ordering holds.
• test_simulated_task_bypasses_the_headless_prompt_path — a simulated task delegates to the dialog loop and never builds the headless prompt.

The second is the one that matters. The design rests entirely on the function boundary, so a refactor that merges the loops should fail loudly.

Verification

• ruff format --check / ruff check on src and tests: clean
• pyright on both changed files: 0 errors
• make lint (custom CE rules): 388 passed
• 450 orchestrator, run-limit, simulation, early-stop and error-handling tests pass

Downstream

UiPath/skills pins this via tests/.coder-eval-version (currently 0.11.5). Once this releases, that pin bumps and all 119 headless flow tasks get the declaration. The 68 hand-copied autonomy lines become redundant and can be swept separately.

[github-actions]

Claude finished @rockymadden's task in 1m 23s —— View job

---

Code Review in Progress

Todo List:

• Read code review guidelines (.github/code_review.md)
• Read project conventions (CLAUDE.md)
• Review full diff (git diff origin/main...HEAD)
• Read complete changed files for context
• Analyze implementation correctness
• Check test coverage and quality
• Perform cross-file consistency checks
• Analyze what's missing
• Provide comprehensive review feedback

[Copilot]

🟢 Approval recommended

The focused change is correctly isolated and covered by regression tests.

Pull request overview

Adds a standard headless-run preamble for autonomous, non-simulated evaluations.

Changes:

• Prepends the preamble after working-directory context.
• Keeps simulated-user dialogs unchanged.
• Adds tests for prompt ordering and simulation routing.

File summaries

File	Description
src/coder_eval/orchestrator.py	Adds and routes the headless preamble.
tests/test_orchestrator.py	Tests headless and simulated paths.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rockymadden]

Closed per thread feedback. Recording why, for anyone who finds this later.

The orchestrator is the wrong layer. experiments.yaml already carries defaults.agent.system_prompt, and the override chain is experiment.variant > task > experiment.defaults, so this is a config change rather than a code change.

Two things this PR got wrong:

Scale. I sized the simulated-task exposure from the uipath-maestro-flow suite alone (9 tasks) and assumed the function-boundary split in _evaluation_loop covered it. Repo-wide it is 319 simulated tasks against 973 non-simulated, with uipath-troubleshoot alone holding 297.

Layer. Even with the simulated path excluded, baking a behavioral policy into the orchestrator makes it unconfigurable per experiment. The config mechanism already existed and I did not look for it.

Replacement work moves to UiPath/skills: the headless declaration goes in tests/experiments/*.yaml, and the skill-side instruction for what to do when headless stays in UiPath/skills#3079.

One open problem for whoever picks up the split: no tag or path convention distinguishes the 319 simulated tasks, and they are not confined to a directory (uipath-maestro-flow keeps them under interactive/, uipath-troubleshoot does not). A separate experiment config for them needs either a new tag on 319 files or a selector in coder-eval run — the latter would land in this repo.