Repository navigation
Description
Current state: The scope of #513 is accepted as a follow-up to the November 19 owner decision in #467 and its merged implementation in #480. Implementation and its acceptance checks remain pending. Scope decision for #513.
Decision needed: Assign implementers to the accepted delivery slices.
Next action: @lirx-pd coordinates slice claims in this thread and records the implementers as they are assigned.
Goal and acceptance
- Problem / desired outcome: Treat profiling as supplemental evidence for a specific round. Per-round acceptance, ranking, parent selection, progression, and
finalfollow the campaign's existing functional prerequisites and performance and quality rules. Profiling has its own execution limits and consumes none of the original campaign budget. - Completion criteria:
-
Complete formal performance testing. Screening is optional and diagnostic. Whether it passes, fails its threshold, returns an error, or is absent, run and retain every configured formal scenario and repetition if the managed service remains valid and the required evaluation budget permits it. Functional checks and service readiness remain prerequisites.
Formal performance Quality Round verdict Inline profiling Valid evidence, gate failed Skipped FAILAttempt if configured and the same service is usable Gate passed Gate passed PASS, eligible for normal rankingAttempt if configured and the same service is usable Gate passed Valid evidence, gate failed FAILSkipped Required evidence missing, invalid, or interrupted Skipped INCONCLUSIVESkipped Gate passed Required evidence missing, invalid, or interrupted INCONCLUSIVESkipped -
Persist the complete evaluation before profiling. Before starting capture, durably save the formal performance samples or their retained evidence references, the aggregate results, the applicable quality result and score or recorded skip, and the gate assessment and verdict. Bind that checkpoint to the original attempt and its revision, effective configuration, conditions, service, and run. Track capture and cleanup separately. Once cleanup is verified,
recoversettles the checkpoint as recorded, including a performanceFAIL, without repeating performance or quality. -
Keep profiling out of every decision. Apply the same rule at round settlement, next-round admission, and
finaladmission and settlement. Capture availability, failure, timeout, interruption, or subsequent loss of a profile leaves evaluation, ranking, and selection governed by the retained performance and quality evidence. Verify the revision, effective configuration, conditions, original attempt binding, and required test evidence as before. Unverified cleanup blocks new service starts for both ordinary evaluation and profile-only capture, including after an interrupted backfill. -
Use a separate profiling time limit. Add an optional positive
sampling.profile_timeout_seconds, defaulting tosampling.stage_timeout_seconds. It bounds inline capture or, for standalone capture, startup, readiness, any required warmup, and capture together. Cleanup uses the existing separate bounded cleanup window. A timeout recordstimed_outin capture diagnostics and makes that capture unavailable. An operator interruption recordsinterrupted. Exclude time spent solely on profiling from the round and campaign evaluation budgets and preserve the time reserved forfinal, round count, and optimization usage. Persist the capture ID and start time before capture starts, and settle the excluded interval in a durable campaign accounting entry keyed by that ID. Repeated recovery applies each entry once. Preserveelapsed_secondsfor round wall time andwall_elapsed_secondsfor campaign wall time. Inline capture holds the campaign lock and adds wall time. External job and agent deadlines still apply as configured.- Apply the same budget accounting to admission and execution. Use the adjusted budget for
statusand its remaining-time fields,can_finalize, round andfinaladmission, and the internal deadlines used for evaluation and its stages. Keep the configuredfinalreserve at its original value. Wall-time fields continue to show actual elapsed time. - Recover only substantiated profiling time. If the process crashes before writing the end receipt or accounting entry, use a durable end receipt for the same capture when available. Otherwise, exclude only elapsed profiling-only time established by durable progress or completed-stage receipts for that capture. A start record alone supports a zero-second exclusion. Any unverified interval, including crash downtime and time spent waiting for manual recovery, remains charged to the original clock. The recovery timestamp and the configured timeout provide no evidence of actual capture duration. Finalize the adjustment once under the capture ID and preserve the original
finalreserve.
- Apply the same budget accounting to admission and execution. Use the adjusted budget for
-
Expose a backward-compatible round summary. Add
profilingto the CLI JSON and dashboard/report round output without a campaign schema version bump. Emit{"profiling":{"available":false}}when no usable capture exists. For a validated, readable capture, emitavailable: trueplus campaign-relativeresultandartifactspaths. The summary prefers the latest usable backfill and otherwise uses the usable inline capture. Derive legacy availability at read time from the same validation, includingfalsefor partial, invalid, or unreadable legacy evidence. Retain the original verdict. Keep failure details in the capture execution record and stage logs. -
Return profiling failures to the Agent. For failed, timed-out, or interrupted inline captures and backfills, return an Agent-facing command diagnostic with the failed stage, the available error details, and the capture log path when available. If a crash prevents the capture command from responding, return the recorded diagnostic during
recoverand identify any missing error details. Keep this diagnostic outside the persisted round'sprofilingobject. An inline capture failure alone leaves the round command's evaluation result and selection unchanged. Standalone capture reports its own result via the exit codes below. -
Provide a profile-only command for settled rounds. Proposed CLI
inferrsi campaign profile --session <campaign> --round <round-number>. Resolve the round number to its last settled attempt withPASSorFAIL, including after retries. A round with only historicalINCONCLUSIVEattempts is ineligible. Allow capture afterfinalor exhaustion of the original budget. Use the existing nonblocking campaign lock and fail immediately if another operation holds it or cleanup is unresolved. Verify the recorded Git revision, retained protocol and effective configuration, and compatible workload, model, precision, device topology, runtime, and timing scope. Reuse the existing service lifecycle. Campaigns may omit the profiler command. A standalone capture requires a usable profiler in the retained protocol. The command runs only capture and its prerequisites. Exit0when a new usable capture is published,2with a command-level diagnostic for an ineligible round, a busy or unresolved campaign, unreproducible inputs, or failed or timed-out capture, and130on operator interruption after cleanup is attempted. -
Publish backfill evidence separately. Give each backfill its own capture directory and service/run IDs, linked to the original attempt, revision, configuration, and conditions. Preserve the inline profile pointer and receipt under their original attempt/service binding, as well as the original measurements, observation, selection, and
final. After validation and verified cleanup, atomically replace a separate backfill index for that attempt under the campaign lock. CLI JSON and dashboards read the same index. Retain unpublished capture directories for diagnosis and recovery. Failed or interrupted backfills leave the previous published references usable. -
Add automated regression coverage. Exercise the verdict table, including screening rejection and script error followed by complete formal measurements. Verify timeout handling, cleanup guards for both commands, and fail-fast lock contention. Check both availability states in CLI JSON and the report, including legacy partial and unreadable evidence.
- Test recovery and decision evidence. Inject an interruption after the complete evaluation checkpoint is saved but before profiling begins, and again during capture. Cover a performance-gate
FAILand a combinedPASS, and verify that both settle from the retained results after cleanup without a new performance or quality run. A deleted or unreadable profile changes only availability across settlement, the next round, andfinal. Missing or invalid required candidate or evaluation evidence still blocks the corresponding decision. - Test backfill and historical records. Backfill a rejected round after the original deadline and an accepted, already-finalized round with a valid inline profile. Verify that CLI JSON and the report expose the separately attributed capture, with the original inline binding, selection, and
finalpreserved. Confirm the command targets the final settled attempt of a round with retries, leaves earlier attempts untouched, rejects anINCONCLUSIVE-only round, and preserves an existing usable profile on failure. - Test a crash before budget settlement. Crash after the start record is durable and before the end receipt or accounting entry is written. Delay recovery, then run
recovertwice after confirming cleanup. Test both a start-only record and one with a durable progress receipt. The first excludes zero seconds and the second excludes exactly the proven profiling duration. Assert that the downtime remains charged, there is only one settled entry for the capture, and the second recovery grants no extra budget orfinalreserve. - Test effective deadlines. Use a controlled clock to verify status, round and
finaladmission, and execution time limits when proven inline capture time crosses the original wall-clock deadline with evaluation budget still available. Verify that standalone capture remains outside the original budget and the configuredfinalreserve is identical in both cases. - Test Agent-facing diagnostics. Fail inline profiling after both gates pass and fail a standalone backfill. Assert that the Agent receives the error in both cases, the
profilingobject keeps its availability-only shape when no usable capture exists, and the original verdict and selection are preserved. Verify thatrecoverreturns the diagnostic for a capture interrupted by a crash. - Test
finalafter a condition change. Refresh both references and verify thatfinalremainsINCONCLUSIVEwithout a new accepted candidate, even if an older round has a usable backfill. A candidate that passes performance and quality under the new conditions satisfies admission without a profile.finalruns its independent evaluation without collecting a profile.
- Test recovery and decision evidence. Inject an interruption after the complete evaluation checkpoint is saved but before profiling begins, and again during capture. Cover a performance-gate
-
Scope and constraints
- In scope / out of scope: Applies to ordinary managed project campaigns using opt-in
service_reuse. The accepted task: Complete formal performance testing and decouple profiling in service-reuse campaigns #513 scope replaces the screening short-circuit and the performance-rejection profiling skip. It also supersedes the November 19 amendment's profiling-dependent verdict, pause, full retry, andfinalrequirements. The accepted fresh-candidate requirement remains in force. Refresh the references after a condition change and require a candidate accepted under the new conditions before a conclusivefinal. Under task: Complete formal performance testing and decouple profiling in service-reuse campaigns #513, performance and quality establish that acceptance. An old round's backfill remains evidence for its recorded conditions. AB/BA, benchmark sessions, gate thresholds, and reference refresh policy are outside the scope. - Relevant dependencies, shared interfaces or limits (if any): Use the existing campaign lock, readiness, cleanup, and artifact validation. A later change to the candidate revision, effective configuration, or test conditions requires its own evaluation before it can support a new acceptance.
Ownership and authorization
- Proposed by / actual requester: lirx-pd.
- Status: Accepted (scope). Decision recorded by Dong-bot on behalf of @Dong1017.
- Accountable parent (Issue-level owner, if assigned): @lirx-pd coordinates delivery.
- Slice implementer(s) / acting contributor (if assigned): @lirx-pd is the acting contributor and coordinator. The three implementation slices remain unassigned pending claims in this thread.
- Goal-acceptance owner: @Dong1017 for the repository decision, with lirx-pd confirming the requested behavior.
- Accepted decision or scoped delegation: The #513 scope decision accepts the completion criteria above and appoints @lirx-pd as accountable parent. The decision supersedes only the provisions identified above in the November 19 owner decision in #467.
Inputs and deliverables
- Relevant sources, revisions or linked issues:
src/inferrsi/harness/project_service.py,src/inferrsi/harness/optimization.py, andtests/test_project_service_reuse.py. Parent issue #467 and implementation PR #480. - Expected artifacts: Managed campaign changes, round output and profile-only CLI documentation, and automated regression results.
Completion and closure
- Who accepts completion and may close: The goal-acceptance owners confirm the criteria above. The repository owner confirms closure authority.
- Remaining uncertainty / next decision: Record implementers as they claim the verdict and recovery, output and budget, and backfill CLI slices. Each slice carries its own regression tests. Completion requires implementation evidence against the accepted criteria.
Activity
Dong1017 commentedon Nov 19, 2026
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
Assessment: The proposal is directionally sound and mostly well specified. It makes profiling supplemental evidence instead of an acceptance gate, and it adds a backfill path for already settled rounds. Before anyone accepts it or delegates implementation, I'd settle the scope boundary and several underspecified points below. I read only the issue text. I did not check
project_service.py,optimization.py, #467 or #480 against it.What it asks
final.profiling.availableon every round.inferrsi campaign profile --session <campaign> --round <n>to capture a profile for a settled round afterfinalor after the budget is exhausted.Scope concerns (should be settled before acceptance)
FAIL,INCONCLUSIVE, qualityFAILand inline profiling in one checkbox. A verdict/profiling decision table would be easier to implement and test: screening result × formal perf × quality → verdict, and whether inline profiling runs.Gaps and risks
final. Excluding it from the budget is fine, but the doc should confirm that external or session deadlines are not affected. "Report actual elapsed time" needs a concrete field name.{"available":false}carries no reason. The proposal should say where operators find the cause, such as logs or a separate diagnostic artifact, so that debugging isn't lost.Test additions I'd suggest
Ownership and process
The parent owner and implementer are still TBD, and the acceptance status is "Proposed".
For @Dong1017: the issue asks for scope confirmation and implementer delegation. The decision hinges on the supersession scope above. I'd accept the profiling-decoupling and backfill parts as written. The screening and perf-fail semantic change should either be explicitly included in the supersession or split out.
Suggested slicing for delegation:
profilingoutput object, legacy derivation and budget exclusion.campaign profileCLI, including atomic publish and recovery.Each slice would carry its own regression tests.
For lirx-pd: please answer the open questions above (window default and config, lock behavior, exit codes, output formats) in the body, then tag the decision owner for acceptance.
Dong-bot (authorized delegate).
david6666666 commentedon Nov 19, 2026
Please use #467’s November 19 amendment as the baseline, including the already-accepted fresh candidate requirement after a condition change. For backfill, please preserve the original inline profile’s attempt/service binding separately from the new capture’s binding. The current verifier requires an inline profile to match the original attempt and service, so replacing that pointer cannot faithfully describe a new service. An acceptance test could backfill an already-finalized round that has a valid inline profile, then assert the old final, selection, and inline receipt are unchanged while the CLI/report expose the separately attributed capture.
zengchuang-hw commentedon Nov 19, 2026
PASS(方案方向检视,不代表实现验收或范围授权)
支持把 profiling 从性能/质量结论中解耦,并允许对已结算轮次单独补采。静态核对了当前 main
9b418d7fe16f3979b67f74f290949629ef103e8f的project_service.py、optimization.py及相关测试。建议把以下具体边界写进实现和回归验收:采样前的持久化检查点必须包含完整测试结论,而不只是 PASS 标记。 目前 profile 调用 先于最终 PASS,
measurements、quality_score又到 函数末尾 才统一写入;recover 会将仍为 running 的执行改判 INCONCLUSIVE。新检查点应同时保存正式样本/汇总、适用的质量结果、结论和原 attempt 绑定,单独保留 profiling 与 cleanup 状态。测试应在“结论落盘后、profile 启动前”和“采样中”分别中断;确认清理后可原样结算,而不是重新测量或丢失性能 FAIL。只解除 profile 对选择的约束,保留其他证据完整性约束。 当前 final 证据检查、结算 和 下一轮准入 都会检查 profile。需要一起调整,不能只修改执行阶段。已有 profile/configuration 损坏测试 应拆开预期:profile 缺失只改变 availability;候选版本、有效配置、测试条件或原测试证据不可核实时,仍不能据此放行。
预算排除要作用于所有实际准入和执行时限。 当前 status/final、启动准入 依赖绝对 deadline,阶段执行 另有 evaluation timeout。仅在报表减去 profile 时长仍可能耗尽下一轮/final。建议持久化每次 capture 的可排除区间和结算标识,回归连续 recover 两次,证明不会重复返还预算;同时明确中断后的停机/人工处理时间如何计费,不能一并误算成 profiling。
补采不能追溯改判旧轮次。 对旧规则下因 profile 失败而记录为 INCONCLUSIVE 的 attempt,即使后来补采成功,也应按本提案“preserve recorded verdict”保持原结论与资格;同样不能满足新条件下的新候选验收要求。补充一个有多次 retry 的轮次用例,确认
--round解析到规定的最终 attempt,不修改早期 attempt、已完成 final、排名或计数。另请更新授权段落对 #467 的历史描述:当前有效基线是 11 月 19 日 owner 修订,其中 profile 失败为 INCONCLUSIVE。本提案仍是待 owner 接受的规则变更。
本次只做静态方案与代码核对,未运行测试、服务或采样,也未以 CI 结果作依据。
lirx-pd commentedon Nov 19, 2026
Thanks for the reviews. I updated the issue to include the screening change and profiling after a valid performance rejection in the proposed scope, and added a verdict table. The November 19 amendment to #467 is the baseline. It requires a fresh accepted candidate after a condition change. Under this follow-up, passing performance and quality in the new conditions satisfies that requirement, and an old round's backfill applies only to its recorded conditions. Once #513 is accepted, I'll cross-link the decision from #467.
The draft now proposes
sampling.profile_timeout_seconds, defaulting tosampling.stage_timeout_seconds, and spells out timeout, interruption, fail-fast lock behavior, exit codes, and where diagnostics live. Profiling time is excluded from the internal evaluation budget. Wall-clock reporting and external deadlines still reflect the actual elapsed time. CLI JSON and the report derive the additive availability field from validated evidence, including for older rounds.Each backfill has its own service and run IDs and publishes through a separate index. The original inline receipt and binding, selection, and
finalremain intact. The regression checklist includes an already-finalized accepted round with a valid inline profile, a rejected round after the deadline, lock contention, legacy records, timeouts, and budget and cleanup recovery.zengchuang-hw commentedon Nov 19, 2026
PASS(增量方案检视)
已核对这次正文更新和作者说明。相对我的上一轮意见,现在已明确判定表、独立 profiling 时间窗、按 capture ID 幂等记账、仅定位最后一个已结算的 PASS/FAIL attempt,以及单独发布 backfill index。历史 verdict、inline 绑定和已完成 final 保持不变,新条件仍要求新的候选验收,这些边界一致。
补充一个预算恢复验收点:开始时间落盘后进程崩溃、尚未写入结束时间或记账记录。不能直接把恢复时刻减去开始时刻全部算作 profiling,否则停机/人工处理时间也会被返还。请明确缺失结束凭据时如何确定可排除区间,并加入“崩溃后延迟恢复,再连续 recover 两次”的用例,验证停机时间未被误排除、同一 capture 只记账一次、final reserve 未被重复增加。这里只是落实既定预算边界的实现建议,不阻塞方案方向。
本轮为静态增量设计检视,未运行代码或测试、未用 CI 作依据;实现与范围接受仍待指定 owner 决定。
[-]task: Make profiling optional and backfillable in service-reuse campaigns[/-][+]task: Complete formal performance testing and decouple profiling in service-reuse campaigns[/+]lirx-pd commentedon Nov 19, 2026
Thanks for the follow-up. I added regression cases for interruptions both before profiling starts and during capture. After verified cleanup, recovery must reuse the complete evaluation checkpoint and preserve either a PASS or a performance FAIL. The existing checks for candidate identity, test conditions, and required evidence still apply to later rounds and final.
If a crash leaves no durable end receipt, recovery excludes only the profiling time supported by records for that capture. A start record alone excludes zero seconds. The test delays recovery and runs recover twice to verify that downtime remains charged, the adjustment is applied once, and the original final reserve is preserved. The adjusted budget applies to admission and execution. Profiling errors are returned to the Agent through the command or recover, while the round summary keeps only availability and artifact paths.
The backfill tests now cover round selection after retries and preservation of the original inline binding, historical verdicts, and completed final results. @Dong1017, could you confirm the revised scope and assign an implementer? The fresh-candidate requirement after a condition change stays in place under the November 19 decision in #467. Once #513 is accepted, I’ll link the decision from #467.
Dong1017 commentedon Nov 19, 2026
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
Accepted. Recording this as the scope decision for #513, with the revised body and thread as the accepted text. This is a design/scope acceptance only, not an implementation acceptance. I did not verify it against the code or run any tests. (Dong-bot, authorized delegate.)
Clauses accepted
recoversettles from the checkpoint, including a performanceFAIL, without re-measuring.final. Candidate revision, effective configuration, conditions, and required test evidence are still verified. Unverified cleanup blocks new service starts for both ordinary evaluation and profile-only capture.sampling.profile_timeout_secondsdefaults tosampling.stage_timeout_seconds. Profiling-only time is excluded from the round and campaign budgets through a durable accounting entry keyed by capture ID.finalreserve is unchanged.profilingobject is additive, with no schema bump. Availability for legacy rounds is derived at read time.recoverafter a crash. They stay outside the persistedprofilingobject.inferrsi campaign profile --session <campaign> --round <n>: A profile-only command for settled rounds.PASSorFAILattempt, and rounds with onlyINCONCLUSIVEattempts are ineligible.0,2and130, as specified.finalare unchanged, and a failed backfill leaves the previous references usable.Supersession
finalrequirements.Implementation assignment
campaign profilecommand and publication of backfill evidence.Non-blocking next steps
6998139373as the owner decision, and david6666666 cites6998135348as the amendment. Make sure the pointer targets the intended one.zengchuang-hw commentedon Nov 19, 2026
PASS
增量设计检视,不代表实现验收。
上一轮指出的崩溃记账问题已解决:只有开始记录时排除零秒,只扣除同一 capture 的持久化结束或进度凭据能证明的 profiling 时长;停机时间仍计入预算。延迟恢复并连续 recover 两次的回归能同时检验停机时间、幂等记账和 final reserve。
我也对照了 #480 的现有实现:原流程在 profiling 后才持久化 PASS,恢复可能把缺少完整结算记录的轮次判为 INCONCLUSIVE。现在要求先持久化完整评估,以核实清理为前提,恢复时直接结算原 PASS 或性能 FAIL,不重跑测量;调整后的预算同时约束准入和执行。重试后只定位最近已结算的 PASS/FAIL,backfill 不改写原 inline 绑定、历史 verdict 或 final。这些要求覆盖了本次变更的关键边界,无新增设计阻塞。
范围决定已记录,正文状态和分片负责人待同步。未查 CI、未运行代码或测试。