Repository navigation
feat(windows): MSI self-update via stable bootstrap and SYSTEM updater - #252
Open
swarit-stepsecurity wants to merge 99 commits into
Open
swarit-stepsecurity wants to merge 99 commits into
swarit-stepsecurity wants to merge 99 commits into
Conversation
Point(name) is an inlinable no-op unless built with selfupdate_faults. Tagged builds act on DMG_FAULT (merged with the link-time Baked value): kill, err, hang, diskfull, exit<N>, with optional @user/@System qualifiers. DMG_FAULT_TRACE records every checkpoint reached.
…ecords Layout paths rooted at an install root and a data root; strict x.y.z versions; schema-1 state.json with corrupt-as-missing Load and atomic Save (temp + fsync + MoveFileEx REPLACE_EXISTING|WRITE_THROUGH, or rename + dir fsync) with <prefix>.state-temp/-renamed fault checkpoints; non-blocking LockFileEx/flock update lock; run records capped at 10 with bad-run classification. Kill-at-checkpoint subprocess tests and fuzzers.
Windows: create the directory with, or re-assert, owner SYSTEM and a protected DACL (SYSTEM F, Administrators F, Users RX, inherited by children) via native security APIs, reporting whether drift was fixed. Rejects files and reparse points. Other platforms only create the dir.
…s and version manifest Port the latest-binary client, GitHub-then-mirror download and SSHSIG verifier from step-security#217 for the MSI updater: native arch via IsWow64Process2, env then WinHTTP machine proxy, strict x.y.z targets, signatures checked before download and sha256 after. Version dirs get a manifest so staged versions can be re-verified.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The MSI installs the launcher a second time as stepsecurity-dev-machine-guard-bootstrap.exe. Under that name it resolves the self-update state.json pointer (current -> previous -> seed), launches versions\<v>\task.exe with run-record env, picks the SYSTEM updater with a seed fallback after 3 consecutive failures, and adopts the MSI seed into versions\ without ever failing the MSI. No network or crypto code. The launcher exits 70 when started by the bootstrap and unable to start the agent, and falls back to a sibling agent.exe in version dirs. updatestate gains the frozen pointer reader, the run-chain resolver and a field-preserving state patch.
…ner with our own WinVerifyTrust (revocation off) on Windows plus a portable PKCS#7 read of the PE signature for the leaf subject and countersignature timestamp. SamePublisher pins candidates to the running updater's own signer subject; an unsigned dev build refuses updates unless a selfupdate_testkeys build pins a subject.
On an MSI bootstrap install, install points both scan tasks at `bootstrap.exe run send-telemetry` and (re)creates the SYSTEM task "StepSecurity Dev Machine Guard (Update)" from a full task XML: hourly with a 10 min random delay plus at boot after 15 min, StartWhenAvailable, battery allowed, 30 min limit, IgnoreNew. The registration is verified. Uninstall of a bootstrap install deletes all three tasks, waits up to 60 s for the scan mutex, terminates DMG processes under the install root, and removes versions\, state\ and RunningVersion. Non-bootstrap uninstalls keep their behaviour and never touch the update task.
Product.wxs installs stepsecurity-dev-machine-guard-bootstrap.exe and
seed\<version>\{agent.exe, task.exe, dmg-linux} in place of the top-level
agent and launcher. A deferred SYSTEM adopt-seed CA (Return=ignore) runs
on install, upgrade and repair; the install CA now runs on repair too
(M-7); the uninstall CA is skipped during a major upgrade and can't block
removal (M-6, M-19).
AUTOUPDATE (default 1) is passed to configure, which gains
--auto-update=<bool> and writes auto_update into the machine config.
msi-smoke asserts the layout, the versions/state ACLs, all three tasks,
a bootstrap run, a real SYSTEM update tick, repair, an N to N+1 upgrade
without downgrade, and a clean uninstall.
One tick of S0-S7: lock, sweep and GC (tolerating in-use dirs), ACL re-assert, probation health with rollback and quarantine (permanent, or seven days for an inconclusive probation), target check every 6 h or when a probation closes, verified staging into versions\<v>, SYSTEM smoke test, mutex-guarded switch with deferral reporting. Events are written to a state.json outbox in the same atomic write as the change they report and retried until the backend takes them.
Update events go to run-status as status=update with retries; 4xx drops the event, anything else stays in the updater's outbox. The SYSTEM updater resolves the device id from the hardware and rewrites the device-id file. Scans mark the bootstrap's run record once run-status started is accepted and report update_pointer_invalid when the bootstrap fell back. Adds the agent.prestart checkpoint, configure --auto-update and the auto_update/update_mirror_url/lag/cooldown config fields.
Overlapping launches (hourly and logon tasks on wake) updated whichever record was newest, so the run that scanned could stay open forever and another run could take its verdict. Records now carry the bootstrap's run_id and MarkRunStarted/FinalizeRun match it; the agent reads STEPSEC_RUN_ID.
The previous updater runs during every probation and rewrote state.json from its own struct, erasing whatever a newer version had added. State, probation, quarantine entries and updater now carry unknown keys through Load and Save verbatim.
A valid state that omitted or nulled them loaded with nil maps, so the first write into them would panic.
An unknown @qualifier, an unknown action or a colon-bearing checkpoint without an action left the rule silently unmatched, so a typo in a fault test passed without injecting anything. Only in selfupdate_faults builds.
…ndows CreateFile without FILE_FLAG_BACKUP_SEMANTICS fails on a directory with access denied, so Load returned an I/O error instead of corrupt and the updater skipped its rebuild path.
…vent fields Each update event keeps the execution_id minted when it was queued, so only its own retries reuse it. Events agent-api would refuse (no install_channel, rollback without update_from) are not posted.
A clean run of each scenario (update, rollback) under DMG_FAULT_TRACE lists the checkpoints of every tick; each traced checkpoint is then hit with kill, err, hang and diskfull in a subprocess, followed by clean ticks, asserting convergence and its event, hygiene, no flip-flop and the frozen bootstrap read against an in-process fake API. A state corruption matrix covers the updater side of the bootstrap cases. The matrix found that a rolled-back version's dir survived until the next tick, so GC now also runs right after a rollback. Runs on Linux and windows-latest.
A second os=linux call could resolve a different version if the release gate moved between calls. dmg-linux now uses linux_checksum and signed_linux_checksum from the windows response; missing fields fail verification.
… backend During an outage the primary updater exits 1 three times and the fallback would then report a broken updater that was never broken. The fallback now reports it only on a tick where its own latest-binary call succeeded.
…/win-msi-self-update-boot Brings in the core updater (C1-C4) so the bootstrap builds against its updatestate API. Conflicts in cli/config (--auto-update) and tests.yml resolved in favour of core; the bootstrap is adapted in the merge so it compiles: release file names (updatestate.AgentFile/LauncherFile), run records keyed by run_id (STEPSEC_RUN_ID), unknown-field-preserving Save instead of the local patch helper, and a manifest for the adopted seed.
…te=false The seed holds stepsecurity-dev-machine-guard.exe, stepsecurity-dev-machine-guard-task.exe and dmg-linux, the same names as versions\<v>\. AUTOUPDATE=0 runs configure --auto-update=false on install, upgrade and repair; the default writes nothing. msi-smoke waits only for msiexec itself with a bound, dumps the log and DMG processes on a hang, checks the adopted seed's manifest, and uploads logs when cancelled too.
…ate specs in drivers A malformed DMG_FAULT/Baked panicked with exit 2, the same code the exit2 action and a real Go panic use, so a typo could pass for a crashing version (a Bad run). It now exits 64 after writing 'faultinject: malformed spec:' to stderr. Validate (all builds) lets test drivers reject a spec before launching; the fault drivers and the kill matrix use it and fail on the marker.
A record with no run_id, or one already in the file, could never be matched by its own launch and would stay open (or be closed by another).
Parallel cases named their Windows mutex from the clock; two created in one clock tick shared it, so one saw a held mutex and deferred its switch.
…heck The case cleared the signature first, so the empty-signature check rejected it and the sha256 shape check was never exercised.
A mirror or release URL answering 302 to http:// was followed and the asset fetched in cleartext. Every updater request (latest-binary, GitHub, mirror, events) now fails on a downgrade redirect.
ProxyFromEnvironment returns nil both for 'no env proxy' and for a host excluded by NO_PROXY, and both fell through to the machine proxy. An env proxy config now decides alone; without one, NO_PROXY hosts go direct before the machine proxy and its bypass list apply.
…mment AUTOUPDATE=0 keeps the current version rather than the MSI's; the MSI does touch versions\/state\ via adopt-seed and uninstall; describe event delivery, the exact health rules, the version floor and device identity; list the AUTOUPDATE values configure accepts; drop the claim that a rolled-back upgrade keeps tasks and state, and the claim that upgrades never touch config.json; remove an internal reference from a test.
…n slow WMI The SYSTEM self-check probed WMI for up to 60 s inside a 60 s smoke limit, so on devices with a wedged WMI every new target timed out and was quarantined for good. The self-check's probe is now bounded at 10 s (the update tick keeps 60 s) and falls through to the SYSTEM cache, the smoke limit keeps a tested margin above it, and a smoke timeout is retried: only the third consecutive timeout for a version (also for the end-of-probation re-smoke) counts as a failure.
telemetry.Run recovers a panic and returned it as an ordinary error, so the agent exited 1: a crashing version counted Healthy (panic after run-status started) or Neutral (before it) and was never rolled back. A recovered panic now wraps ErrPanic and the agent exits 2, after still posting run-status failed. Adds a 'panic' fault action (a real Go panic) with scan checkpoints before and after the started marker, and the kill matrix now injects real panics into the updater too.
…fter sleep as neutral Go's monotonic clock on Windows includes sleep, so a laptop closed mid-scan for longer than the deadline was killed by the agent's and the bootstrap's watchdogs on resume, and the run counted as a crash. Both now measure awake time (QueryUnbiasedInterruptTime), and a deadline kill during which the system slept exits/finalizes with ExitSlept (75), which the health check treats as Neutral.
… has it The SYSTEM cache records its source; a configured id outlived its removal from config.json and kept being reported. Only a hardware-sourced cache entry is used when no id is configured.
A logoff or shutdown that ended the agent before run-status started (exit 130, STATUS_CONTROL_C_EXIT 0xC000013A, DBG_TERMINATE_PROCESS 0x40010004) counted as a crash. These are now Neutral; the docs list the exact set alongside the other classification rules.
A backward clock correction between the switch and a run made the run's timestamp fall before the probation start and the run was dropped. The bootstrap's AppendRun now assigns each record a per-file sequence number (max+1, kept across trimming), the switch records the file's current max as probation.since_seq, and verdicts count records with a higher seq. Records or probations without one keep the time rule; the 72 h / tick-count window is unchanged.
adopt-seed gives up after a bounded wait when an update tick holds the lock (the MSI must never block), so a newer MSI-pushed seed stayed unadopted until the next repair, with no event. Each updater tick now adopts it under the lock when no probation is open and the seed isn't quarantined, reporting update_ok with error_message seed_adopted. The seed copy and the pointer rule moved to updatestate and are shared with adopt-seed.
… at exit A -race binary sleeps 1 s on exit (GORACE atexit_sleep_ms defaults to 1000), and every matrix tick starts two of them (the updater and its smoke self-check), so each tick cost ~2 s of sleep: ~1900 s over 300 cases, which a 2-4 core CI runner couldn't finish inside the 10 minute test timeout once D24 added the panic cases. Children now run with atexit_sleep_ms=0 (a clean tick: 2.07 s -> 0.06 s; the package: 190 s -> 52 s locally). No product code waited on anything. Tick durations are now logged with each tick.
…ailure A latest-binary response with a valid version but no signed_checksum, or no launcher fields, was reported as update_check_failed; the update path treats an unverifiable release as update_verify_failed. Missing fields now reach VerifySignatures, which rejects them before any download.
The watchdogs already measure awake time only, so a kill is a real hang; e165fa5's 'slept' exemption (exit 75, Neutral) turned genuine hangs into neutral runs, and a crash code after the started marker read Healthy. Exit 75 is gone, a deadline kill is always a watchdog kill, and kill and crash codes (watchdog, 2, 70) are judged before the started marker.
A failed major upgrade (1603) rolled the MSI back but not adopt-seed's state write, leaving current on a version no MSI owned. adopt-seed now snapshots state.json (and whether it created versions\<seed>) in state\ before changing anything; a rollback custom action, sequenced just before it, restores the snapshot through the preserving Save (only if nobody wrote state since) and removes the seed dir it created unless the restored state points at it; a commit action drops the snapshot on success, and stale snapshots older than 1 h are swept by adopt-seed and the updater. msi-smoke forces a late failure on an N+1 upgrade (a test-only FailWhenDeferred hook) and asserts current, versions, AgentVersion, tasks and a scan.
Seed adoption (adopt-seed and the updater's D25 path, shared code) now checks the seed's agent and launcher against the running binary's publisher (an unsigned dev build has none to compare), hashes the source files before copying and requires the copy to match them (hash and size), and the updater skips adoption while Windows Installer holds Global\_MSIExecute, retrying next tick.
Blocks at the checkpoint for the given Go duration (up to 10m, enforced by Validate) and then continues normally. S6.mutex fires with the scan instance mutex held, so S6.mutex:sleep<d> keeps a scan out for d, which the harness needs to start a scan during the switch.
A terminated process keeps its image file in use until the kernel tears it down; WiX CloseApplication's terminate-and-continue let the MSI copy the bootstrap ~10 ms later while it was still held, which needs a reboot. New bootstrap verb stop-for-msi stops every process under the install root and waits (bounded) until each MSI-owned file (the bootstrap and seed\) opens for exclusive write. adopt-seed keeps a copy of the bootstrap in state\msi\ for the MSI to run it from (not an MSI file). The uninstall path's terminateUnder now waits the same way.
REBOOT=ReallySuppress in the Property table (msiexec /f repairs ignore /norestart and all command-line properties), and a first deferred action that runs stop-for-msi from state\msi\ before RemoveExistingProducts / file actions, replacing util:CloseApplication. msi-smoke repairs (both /i REINSTALL=ALL and plain /fa) while a hung bootstrap holds its image and asserts exit 0, no file held in use, no pending renames, no reboot, current and tasks intact. Docs: /f repairs ignore properties; the recommended repair command.
RemoveExistingProducts scheduled afterInstallInitialize must directly follow InstallInitialize (error 2613 failed the install). stop-for-msi now runs right after it, still before RemoveFiles and InstallFiles.
The check matched entries earlier steps had left (the forced-failure rollback runs just before). Compare against a baseline taken before each repair and print the entries.
… from A damaged current was restaged on every due tick, so an AV that kept deleting a file caused a download every tick. After 3 restages of one version within 24 h, restaging it pauses for 24 h and one update_stage_failed (av_deleted) is reported; the bootstrap keeps running the fallback and the version is not quarantined. The count resets once the version stays intact for 24 h.
The install action starts a scan, so a bootstrap could be running when a failed upgrade rolled back and restored the old bootstrap over it: an in-use file, parked as *.tmp for deletion at reboot. A rollback-phase stop-for-msi, sequenced last so it runs first in rollback, stops DMG processes and waits for the files before anything is restored. msi-smoke now asserts the forced-failure upgrade leaves no new pending renames and no file held in use.
…elease that ships it
swarit-stepsecurity
marked this pull request as ready for review
October 20, 2026 20:38
Member
Author
|
Ready for review. Pre-merge checklist:
Testing summary: unit + trace-driven kill matrix (Linux + windows-latest), msi-smoke (install/repair/upgrade/forced-failure rollback/uninstall, no-reboot assertions), and the real-Windows upgrade-safety catalogue in integration-test (all shards green, nothing masked). |
Member
Author
|
Additional release prerequisite: the updater verifies |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Windows MSI installs (Intune, SCCM) now keep themselves on the release the tenant selects. Today they only move when IT pushes a new MSI, so fixes reach them last.
state.jsonand runsversions\<current>\, falling back to previous, then to the seed.agent.exe self-update --systemfrom the confirmed version. Update logic lives in the versioned agent, so updater bugs are fixed by self-update too. A broken updater falls back to the previous version's or the seed's updater.Layout
versions\andstate\are SYSTEM-owned with a protected ACL, re-applied on every update run. The updater never touches the MSI's files (bootstrap, seed,AgentVersion); the MSI touchesversions\/state\only throughadopt-seed(copy the seed in, movecurrentforward, never back) and a real uninstall (remove both).AgentVersion(the Intune detection value) stays the MSI version.Safety properties
.staging-*; only a verified, manifest-stamped directory becomesversions\<v>.state.jsonis written atomically with a generation counter, unknown fields are preserved for older and newer readers, and every event is stored in the same write as the change it reports.update_*events go torun-status(status: "update"). Each is saved with the state change it reports; a validation reject (400/422) is dropped, any other failure is retried with a 1 h → 24 h backoff, and the outbox keeps up to 64 events (oldest dropped and counted).current; uninstall removes all three tasks,versions\,state\andRunningVersion.Testing
previous, skipping the re-read after a failed write, dedupe ignoring the event class) turn it red.AUTOUPDATEhandling, and uninstall cleanup.Rollout notes
minWindowsSelfUpdateVersionis set to 1.18.0 (the first release with this updater). A target below it is refused, because it would have no updater.AUTOUPDATE=0: stops taking new targets; the device stays on its current version (possibly a self-updated one) and reportsupdate_disabledwith the target it isn't taking.docs/windows-self-update.mdandpackaging/windows/README.mddescribe the design,AUTOUPDATEand the Intune detection rule.