Skip to content

Support for instance 'graceful' shutdown policy - #11565

Open
lifning wants to merge 34 commits into
mainfrom
lif/acpi-shutdown
Open

lifning wants to merge 34 commits into
mainfrom
lif/acpi-shutdown

Conversation

@lifning

@lifning lifning commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

This feature is intended to allow VM guest shutdowns to be triggered through the API and as part of sled evacuations for updates. An intent of this feature is to afford non-ephemeral instances a chance to cleanly unmount their filesystems, such that taking a new Oxide software release does not carry with it an unnecessary risk of filesystem corruption.

To quote RFD 622:

Oxide users must manually access each instance to perform a stop, which is very inconvenient when dealing with large numbers of instances. This is particularly relevant for mupdates, and until live-migration is available users are likely to frequently want to stop their instances. Even with live-migration, they may want to gracefully stop instances that are too large to migrate on a fully utilized rack.

A note on terminology: ACPI calls the feature currently implemented on the Propolis side "Fixed Power Button". ACPI also specifies an altogether different feature that it calls "Graceful Shutdown Request" which is not presently implemented by Propolis. We may sometimes talk about "gracefully" stopping an instance, but for the sake of clarity, we may wish to be take care not to confuse operators and customers with the term "Graceful Shutdown" in documentation.

POST /v1/instances now takes a shutdown_policy parameter, either HardOff or PowerButton with a (mandatory) timeout between 1 and 3600 seconds. New instances with no shutdown policy specified in their create request will default to a policy of the pre-existing behavior: Hard-poweroff of the guest OS with no forewarning. Existing instances will also default to hard_off in the CRDB migration.

PUT /v1/instances/{instance} will only change an instance's shutdown_policy if one is explicitly included in the request, such that clients of an older API revision don't potentially clobber an existing instance's shutdown policy.

POST /v1/instances/{instance}/stop and POST /v1/instances/{instance}/reboot now may be passed skip_os_shutdown=true to force a hard-poweroff even if an instance has a soft shutdown policy. If an instance does have a PowerButton shutdown policy, and skip_os_shutdown isn't passed, the instance will have its simulated Power Button pressed, and then following either the guest-initiated power event or the exhaustion of the configured timeout, the instance will be accordingly stopped or rebooted by Propolis.

Further requests to stop or reboot an instance while it is still in the process of shutting down (i.e. the guest hasn't emitted a power event and the configured timeout hasn't expired) are permitted to change the instance's fate, i.e. whether or not Propolis will reboot the instance once it has stopped. However, changes to the shutdown policy's timeout will not be propagated to instances that are already shutting down; only the first timeout Propolis receives in a stop or reboot request is considered. Non-ACPI stop/reset requests, either through skip_os_shutdown or changing to a HardOff policy, will be honored.

The sled-agent's existing 10-minute propolis-zone-reaping timeout is extended by the configured timeout of a PowerButton-type shutdown; that is to say, the zone will be forcibly terminated if propolis does not stop in the ACPI timeout plus 10 minutes. If a hard-stop request is sent during this timeout, and there are more than 10 minutes remaining, it will be replaced by a 10-minute timeout. If a reboot request is sent to propolis while it is still waiting for an ACPI shutdown, the zone-reaping timeout is dropped.

@hawkw
hawkw self-requested a review October 22, 2026 20:55
@karencfv

Copy link
Copy Markdown
Contributor

Heya! Just a heads up that this -> #11380 will be landing very soon. It's part of a larger issue to orchestrate instance restarts during live update (RFD 739).

Notably, in the specific PR that I linked we have to use vmm_put_state() directly because we need to make sure the intended state continues to be what it was. #11380 (comment) . Looking at the code here, I'm not sure how the graceful shutdown process would fit in as it is in this draft.

Anyway, happy to discuss further if you have any questions about what we're doing in update land or whatever :)

@lifning

lifning commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author

Notably, in the specific PR that I linked we have to use vmm_put_state() directly because we need to make sure the intended state continues to be what it was. #11380 (comment) . Looking at the code here, I'm not sure how the graceful shutdown process would fit in as it is in this draft.

re: your note from from that PR (copied here),

Note: I took a look at how graceful shutdown is being implemented on the control plane side in main...lif/acpi-shutdown. It appears there will be a shutdown_policy_action knob similar to how the auto_restart_policy works today. Once this functionality is implemented, we can decide whether we override this preference or just let whatever settings the user set to carry on to this task.

my understanding is that one of the main desires for shutting down instances gracefully is customers wanting not to have uncleanly unmounted disks in the "shutting everything down to take an update" use-case, so it's my assumption that we would want to send the vmm_put_state with the appropriate timeout from policy in the instance table.

@karencfv

Copy link
Copy Markdown
Contributor

re: your note from from that PR (copied here),

Argh, I wrote that note ages ago, sorry! I should have removed it before writing this 😅

my understanding is that one of the main desires for shutting down instances gracefully is customers wanting not to have uncleanly unmounted disks in the "shutting everything down to take an update" use-case, so it's my assumption that we would want to send the vmm_put_state with the appropriate timeout from policy in the instance table.

Yes, definitely. Unless the customers don't want to wait for graceful shutdown (disabling graceful shutdown would really speed up updates). In this case, I guess that means if they chose the HardOff shutdown policy.

@lifning

lifning commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author

in that case, looking at stop_batch in there, should i perhaps consider propagating the shutdown policy info into the vmm table so it's more readily accessible to the background task for that purpose? (forgive my relative diesel naivete if that's a silly question)

@karencfv

Copy link
Copy Markdown
Contributor

Do you mean propagating the policy to each VMM entry and then passing the policy via vmm_put_state() in stop_vmm() ?

I've only taken a look at this PR superficially, but I wonder if there is a way to have it so the background task doesn't need to know about the policy at all. When this task stops the VMM the system automatically shuts down the instances gracefully or not depending on the policy in each instance. Is that possible?

@lifning

lifning commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

I've only taken a look at this PR superficially, but I wonder if there is a way to have it so the background task doesn't need to know about the policy at all. When this task stops the VMM the system automatically shuts down the instances gracefully or not depending on the policy in each instance. Is that possible?

as i've implemented the sled-agent side of this (per the rfd), vmm_put_state just forwards whatever timeout is passed to it by nexus to propolis. so nexus is the thing responsible for retrieving the instance's shutdown policy from the instance table in the database whenever an instance stop/reboot is requested, and the sled-agent has no knowledge of what the instance's shutdown policy when it's servicing a vmm_put_state request besides what's included in the request itself.

@karencfv

Copy link
Copy Markdown
Contributor

Ah aha! I see now. Then yeah, I think your proposal of propagating the information to the VMM table makes sense.

karencfv added a commit that referenced this pull request Oct 27, 2026
Adds a task that gathers all stoppable VMMs that are marked to stop for
a sled evacuation during an update, and stops them.

Sample of how the information will be displayed:

Various successes, failures and errors
```console
    VMMs stopped for sled evacuation: 13 total
      sled_id=1b8c2e9a-4d3f-4b1e-9c7a-2f5e8d6b0a11 n_stopped=6
      sled_id=a4f0d7c3-8e2b-4c6d-b1f9-7e3a5c9d2b44 n_stopped=7
    VMMs that failed to stop: 7 total
      sled_id=1b8c2e9a-4d3f-4b1e-9c7a-2f5e8d6b0a11 n_failed_to_stop=3
        vmm_id=5c3e9f12-7a4b-4d8e-a2c6-1f9b3e7d5a08
        vmm_id=7f1a2b3c-4d5e-4f60-8a7b-9c0d1e2f3a41
        vmm_id=9d4e6f80-2b1c-4a3d-b5e6-0f7a8b9c1d52
      sled_id=a4f0d7c3-8e2b-4c6d-b1f9-7e3a5c9d2b44 n_failed_to_stop=4
        vmm_id=e2d8b4f6-1c9a-4e7b-8d3f-6a2c5b9e1f37
        vmm_id=0a1b2c3d-5e6f-4a70-9b81-c2d3e4f5a663
        vmm_id=3c4d5e6f-7a8b-4c9d-8e0f-1a2b3c4d5e74
        vmm_id=b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d85
    Error messages: 2 total
      fake error 1
      fake error 2
```

No failures

```console
    VMMs stopped for sled evacuation: 13 total
      sled_id=1b8c2e9a-4d3f-4b1e-9c7a-2f5e8d6b0a11 n_stopped=6
      sled_id=a4f0d7c3-8e2b-4c6d-b1f9-7e3a5c9d2b44 n_stopped=7
```

Nothing happened
```console
    VMMs stopped for sled evacuation: 0 total
```

Note: graceful shutdown preferences will need to be integrated in
#11565, or once it lands

Related: #11169
Comment thread nexus/db-model/src/instance.rs Resolved
Comment thread nexus/src/app/mod.rs Resolved

@hawkw hawkw left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed in the hypervisor huddle today, I think the best way to resolve the problem of needing the configured policy in the vmm_stop_for_update task is to make the query it's running do a JOIN with the VMM's instance record, rather than copying the policy into the vmm record. The latter means we would have to keep the two in sync when the policy changes, which seems fraught.

Comment thread nexus/defaults/src/lib.rs Resolved
Comment thread nexus/src/app/instance.rs Resolved
Comment thread schema/crdb/dbinit.sql Resolved
Comment thread schema/crdb/dbinit.sql Resolved
Comment thread sled-agent/src/instance.rs Resolved

@jgallagher jgallagher left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skimmed over this in the same context as Karen (i.e., how this fits in with the reconfigurator work for sled evacuation) and had a few questions. Sorry if they're premature - I know this is a draft, but I'd be happy to cobble together a test branch that includes this and the outstanding reconfigurator bits to start testing it as soon as that's feasible.

Comment thread sled-agent/src/instance.rs Resolved
Comment thread sled-agent/src/instance.rs Resolved
jgallagher added a commit that referenced this pull request Nov 4, 2026
(This is pointed at #11565 and is one of that patches needed for it to
work with sled evacuation during updates.)

The main change here is that
`vmm_list_stoppable_for_update_by_sled_id()` now returns each VMM's
instance's shutdown policy alongside the VMM, and the
`vmm_stop_for_update` background task now sends that policy when it's
shutting down instances for sled evacuation.

A smaller change that came along for the ride was some cleanup around
the db representation of the shutdown policy:

* the model fields are now private, forcing callers to go through the
`shutdown_policy()` helper that reconstructs the Rust enum for the
policy
* `shutdown_policy()` now returns a `Result` instead of an `Option` -
returning an `Option` means the caller has to decide what to do with
`None`, which doesn't seem correct if the only way it can happen means
something has gone horribly wrong (i.e., our db constraints have been
violated). This bubbled out to changing some conversions from `.into()`
to `.try_into()`, but I think that's basically fine and consistent with
other types where we have to check at runtime that some unrepresentable
state hasn't made its way back to us from the db.

I don't expect CI tests to work on this branch since they don't on
#11565 either; I don't think that should block merging this since it's
really a part of that branch anyway.
jgallagher added a commit that referenced this pull request Nov 5, 2026
This PR addresses items 1 and 2 from
#11565 (comment).
(Item 3 was addressed yesterday in
28daa74,
which changed the default for the API to `hard_off`.) The changes for
those two are:

1. The `shutdown_policy` field is no longer an `Option` for either
`Instance` or `InstanceCreate`. `InstanceCreate` has a
`#[serde(default)]`, so it's not required at the API layer, but this way
(a) we don't have to deal with `None` and (b) we don't have the oddity
of always returning `Some(_)` to clients even if they had specified
`None`.
2. The type conversion for old clients just throws away the
`shutdown_policy` field instead of failing if it isn't `HardOff`. This
is less important now that the default is `HardOff`, but I still think
it's reasonable - this way old clients can still see / list instances
that were created by new clients. They just won't be able to tell what
those instance's shutdown policies are. (`shutdown_policy` is still
`None` in `InstanceUpdate`, which means "don't change", so an instance
update from an old client is fine: it doesn't know about
`shutdown_policy`, which turns into `None`, which turns into "don't
change it".)

Most of the bulk off the diff is trivial fallout from 1 - changing a
bunch of `None`s to `InstanceShutdownPolicy::default()`s in a lot of
tests. The end-to-end test had to switch from specifying all the fields
to using the progenitor-client's builder API (which allows you to _not_
specify defaultable fields).

A few other cleanup things came along for the ride:

1. I moved the enforcement of the ACPI timeout second range into a
newtype in the API. This means it's enforced at deserialization time, so
we don't have to remember to validate it every place it comes in in
Nexus.
2. The instance update handler had duplicated the logic of "convert the
`InstanceShutdownPolicy` into the 2-column database representation". I
moved this into an `InstanceUpdate::new()` function in the db-model; I
don't love that it takes a bunch of args but I think it's still a net
win?
3. I changed the propolis pointer in `package-manifest.toml` to match
what's in `Cargo.toml` (all of which now point to the latest commit on
oxidecomputer/propolis#1217). I'm hoping this
lets CI make more progress; I'd like to confirm the end-to-end test
change mentioned above, at least, which I think requires CI.
lifning pushed a commit that referenced this pull request Nov 5, 2026
(This is pointed at #11565 and is one of that patches needed for it to
work with sled evacuation during updates.)

The main change here is that
`vmm_list_stoppable_for_update_by_sled_id()` now returns each VMM's
instance's shutdown policy alongside the VMM, and the
`vmm_stop_for_update` background task now sends that policy when it's
shutting down instances for sled evacuation.

A smaller change that came along for the ride was some cleanup around
the db representation of the shutdown policy:

* the model fields are now private, forcing callers to go through the
`shutdown_policy()` helper that reconstructs the Rust enum for the
policy
* `shutdown_policy()` now returns a `Result` instead of an `Option` -
returning an `Option` means the caller has to decide what to do with
`None`, which doesn't seem correct if the only way it can happen means
something has gone horribly wrong (i.e., our db constraints have been
violated). This bubbled out to changing some conversions from `.into()`
to `.try_into()`, but I think that's basically fine and consistent with
other types where we have to check at runtime that some unrepresentable
state hasn't made its way back to us from the db.

I don't expect CI tests to work on this branch since they don't on
#11565 either; I don't think that should block merging this since it's
really a part of that branch anyway.
lifning pushed a commit that referenced this pull request Nov 5, 2026
This PR addresses items 1 and 2 from
#11565 (comment).
(Item 3 was addressed yesterday in
28daa74,
which changed the default for the API to `hard_off`.) The changes for
those two are:

1. The `shutdown_policy` field is no longer an `Option` for either
   `Instance` or `InstanceCreate`. They do both have a
   `#[serde(default)]` on them, so they're not required at the API
   layer, but this way (a) we don't have to deal with `None` and (b) we
   don't have the oddity of always returning `Some(_)` to clients even
   if they had specified `None`.
2. The type conversion for old clients just throws away the
   `shutdown_policy` field instead of failing if it isn't `HardOff`.
   This is less important now that the default is `HardOff`, but I still
   think it's reasonable - this way old clients can still see / list
   instances that were created by new clients. They just won't be able
   to tell what those instance's shutdown policies are.
   (`shutdown_policy` is still `None` in `InstanceUpdate`, which means
   "don't change", so an instance update from an old client is fine: it
   doesn't know about `shutdown_policy`, which turns into `None`, which
   turns into "don't change it".)

Most of the bulk off the diff is trivial fallout from 1 - changing a
bunch of `None`s to `InstanceShutdownPolicy::default()`s in a lot of
tests. The end-to-end test had to switch from specifying all the fields
to using the progenitor-client's builder API (which allows you to _not_
specify defaultable fields).

A few other cleanup things came along for the ride:

1. I moved the enforcement of the ACPI timeout second range into a
   newtype in the API. This means it's enforced at deserialization time,
   so we don't have to remember to validate it every place it comes in
   in Nexus.
2. The instance update handler had duplicated the logic of "convert the
   `InstanceShutdownPolicy` into the 2-column database representation".
   I moved this into an `InstanceUpdate::new()` function in the
   db-model; I don't love that it takes a bunch of args but I think it's
   still a net win?
3. I changed the propolis pointer in `package-manifest.toml` to match
   what's in `Cargo.toml` (all of which now point to the latest commit
   on oxidecomputer/propolis#1217). I'm hoping
   this lets CI make more progress; I'd like to confirm the end-to-end
   test change mentioned above, at least, which I think requires CI.
lifning pushed a commit that referenced this pull request Nov 6, 2026
(This is pointed at #11565 and is one of that patches needed for it to
work with sled evacuation during updates.)

The main change here is that
`vmm_list_stoppable_for_update_by_sled_id()` now returns each VMM's
instance's shutdown policy alongside the VMM, and the
`vmm_stop_for_update` background task now sends that policy when it's
shutting down instances for sled evacuation.

A smaller change that came along for the ride was some cleanup around
the db representation of the shutdown policy:

* the model fields are now private, forcing callers to go through the
`shutdown_policy()` helper that reconstructs the Rust enum for the
policy
* `shutdown_policy()` now returns a `Result` instead of an `Option` -
returning an `Option` means the caller has to decide what to do with
`None`, which doesn't seem correct if the only way it can happen means
something has gone horribly wrong (i.e., our db constraints have been
violated). This bubbled out to changing some conversions from `.into()`
to `.try_into()`, but I think that's basically fine and consistent with
other types where we have to check at runtime that some unrepresentable
state hasn't made its way back to us from the db.

I don't expect CI tests to work on this branch since they don't on
#11565 either; I don't think that should block merging this since it's
really a part of that branch anyway.
lifning pushed a commit that referenced this pull request Nov 6, 2026
This PR addresses items 1 and 2 from
#11565 (comment).
(Item 3 was addressed yesterday in
28daa74,
which changed the default for the API to `hard_off`.) The changes for
those two are:

1. The `shutdown_policy` field is no longer an `Option` for either
   `Instance` or `InstanceCreate`. They do both have a
   `#[serde(default)]` on them, so they're not required at the API
   layer, but this way (a) we don't have to deal with `None` and (b) we
   don't have the oddity of always returning `Some(_)` to clients even
   if they had specified `None`.
2. The type conversion for old clients just throws away the
   `shutdown_policy` field instead of failing if it isn't `HardOff`.
   This is less important now that the default is `HardOff`, but I still
   think it's reasonable - this way old clients can still see / list
   instances that were created by new clients. They just won't be able
   to tell what those instance's shutdown policies are.
   (`shutdown_policy` is still `None` in `InstanceUpdate`, which means
   "don't change", so an instance update from an old client is fine: it
   doesn't know about `shutdown_policy`, which turns into `None`, which
   turns into "don't change it".)

Most of the bulk off the diff is trivial fallout from 1 - changing a
bunch of `None`s to `InstanceShutdownPolicy::default()`s in a lot of
tests. The end-to-end test had to switch from specifying all the fields
to using the progenitor-client's builder API (which allows you to _not_
specify defaultable fields).

A few other cleanup things came along for the ride:

1. I moved the enforcement of the ACPI timeout second range into a
   newtype in the API. This means it's enforced at deserialization time,
   so we don't have to remember to validate it every place it comes in
   in Nexus.
2. The instance update handler had duplicated the logic of "convert the
   `InstanceShutdownPolicy` into the 2-column database representation".
   I moved this into an `InstanceUpdate::new()` function in the
   db-model; I don't love that it takes a bunch of args but I think it's
   still a net win?
3. I changed the propolis pointer in `package-manifest.toml` to match
   what's in `Cargo.toml` (all of which now point to the latest commit
   on oxidecomputer/propolis#1217). I'm hoping
   this lets CI make more progress; I'd like to confirm the end-to-end
   test change mentioned above, at least, which I think requires CI.
Comment on lines -809 to -811
VmmStateRequested::Running => {
let vmms = self.vmms.clone();
let log = self.log.new(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the previous behavior of sled-agent-sim here: when there's a mock propolis, and Run is requested: not send anything to propolis, spawn a task that sleeps 10 seconds before calling sim_ensure(), and early-return VmmPutStateResponse { updated_runtime:None }. this was peculiar and seems like just a bug, as none of the small handful of integ tests calling start_local_mock_propolis_server rely on this behavior at all, and it does not apply to the vast majority of tests not starting a mock propolis.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the old behavior here is from #4691. The PR description seems reasonable, I think? Specifically the

in this world where instance create calls to nexus now no longer block on sled-agent's propolis zone installation finishing, care must be taken in e.g. integ tests not to assume you can, say, connect to the instance as soon as the instance create call returns. accordingly, integ tests have been changed to poll until the instance is ready.

bit. Do you happen to remember more about the original intent?

@lifning lifning Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i believe the intent for this snippet was to make the one integ test that used a mock propolis behave realistically, as described.

noble as the intent may have been, the code block as written isn't really advancing that goal in a way that matters any more. it isn't relevant to any current tests other than test_instance_serial; mock_lock is none everywhere else*.

(* a few tests to do with handling of anti-affinity groups and ssh keys do invoke .start_local_mock_propolis_server(), but have nothing in their test logic about whether instances reach the running state that this would impact; perhaps its invocation at all in these was an artifact of duplicated test heading boilerplate?)

nowadays, i believe the goal of not having instance states be immediately reflected in instance integ tests is achieved by way of instance_simulate and instance_wait_for_state - and the former is called by the serial test in its own custom version of the latter

prototyping rfd622: acpi api plumbing

further wip: figuring out db schema for acpi power button

external api versioning wrappers for instance endpoints

still working on the db schema

wip - wired enough up of shutdown policy to pass simple cargo check

further wip: now with cargo check --tests

sled agent api versioning, regen openapi

consistent column order between dbinit and up*

missed a spot with column order consistency & default to 10 minutes when shutdown policy underspecified in create request & also just point to the propolis PR branch for this

fill in some TODOs, mostly doc comments

add override param to instance stop and reboot nexus endpoints, and wire up instance update params in diesel

start making tests for shutdown policy. also, consider the case of an instance update request that *doesn't* change the shutdown policy

more acpi shutdown integ test. i think the most appropriate way to test this will be to add a little escape hatch to mock propolis

integ test for shutdown feature

wip - acpi shutdown cleanups, trying to get integ test working

still working on shutdown integ test, but now figured out the workarounds needed for mock propolis server to participate in the way we need

don't clamp acpi timeout in DbInstance

elaborate a bit on InstanceCreate and InstanceUpdate request shutdown_policy fields

wip - notes on my current fixquest

wip - out of one bug, into another..

fix further issues with mock-propolis / sled-agent-sim (former does not like being started with different vmm id), now shutdown integ tests partially passing, but need to convince nexus to be comfortable sending hard-stop requests to an instance that only received a soft-stop so far

wip - another test case. still puzzling on the most appropriate way to address the fact the instance update saga yoinks the vmm record for the first soft-off request such that the subsequent hard-off doesn't make it through

oh, it was sled-agent-sim again. good.

oops sorry clippy

move consts to a common denominator of nexus-db-model and nexus - per pr feedback (thanks eliza)

helix got a bit confused this evening

addressing a few pr comments

also fixup: unwritten buffer amid a rebase :(

missed the upper bound on the CHECK conditions
smklein added a commit that referenced this pull request Nov 11, 2026
Fixes #11769

- That issue has a lot more info, but the shortest summary is thus: (1)
we compare dbinit.sql to "the sum of all migration files", and hope they
produce the same result. (2) Our mechanism for comparing column types
used the CockroachDB information-schema field `data_type` to compare
types. (3) `data_type` is lossy - it does not include precision, and it
perceives "all arrays as being equal" (yikes). This is bad.
- There is a CRDB-specific information field called `crdb_sql_type`,
which contains a more exact representation of the column type. We opt to
use that in this PR instead. As a note: I *think* this type isn't
necessarily stable across CRDB versions, but (a) we only compare schemas
using a "single CRDB version at a time", so, this should be fine, and
(b) we aren't really flying through CRDB upgrades at this time.

This PR is composed of two critical commits:
1.
05f6864
- add a regression test, observe that we failed these cases of "columns
with mismatched types"
2.
f8ec8d6
- fix the regression test by comparing `crdb_sql_type`.

<details>
<summary>Details</summary>

## Problem

The schema migration tests (`validate_migration_from_base_version`)
compare a database built by applying migrations against one built from
`dbinit.sql`, using a snapshot of `information_schema`. For columns,
that
snapshot used only `data_type`, which is the base type name: it omits
precision, length, interval fields, and array element types.

For example:

| declared | `data_type` | `crdb_sql_type` |
|---|---|---|
| `INTERVAL` | `interval` | `INTERVAL` |
| `INTERVAL(0)` | `interval` | `INTERVAL(0)` |
| `STRING(63)` | `text` | `STRING(63)` |
| `INET[]` | `ARRAY` | `INET[]` |
| `UUID[]` | `ARRAY` | `UUID[]` |

These are not equivalent: `INTERVAL(0)` rounds fractional seconds, and
`STRING(63)` rejects longer values. So a fresh install and an upgraded
rack
could end up with different column types without any test failing. This
is
how the `INTERVAL` vs `INTERVAL(0)` mismatch in #11565 went unnoticed
(#11769).

## Changes

1. Add `compare_table_differing_column_type_modifiers`. For each pair of
types,
one database creates the column with `CREATE TABLE`, and the other adds
it
with `ALTER TABLE ... ADD COLUMN` (as a migration would). It then checks
whether `pretty_assert_eq` (the same comparison used by the migration
tests)
reports a difference. Identical-type pairs act as a control. On its own,
   this commit fails: none of the six differing pairs are detected.
3. Also compare `crdb_sql_type`, a CockroachDB-specific column of
   `information_schema.columns` containing the full type

([docs](https://www.cockroachlabs.com/docs/v23.1/information-schema#columns)).

Adding the individual precision columns (`datetime_precision`,
`character_maximum_length`, `numeric_precision`, ...) was considered,
but
they're redundant with `crdb_sql_type` and incomplete: e.g., CockroachDB
reports `interval_type` as `NULL`, so `INTERVAL` vs `INTERVAL HOUR` is
indistinguishable with those columns.

## Testing

- All tests in `integration_tests::schema` pass, including
`validate_migration_from_base_version`: existing migrations already
match
`dbinit.sql` under the stricter comparison, so no schema changes are
needed.
- The new test fails on the first commit and passes on the second.

Fixes #11769

</details>
Comment on lines +37 to +38
// RFD 622: "A user-configurable timeout for the instance to finish stopping
// itself should be present, bounded between 1 and 3600 seconds, inclusive."

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know the RFD says 3600 seconds but I think there's some hesitance to immediately allowing such a long timeout without any kind of motivator. if waiting fifty minutes is not enough, but 56 is, that feels like it strains credulity? there's a real disruptive aspect to this too, if we're telling an operator "you must wait at least an hour for this sled to be clear of workloads so you can do maintenance on it, or risk causing data corruption for your users' applications by force-stopping instances configured to want graceful shutdown"

what do you think about picking a shorter time here, say, 15 minutes? since the longest default configurations I'd found were on the order of ten and even that seemed pretty long to people on bug trackers. if someone really needs a longer time here, then they really ought to tell us and we work together on the problem at hand. and there's a nudge to do that, since someone who needs a longer timeout would get told by the API and if it truly is necessary then we can figure out a suitable path forward for the product. that could be "we should really get live migration shipped so you don't need to stop instances", and the corresponding issues to take care of there.

as-is an hour makes me ask why not two hours? a day? it seems like it would be good if we can go back to the RFD with the more incremental stance - IIUC the numbers in the RFD were just because they were most comparable to other systems, rather than a strongly-considered choice.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i suppose we can set it to 15 and then decide whether to increase it based on customers who bring a compelling case for it. much easier than trying to reduce it to 15 after it's been deployed and people have 30min timeouts in their databases, certainly

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, I think a much shorter timeout is prudent until we hear customers clamoring for it for some reason (and it would be good to understand why).

@jordanhendricks
jordanhendricks self-requested a review November 16, 2026 20:04
#[allow(clippy::derivable_impls)] // impl_enum_type doesn't afford #[default]
impl Default for InstanceShutdownAction {
fn default() -> Self {
Self::HardOff

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the vast majority of launches would want PowerButton as the shutdown policy.

In the solutions eng huddle, we were talking about updating basically all of our integrations to set the shutdown policy to PowerButton.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

others can confirm, but I think this defaults to HardOff for backwards compatibility

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what I thought, but I was curious if that's worth maintaining in this case. I still think most launches would rather switch to PowerButton rather than maintaining HardOff.

I would guess that we will eventually change this to PowerButton as the default at some point since that's what users will expect intuitively (and what AWS, GCP, and Azure default to). Are there specific cases that we're worried about? The only downside I could see is VMs taking slightly longer to turn off, but if we also default the grace period to 30s (or something fairly low), that isn't too much of a slow down.

@lifning lifning Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in fact it was originally PowerButton (10 min) by default, but after this point (implementation complexities around what behaving consistently means when upgrading a rack w/ older instances vs. creating new ones, as well as the commitment of having an opinion of what the default timeout should be), we changed it to HardOff during review

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

towards the end of last week I tried to digest the background and some motivations for defaults (and some points of comparison for grace windows) into https://rfd.shared.oxide.computer/rfd/622. here: yeah, this does not preclude us from changing the default in the future, but if there are specific applications that really need graceful stops always that's actually a pretty useful datapoint for the product at large.

for example, Packer expecting "stop" to get guests to flush caches feels obvious from a certain point of view. but if, in the future, there's a more "ephemeral" instance type (microvms? "spot-like" opportunistic execution?), should that stop behavior graceful or not? i would say it is a bug generator for Packer to use such an "ephemeral" instance, because they might not get a graceful stop or if they do it might be well below any expected completion time. so if we're building integrations it's really valuable for us to understand what the integration needs so we continue to integrate it properly in the face of new product features.

(30s would be way too short for a default IMO, unfortunately. I've tried to talk about Windows a bit in the RFD, but 30s is the perfect danger zone for Windows instances to get unceremoniously killed mid-update and turned into paperweights)

@bwagner5 bwagner5 Nov 16, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all the additional context! The one I was thinking of is Kubernete's Graceful Node Shutdown feature. This is something that "just works" in other clouds since the default is to gracefully shutdown VMs (with a long enough grace period).

I've tried to talk about Windows a bit in the RFD, but 30s is the perfect danger zone for Windows instances to get unceremoniously killed mid-update and turned into paperweights

that's fun...

@jordanhendricks jordanhendricks left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looked through this just for additional context / knowledge sharing. One important comment around the API restrictions (lowering the timeout allowed) but nothing else.

I also left a comment on the propolis side that we should consider moving the shtudown timeout handling into sled-agent instead of having propolis do that: oxidecomputer/propolis#1217 (comment).

I don't know what the lift is there, and it doesn't seem super important to figure out before merging this.

Comment on lines +37 to +38
// RFD 622: "A user-configurable timeout for the instance to finish stopping
// itself should be present, bounded between 1 and 3600 seconds, inclusive."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, I think a much shorter timeout is prudent until we hear customers clamoring for it for some reason (and it would be good to understand why).

@lifning
lifning marked this pull request as ready for review November 19, 2026 21:43

@david-crespo david-crespo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API and docs look good to me. Looks like you need to merge main because your API version got taken.

@iximeow

iximeow commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

your API version got taken

that was me adding conflicts on every InstanceCreate by merging #11858 lol. I'd promised to merge back up here and resolve conflicts, which 9e859b1 does.

... unfortunately that commit is victim to "buildomat is currently wedged", so that's deeply unfortunate.

@david-crespo

david-crespo commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Sorry yeah I misread, I see now 111000 and 111100 are different :)

image

@iximeow iximeow left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a smattering of comments but the top-line is I'm down to click the button too. most important "we should probably not pub use sled_agent_types_versions in Nexus", and if we give this a moment (helios/deploy is helping in its own way..) I'll push a commit to fix that in this initial external API version, otherwise we can of course Just Fix It at any later point.

}

impl UngracefulTermGraceTime {
fn set_stop_timeout(&mut self, method: InstanceShutdownPolicy) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, and for emphasis: pick! we should not wait another however long on backed-up CI for this:

I'd probably call this update so that callers would read like force_stop_timer.update(), rather than force_stop_timer.set_stop_timeout()-which-sometimes-doesnt-set-anything

Comment on lines +3079 to +3081
#[derive(Default)]
struct UngracefulTermGraceTime {
timeout_duration: std::time::Duration,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might just be me, but Default and Duration always make me super itchy. is Duration::default.. 0 seconds? one second? something else? I don't think my opinion is super important here in isolation, but may make more sense in the context of my next comment below..

edit: alternatively, maybe this should be part of a tuple under stop_timeout, as this field isn't meaningful unless there is a timeout set and that's kinda what that option means?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a little awkward, because method could also be part of the tuple, which gets back toward what that made me split this out... maybe i just do a Option<UngracefulTermGraceTimeInner>


fn disarm(&mut self) {
if self.stop_req_method != InstanceShutdownPolicy::HardOff {
self.stop_timeout = None;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a disarmed UngracefulTermGraceTime at this point would have self.stop_req_method == PowerButton { whatever } && self.stop_timeout.is_none() - it would behave correctly but the struct's state at that point is a bit funky. incidentally, since we derive Default the "normal" stop_req_method had been PowerButton until that was made HardStop too.

since we only hold onto the stop method to ask "was this deadline established due to a graceful stop which we are willing to adjust", maybe we should only retain method != InstanceShutdownPolicy::HardOff when we're updating the internal bits of the struct? even so "what is an UngracefulTermGraceTime before a timeout is armed" is kind of nebulous to me.

Comment on lines +1055 to +1059
// If we receive a reboot request while a stop
// timeout is running that was initiated by an
// ACPI shutdown, disarm the timeout, but only
// if Propolis was still there and honored the
// request.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw it's a bit confusing that we mention "was initiated by an ACPI shutdown" here, but that's only checked inside disarm() (the comment is right in effect but I was pretty confused for a second!)

more importantly for the check here, since we only disarm the timer after we've heard back from Propolis, we'll presumably yield on the .await and potentially see the timer fire while Propolis is handling a reboot which would have disarmed the timer. that's a pretty narrow window, I think, so I'm not really concerned about it but it makes me wonder if we could rephrase the conditions before/after the operation.

my thinking here is: there would have had to be an ACPI-style reboot that arrived right as Propolis was approaching a previously-established deadline from an ACPI-style shutdown .. that's a bit difficult to motivate myself into being concerned about, since the timing means it was really kind of a dice roll on if the ACPI-style shutdown stopped the instance just before the reboot arrived.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think we would either have to yield on the .await for at least STOP_GRACE_PERIOD (= 10 minutes) for this timer to fire, or expect a propolis that had been unresponsive for acpi_timeout_secs + STOP_GRACE_PERIOD to suddenly honor a last-moment reboot request, in order for that to be an issue.

it does make me think that if somehow we saw a successful-looking reboot request response from a propolis that had already consumed its configured acpi policy timeout and was into STOP_GRACE_PERIOD territory, we would disarm here, and maybe we shouldn't. i think per jordan's feedback, a more sophisticated timeout-handling mechanism that perhaps moves the timeout logic out of propolis's state driver could cover this case more robustly

use crate::v1::instance::InstanceMigrationTargetParams;
use crate::v1::instance::VmmStateRequested as VmmStateRequestedV1;

/// Instance-stopping method choices

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since this ends up being the public docs on InstanceShutdownPolicy we probably should say a bit more here..

@lifning lifning Nov 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +25 to +26
pub use sled_agent_types_versions::v66::instance::{
AcpiTimeoutSecs, InstanceShutdownPolicy,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-using the client types from a different API here was very surprising - I was going to remark that we should leave a better comment on InstanceShutdownPolicy for ourselves reading later, that the doc comment on InstanceCreate::shutdown_policy seems great, and that we should probably just copy it before realizing that the sled-agent client type is the external API type.

I realize we've done this elsewhere (bgp_peer_src_addr does this with RouterPeerType in the external BgpPeer struct) but it definitely pokes a hole in my mental model that sled-agent's API is very internal!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it was certainly a little strange for me too, but per #11565 (comment) they're at least both versioned.. though yeah, the difference in docstring audience is certainly a factor

Comment on lines +81 to +82
fn try_from(value: u64) -> Result<Self, Self::Error> {
if value < Self::MIN || value > Self::MAX {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is because we're trying to keep schemars in line with what the column constraints actually are, right? since the constraint still has 3600 seconds as a maximum (as noted on the AcpiTimeoutSecs doc comment) at the very least we're checking different things in different places..

@lifning lifning Nov 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whoops 💀 definitely wanna fix that once the whole waiting-for-buildomat situation clears up... but would want to get that in right away so as not to make a whole new schema revision about it, i think? maybe it's ok to eat the CI delay to not risk it

/// CRDB.
pub fn shutdown_policy(
&self,
) -> Result<instance_types::InstanceShutdownPolicy, Error> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my AcpiTimeoutSecs comment is partially because I was at the bargaining stage of "Instance in the database is only fallibly convertible to API types" - I was hoping we could avoid having the fallible conversion by trusting the database has enforced the constraints we want on the API types. as-is, looking at a cursèd instance is impossible! paginated views including such an instance are also cursèd!

but I think even if we didn't check the timeout specifically, the relationship that "PowerButton implies a configured number" (and HardOff implies NULL, which we should probably check here too!) seems worth validating..

@lifning

lifning commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

I have a smattering of comments but the top-line is I'm down to click the button too. most important "we should probably not pub use sled_agent_types_versions in Nexus", and if we give this a moment (helios/deploy is helping in its own way..) I'll push a commit to fix that in this initial external API version, otherwise we can of course Just Fix It at any later point.

if the tyranny of helios/deploy is not too daunting i tried to address most of this in https://github.com/oxidecomputer/omicron/compare/lif/acpi-shutdown..lif/acpi-shutdown-followup

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants