Repository navigation
Conversation
|
Heya! Just a heads up that this -> #11380 will be landing very soon. It's part of a larger issue to orchestrate instance restarts during live update (RFD 739). Notably, in the specific PR that I linked we have to use vmm_put_state() directly because we need to make sure the intended state continues to be what it was. #11380 (comment) . Looking at the code here, I'm not sure how the graceful shutdown process would fit in as it is in this draft. Anyway, happy to discuss further if you have any questions about what we're doing in update land or whatever :) |
re: your note from from that PR (copied here),
my understanding is that one of the main desires for shutting down instances gracefully is customers wanting not to have uncleanly unmounted disks in the "shutting everything down to take an update" use-case, so it's my assumption that we would want to send the |
Argh, I wrote that note ages ago, sorry! I should have removed it before writing this 😅
Yes, definitely. Unless the customers don't want to wait for graceful shutdown (disabling graceful shutdown would really speed up updates). In this case, I guess that means if they chose the |
|
in that case, looking at |
|
Do you mean propagating the policy to each VMM entry and then passing the policy via I've only taken a look at this PR superficially, but I wonder if there is a way to have it so the background task doesn't need to know about the policy at all. When this task stops the VMM the system automatically shuts down the instances gracefully or not depending on the policy in each instance. Is that possible? |
as i've implemented the sled-agent side of this (per the rfd), |
|
Ah aha! I see now. Then yeah, I think your proposal of propagating the information to the VMM table makes sense. |
Adds a task that gathers all stoppable VMMs that are marked to stop for
a sled evacuation during an update, and stops them.
Sample of how the information will be displayed:
Various successes, failures and errors
```console
VMMs stopped for sled evacuation: 13 total
sled_id=1b8c2e9a-4d3f-4b1e-9c7a-2f5e8d6b0a11 n_stopped=6
sled_id=a4f0d7c3-8e2b-4c6d-b1f9-7e3a5c9d2b44 n_stopped=7
VMMs that failed to stop: 7 total
sled_id=1b8c2e9a-4d3f-4b1e-9c7a-2f5e8d6b0a11 n_failed_to_stop=3
vmm_id=5c3e9f12-7a4b-4d8e-a2c6-1f9b3e7d5a08
vmm_id=7f1a2b3c-4d5e-4f60-8a7b-9c0d1e2f3a41
vmm_id=9d4e6f80-2b1c-4a3d-b5e6-0f7a8b9c1d52
sled_id=a4f0d7c3-8e2b-4c6d-b1f9-7e3a5c9d2b44 n_failed_to_stop=4
vmm_id=e2d8b4f6-1c9a-4e7b-8d3f-6a2c5b9e1f37
vmm_id=0a1b2c3d-5e6f-4a70-9b81-c2d3e4f5a663
vmm_id=3c4d5e6f-7a8b-4c9d-8e0f-1a2b3c4d5e74
vmm_id=b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d85
Error messages: 2 total
fake error 1
fake error 2
```
No failures
```console
VMMs stopped for sled evacuation: 13 total
sled_id=1b8c2e9a-4d3f-4b1e-9c7a-2f5e8d6b0a11 n_stopped=6
sled_id=a4f0d7c3-8e2b-4c6d-b1f9-7e3a5c9d2b44 n_stopped=7
```
Nothing happened
```console
VMMs stopped for sled evacuation: 0 total
```
Note: graceful shutdown preferences will need to be integrated in
#11565, or once it lands
Related: #11169
d1d85f0 to
1e06789
Compare
hawkw
left a comment
There was a problem hiding this comment.
As we discussed in the hypervisor huddle today, I think the best way to resolve the problem of needing the configured policy in the vmm_stop_for_update task is to make the query it's running do a JOIN with the VMM's instance record, rather than copying the policy into the vmm record. The latter means we would have to keep the two in sync when the policy changes, which seems fraught.
jgallagher
left a comment
There was a problem hiding this comment.
Skimmed over this in the same context as Karen (i.e., how this fits in with the reconfigurator work for sled evacuation) and had a few questions. Sorry if they're premature - I know this is a draft, but I'd be happy to cobble together a test branch that includes this and the outstanding reconfigurator bits to start testing it as soon as that's feasible.
(This is pointed at #11565 and is one of that patches needed for it to work with sled evacuation during updates.) The main change here is that `vmm_list_stoppable_for_update_by_sled_id()` now returns each VMM's instance's shutdown policy alongside the VMM, and the `vmm_stop_for_update` background task now sends that policy when it's shutting down instances for sled evacuation. A smaller change that came along for the ride was some cleanup around the db representation of the shutdown policy: * the model fields are now private, forcing callers to go through the `shutdown_policy()` helper that reconstructs the Rust enum for the policy * `shutdown_policy()` now returns a `Result` instead of an `Option` - returning an `Option` means the caller has to decide what to do with `None`, which doesn't seem correct if the only way it can happen means something has gone horribly wrong (i.e., our db constraints have been violated). This bubbled out to changing some conversions from `.into()` to `.try_into()`, but I think that's basically fine and consistent with other types where we have to check at runtime that some unrepresentable state hasn't made its way back to us from the db. I don't expect CI tests to work on this branch since they don't on #11565 either; I don't think that should block merging this since it's really a part of that branch anyway.
This PR addresses items 1 and 2 from #11565 (comment). (Item 3 was addressed yesterday in 28daa74, which changed the default for the API to `hard_off`.) The changes for those two are: 1. The `shutdown_policy` field is no longer an `Option` for either `Instance` or `InstanceCreate`. `InstanceCreate` has a `#[serde(default)]`, so it's not required at the API layer, but this way (a) we don't have to deal with `None` and (b) we don't have the oddity of always returning `Some(_)` to clients even if they had specified `None`. 2. The type conversion for old clients just throws away the `shutdown_policy` field instead of failing if it isn't `HardOff`. This is less important now that the default is `HardOff`, but I still think it's reasonable - this way old clients can still see / list instances that were created by new clients. They just won't be able to tell what those instance's shutdown policies are. (`shutdown_policy` is still `None` in `InstanceUpdate`, which means "don't change", so an instance update from an old client is fine: it doesn't know about `shutdown_policy`, which turns into `None`, which turns into "don't change it".) Most of the bulk off the diff is trivial fallout from 1 - changing a bunch of `None`s to `InstanceShutdownPolicy::default()`s in a lot of tests. The end-to-end test had to switch from specifying all the fields to using the progenitor-client's builder API (which allows you to _not_ specify defaultable fields). A few other cleanup things came along for the ride: 1. I moved the enforcement of the ACPI timeout second range into a newtype in the API. This means it's enforced at deserialization time, so we don't have to remember to validate it every place it comes in in Nexus. 2. The instance update handler had duplicated the logic of "convert the `InstanceShutdownPolicy` into the 2-column database representation". I moved this into an `InstanceUpdate::new()` function in the db-model; I don't love that it takes a bunch of args but I think it's still a net win? 3. I changed the propolis pointer in `package-manifest.toml` to match what's in `Cargo.toml` (all of which now point to the latest commit on oxidecomputer/propolis#1217). I'm hoping this lets CI make more progress; I'd like to confirm the end-to-end test change mentioned above, at least, which I think requires CI.
(This is pointed at #11565 and is one of that patches needed for it to work with sled evacuation during updates.) The main change here is that `vmm_list_stoppable_for_update_by_sled_id()` now returns each VMM's instance's shutdown policy alongside the VMM, and the `vmm_stop_for_update` background task now sends that policy when it's shutting down instances for sled evacuation. A smaller change that came along for the ride was some cleanup around the db representation of the shutdown policy: * the model fields are now private, forcing callers to go through the `shutdown_policy()` helper that reconstructs the Rust enum for the policy * `shutdown_policy()` now returns a `Result` instead of an `Option` - returning an `Option` means the caller has to decide what to do with `None`, which doesn't seem correct if the only way it can happen means something has gone horribly wrong (i.e., our db constraints have been violated). This bubbled out to changing some conversions from `.into()` to `.try_into()`, but I think that's basically fine and consistent with other types where we have to check at runtime that some unrepresentable state hasn't made its way back to us from the db. I don't expect CI tests to work on this branch since they don't on #11565 either; I don't think that should block merging this since it's really a part of that branch anyway.
This PR addresses items 1 and 2 from #11565 (comment). (Item 3 was addressed yesterday in 28daa74, which changed the default for the API to `hard_off`.) The changes for those two are: 1. The `shutdown_policy` field is no longer an `Option` for either `Instance` or `InstanceCreate`. They do both have a `#[serde(default)]` on them, so they're not required at the API layer, but this way (a) we don't have to deal with `None` and (b) we don't have the oddity of always returning `Some(_)` to clients even if they had specified `None`. 2. The type conversion for old clients just throws away the `shutdown_policy` field instead of failing if it isn't `HardOff`. This is less important now that the default is `HardOff`, but I still think it's reasonable - this way old clients can still see / list instances that were created by new clients. They just won't be able to tell what those instance's shutdown policies are. (`shutdown_policy` is still `None` in `InstanceUpdate`, which means "don't change", so an instance update from an old client is fine: it doesn't know about `shutdown_policy`, which turns into `None`, which turns into "don't change it".) Most of the bulk off the diff is trivial fallout from 1 - changing a bunch of `None`s to `InstanceShutdownPolicy::default()`s in a lot of tests. The end-to-end test had to switch from specifying all the fields to using the progenitor-client's builder API (which allows you to _not_ specify defaultable fields). A few other cleanup things came along for the ride: 1. I moved the enforcement of the ACPI timeout second range into a newtype in the API. This means it's enforced at deserialization time, so we don't have to remember to validate it every place it comes in in Nexus. 2. The instance update handler had duplicated the logic of "convert the `InstanceShutdownPolicy` into the 2-column database representation". I moved this into an `InstanceUpdate::new()` function in the db-model; I don't love that it takes a bunch of args but I think it's still a net win? 3. I changed the propolis pointer in `package-manifest.toml` to match what's in `Cargo.toml` (all of which now point to the latest commit on oxidecomputer/propolis#1217). I'm hoping this lets CI make more progress; I'd like to confirm the end-to-end test change mentioned above, at least, which I think requires CI.
06a367e to
8a5fdc9
Compare
(This is pointed at #11565 and is one of that patches needed for it to work with sled evacuation during updates.) The main change here is that `vmm_list_stoppable_for_update_by_sled_id()` now returns each VMM's instance's shutdown policy alongside the VMM, and the `vmm_stop_for_update` background task now sends that policy when it's shutting down instances for sled evacuation. A smaller change that came along for the ride was some cleanup around the db representation of the shutdown policy: * the model fields are now private, forcing callers to go through the `shutdown_policy()` helper that reconstructs the Rust enum for the policy * `shutdown_policy()` now returns a `Result` instead of an `Option` - returning an `Option` means the caller has to decide what to do with `None`, which doesn't seem correct if the only way it can happen means something has gone horribly wrong (i.e., our db constraints have been violated). This bubbled out to changing some conversions from `.into()` to `.try_into()`, but I think that's basically fine and consistent with other types where we have to check at runtime that some unrepresentable state hasn't made its way back to us from the db. I don't expect CI tests to work on this branch since they don't on #11565 either; I don't think that should block merging this since it's really a part of that branch anyway.
This PR addresses items 1 and 2 from #11565 (comment). (Item 3 was addressed yesterday in 28daa74, which changed the default for the API to `hard_off`.) The changes for those two are: 1. The `shutdown_policy` field is no longer an `Option` for either `Instance` or `InstanceCreate`. They do both have a `#[serde(default)]` on them, so they're not required at the API layer, but this way (a) we don't have to deal with `None` and (b) we don't have the oddity of always returning `Some(_)` to clients even if they had specified `None`. 2. The type conversion for old clients just throws away the `shutdown_policy` field instead of failing if it isn't `HardOff`. This is less important now that the default is `HardOff`, but I still think it's reasonable - this way old clients can still see / list instances that were created by new clients. They just won't be able to tell what those instance's shutdown policies are. (`shutdown_policy` is still `None` in `InstanceUpdate`, which means "don't change", so an instance update from an old client is fine: it doesn't know about `shutdown_policy`, which turns into `None`, which turns into "don't change it".) Most of the bulk off the diff is trivial fallout from 1 - changing a bunch of `None`s to `InstanceShutdownPolicy::default()`s in a lot of tests. The end-to-end test had to switch from specifying all the fields to using the progenitor-client's builder API (which allows you to _not_ specify defaultable fields). A few other cleanup things came along for the ride: 1. I moved the enforcement of the ACPI timeout second range into a newtype in the API. This means it's enforced at deserialization time, so we don't have to remember to validate it every place it comes in in Nexus. 2. The instance update handler had duplicated the logic of "convert the `InstanceShutdownPolicy` into the 2-column database representation". I moved this into an `InstanceUpdate::new()` function in the db-model; I don't love that it takes a bunch of args but I think it's still a net win? 3. I changed the propolis pointer in `package-manifest.toml` to match what's in `Cargo.toml` (all of which now point to the latest commit on oxidecomputer/propolis#1217). I'm hoping this lets CI make more progress; I'd like to confirm the end-to-end test change mentioned above, at least, which I think requires CI.
8a5fdc9 to
e385a54
Compare
| VmmStateRequested::Running => { | ||
| let vmms = self.vmms.clone(); | ||
| let log = self.log.new( |
There was a problem hiding this comment.
the previous behavior of sled-agent-sim here: when there's a mock propolis, and Run is requested: not send anything to propolis, spawn a task that sleeps 10 seconds before calling sim_ensure(), and early-return VmmPutStateResponse { updated_runtime:None }. this was peculiar and seems like just a bug, as none of the small handful of integ tests calling start_local_mock_propolis_server rely on this behavior at all, and it does not apply to the vast majority of tests not starting a mock propolis.
There was a problem hiding this comment.
It looks like the old behavior here is from #4691. The PR description seems reasonable, I think? Specifically the
in this world where instance create calls to nexus now no longer block on sled-agent's propolis zone installation finishing, care must be taken in e.g. integ tests not to assume you can, say, connect to the instance as soon as the instance create call returns. accordingly, integ tests have been changed to poll until the instance is ready.
bit. Do you happen to remember more about the original intent?
There was a problem hiding this comment.
i believe the intent for this snippet was to make the one integ test that used a mock propolis behave realistically, as described.
noble as the intent may have been, the code block as written isn't really advancing that goal in a way that matters any more. it isn't relevant to any current tests other than test_instance_serial; mock_lock is none everywhere else*.
(* a few tests to do with handling of anti-affinity groups and ssh keys do invoke .start_local_mock_propolis_server(), but have nothing in their test logic about whether instances reach the running state that this would impact; perhaps its invocation at all in these was an artifact of duplicated test heading boilerplate?)
nowadays, i believe the goal of not having instance states be immediately reflected in instance integ tests is achieved by way of instance_simulate and instance_wait_for_state - and the former is called by the serial test in its own custom version of the latter
prototyping rfd622: acpi api plumbing further wip: figuring out db schema for acpi power button external api versioning wrappers for instance endpoints still working on the db schema wip - wired enough up of shutdown policy to pass simple cargo check further wip: now with cargo check --tests sled agent api versioning, regen openapi consistent column order between dbinit and up* missed a spot with column order consistency & default to 10 minutes when shutdown policy underspecified in create request & also just point to the propolis PR branch for this fill in some TODOs, mostly doc comments add override param to instance stop and reboot nexus endpoints, and wire up instance update params in diesel start making tests for shutdown policy. also, consider the case of an instance update request that *doesn't* change the shutdown policy more acpi shutdown integ test. i think the most appropriate way to test this will be to add a little escape hatch to mock propolis integ test for shutdown feature wip - acpi shutdown cleanups, trying to get integ test working still working on shutdown integ test, but now figured out the workarounds needed for mock propolis server to participate in the way we need don't clamp acpi timeout in DbInstance elaborate a bit on InstanceCreate and InstanceUpdate request shutdown_policy fields wip - notes on my current fixquest wip - out of one bug, into another.. fix further issues with mock-propolis / sled-agent-sim (former does not like being started with different vmm id), now shutdown integ tests partially passing, but need to convince nexus to be comfortable sending hard-stop requests to an instance that only received a soft-stop so far wip - another test case. still puzzling on the most appropriate way to address the fact the instance update saga yoinks the vmm record for the first soft-off request such that the subsequent hard-off doesn't make it through oh, it was sled-agent-sim again. good. oops sorry clippy move consts to a common denominator of nexus-db-model and nexus - per pr feedback (thanks eliza) helix got a bit confused this evening addressing a few pr comments also fixup: unwritten buffer amid a rebase :( missed the upper bound on the CHECK conditions
Fixes #11769 - That issue has a lot more info, but the shortest summary is thus: (1) we compare dbinit.sql to "the sum of all migration files", and hope they produce the same result. (2) Our mechanism for comparing column types used the CockroachDB information-schema field `data_type` to compare types. (3) `data_type` is lossy - it does not include precision, and it perceives "all arrays as being equal" (yikes). This is bad. - There is a CRDB-specific information field called `crdb_sql_type`, which contains a more exact representation of the column type. We opt to use that in this PR instead. As a note: I *think* this type isn't necessarily stable across CRDB versions, but (a) we only compare schemas using a "single CRDB version at a time", so, this should be fine, and (b) we aren't really flying through CRDB upgrades at this time. This PR is composed of two critical commits: 1. 05f6864 - add a regression test, observe that we failed these cases of "columns with mismatched types" 2. f8ec8d6 - fix the regression test by comparing `crdb_sql_type`. <details> <summary>Details</summary> ## Problem The schema migration tests (`validate_migration_from_base_version`) compare a database built by applying migrations against one built from `dbinit.sql`, using a snapshot of `information_schema`. For columns, that snapshot used only `data_type`, which is the base type name: it omits precision, length, interval fields, and array element types. For example: | declared | `data_type` | `crdb_sql_type` | |---|---|---| | `INTERVAL` | `interval` | `INTERVAL` | | `INTERVAL(0)` | `interval` | `INTERVAL(0)` | | `STRING(63)` | `text` | `STRING(63)` | | `INET[]` | `ARRAY` | `INET[]` | | `UUID[]` | `ARRAY` | `UUID[]` | These are not equivalent: `INTERVAL(0)` rounds fractional seconds, and `STRING(63)` rejects longer values. So a fresh install and an upgraded rack could end up with different column types without any test failing. This is how the `INTERVAL` vs `INTERVAL(0)` mismatch in #11565 went unnoticed (#11769). ## Changes 1. Add `compare_table_differing_column_type_modifiers`. For each pair of types, one database creates the column with `CREATE TABLE`, and the other adds it with `ALTER TABLE ... ADD COLUMN` (as a migration would). It then checks whether `pretty_assert_eq` (the same comparison used by the migration tests) reports a difference. Identical-type pairs act as a control. On its own, this commit fails: none of the six differing pairs are detected. 3. Also compare `crdb_sql_type`, a CockroachDB-specific column of `information_schema.columns` containing the full type ([docs](https://www.cockroachlabs.com/docs/v23.1/information-schema#columns)). Adding the individual precision columns (`datetime_precision`, `character_maximum_length`, `numeric_precision`, ...) was considered, but they're redundant with `crdb_sql_type` and incomplete: e.g., CockroachDB reports `interval_type` as `NULL`, so `INTERVAL` vs `INTERVAL HOUR` is indistinguishable with those columns. ## Testing - All tests in `integration_tests::schema` pass, including `validate_migration_from_base_version`: existing migrations already match `dbinit.sql` under the stricter comparison, so no schema changes are needed. - The new test fails on the first commit and passes on the second. Fixes #11769 </details>
| // RFD 622: "A user-configurable timeout for the instance to finish stopping | ||
| // itself should be present, bounded between 1 and 3600 seconds, inclusive." |
There was a problem hiding this comment.
I know the RFD says 3600 seconds but I think there's some hesitance to immediately allowing such a long timeout without any kind of motivator. if waiting fifty minutes is not enough, but 56 is, that feels like it strains credulity? there's a real disruptive aspect to this too, if we're telling an operator "you must wait at least an hour for this sled to be clear of workloads so you can do maintenance on it, or risk causing data corruption for your users' applications by force-stopping instances configured to want graceful shutdown"
what do you think about picking a shorter time here, say, 15 minutes? since the longest default configurations I'd found were on the order of ten and even that seemed pretty long to people on bug trackers. if someone really needs a longer time here, then they really ought to tell us and we work together on the problem at hand. and there's a nudge to do that, since someone who needs a longer timeout would get told by the API and if it truly is necessary then we can figure out a suitable path forward for the product. that could be "we should really get live migration shipped so you don't need to stop instances", and the corresponding issues to take care of there.
as-is an hour makes me ask why not two hours? a day? it seems like it would be good if we can go back to the RFD with the more incremental stance - IIUC the numbers in the RFD were just because they were most comparable to other systems, rather than a strongly-considered choice.
There was a problem hiding this comment.
i suppose we can set it to 15 and then decide whether to increase it based on customers who bring a compelling case for it. much easier than trying to reduce it to 15 after it's been deployed and people have 30min timeouts in their databases, certainly
There was a problem hiding this comment.
+1, I think a much shorter timeout is prudent until we hear customers clamoring for it for some reason (and it would be good to understand why).
…ance update, and only support instance view of non-default policy on up-to-date clients
| #[allow(clippy::derivable_impls)] // impl_enum_type doesn't afford #[default] | ||
| impl Default for InstanceShutdownAction { | ||
| fn default() -> Self { | ||
| Self::HardOff |
There was a problem hiding this comment.
I think the vast majority of launches would want PowerButton as the shutdown policy.
In the solutions eng huddle, we were talking about updating basically all of our integrations to set the shutdown policy to PowerButton.
There was a problem hiding this comment.
others can confirm, but I think this defaults to HardOff for backwards compatibility
There was a problem hiding this comment.
That's what I thought, but I was curious if that's worth maintaining in this case. I still think most launches would rather switch to PowerButton rather than maintaining HardOff.
I would guess that we will eventually change this to PowerButton as the default at some point since that's what users will expect intuitively (and what AWS, GCP, and Azure default to). Are there specific cases that we're worried about? The only downside I could see is VMs taking slightly longer to turn off, but if we also default the grace period to 30s (or something fairly low), that isn't too much of a slow down.
There was a problem hiding this comment.
in fact it was originally PowerButton (10 min) by default, but after this point (implementation complexities around what behaving consistently means when upgrading a rack w/ older instances vs. creating new ones, as well as the commitment of having an opinion of what the default timeout should be), we changed it to HardOff during review
There was a problem hiding this comment.
towards the end of last week I tried to digest the background and some motivations for defaults (and some points of comparison for grace windows) into https://rfd.shared.oxide.computer/rfd/622. here: yeah, this does not preclude us from changing the default in the future, but if there are specific applications that really need graceful stops always that's actually a pretty useful datapoint for the product at large.
for example, Packer expecting "stop" to get guests to flush caches feels obvious from a certain point of view. but if, in the future, there's a more "ephemeral" instance type (microvms? "spot-like" opportunistic execution?), should that stop behavior graceful or not? i would say it is a bug generator for Packer to use such an "ephemeral" instance, because they might not get a graceful stop or if they do it might be well below any expected completion time. so if we're building integrations it's really valuable for us to understand what the integration needs so we continue to integrate it properly in the face of new product features.
(30s would be way too short for a default IMO, unfortunately. I've tried to talk about Windows a bit in the RFD, but 30s is the perfect danger zone for Windows instances to get unceremoniously killed mid-update and turned into paperweights)
There was a problem hiding this comment.
Thanks for all the additional context! The one I was thinking of is Kubernete's Graceful Node Shutdown feature. This is something that "just works" in other clouds since the default is to gracefully shutdown VMs (with a long enough grace period).
I've tried to talk about Windows a bit in the RFD, but 30s is the perfect danger zone for Windows instances to get unceremoniously killed mid-update and turned into paperweights
that's fun...
There was a problem hiding this comment.
Looked through this just for additional context / knowledge sharing. One important comment around the API restrictions (lowering the timeout allowed) but nothing else.
I also left a comment on the propolis side that we should consider moving the shtudown timeout handling into sled-agent instead of having propolis do that: oxidecomputer/propolis#1217 (comment).
I don't know what the lift is there, and it doesn't seem super important to figure out before merging this.
| // RFD 622: "A user-configurable timeout for the instance to finish stopping | ||
| // itself should be present, bounded between 1 and 3600 seconds, inclusive." |
There was a problem hiding this comment.
+1, I think a much shorter timeout is prudent until we hear customers clamoring for it for some reason (and it would be good to understand why).
david-crespo
left a comment
There was a problem hiding this comment.
API and docs look good to me. Looks like you need to merge main because your API version got taken.
iximeow
left a comment
There was a problem hiding this comment.
I have a smattering of comments but the top-line is I'm down to click the button too. most important "we should probably not pub use sled_agent_types_versions in Nexus", and if we give this a moment (helios/deploy is helping in its own way..) I'll push a commit to fix that in this initial external API version, otherwise we can of course Just Fix It at any later point.
| } | ||
|
|
||
| impl UngracefulTermGraceTime { | ||
| fn set_stop_timeout(&mut self, method: InstanceShutdownPolicy) { |
There was a problem hiding this comment.
nit, and for emphasis: pick! we should not wait another however long on backed-up CI for this:
I'd probably call this update so that callers would read like force_stop_timer.update(), rather than force_stop_timer.set_stop_timeout()-which-sometimes-doesnt-set-anything
| #[derive(Default)] | ||
| struct UngracefulTermGraceTime { | ||
| timeout_duration: std::time::Duration, |
There was a problem hiding this comment.
might just be me, but Default and Duration always make me super itchy. is Duration::default.. 0 seconds? one second? something else? I don't think my opinion is super important here in isolation, but may make more sense in the context of my next comment below..
edit: alternatively, maybe this should be part of a tuple under stop_timeout, as this field isn't meaningful unless there is a timeout set and that's kinda what that option means?
There was a problem hiding this comment.
it's a little awkward, because method could also be part of the tuple, which gets back toward what that made me split this out... maybe i just do a Option<UngracefulTermGraceTimeInner>
|
|
||
| fn disarm(&mut self) { | ||
| if self.stop_req_method != InstanceShutdownPolicy::HardOff { | ||
| self.stop_timeout = None; |
There was a problem hiding this comment.
a disarmed UngracefulTermGraceTime at this point would have self.stop_req_method == PowerButton { whatever } && self.stop_timeout.is_none() - it would behave correctly but the struct's state at that point is a bit funky. incidentally, since we derive Default the "normal" stop_req_method had been PowerButton until that was made HardStop too.
since we only hold onto the stop method to ask "was this deadline established due to a graceful stop which we are willing to adjust", maybe we should only retain method != InstanceShutdownPolicy::HardOff when we're updating the internal bits of the struct? even so "what is an UngracefulTermGraceTime before a timeout is armed" is kind of nebulous to me.
| // If we receive a reboot request while a stop | ||
| // timeout is running that was initiated by an | ||
| // ACPI shutdown, disarm the timeout, but only | ||
| // if Propolis was still there and honored the | ||
| // request. |
There was a problem hiding this comment.
fwiw it's a bit confusing that we mention "was initiated by an ACPI shutdown" here, but that's only checked inside disarm() (the comment is right in effect but I was pretty confused for a second!)
more importantly for the check here, since we only disarm the timer after we've heard back from Propolis, we'll presumably yield on the .await and potentially see the timer fire while Propolis is handling a reboot which would have disarmed the timer. that's a pretty narrow window, I think, so I'm not really concerned about it but it makes me wonder if we could rephrase the conditions before/after the operation.
my thinking here is: there would have had to be an ACPI-style reboot that arrived right as Propolis was approaching a previously-established deadline from an ACPI-style shutdown .. that's a bit difficult to motivate myself into being concerned about, since the timing means it was really kind of a dice roll on if the ACPI-style shutdown stopped the instance just before the reboot arrived.
There was a problem hiding this comment.
i think we would either have to yield on the .await for at least STOP_GRACE_PERIOD (= 10 minutes) for this timer to fire, or expect a propolis that had been unresponsive for acpi_timeout_secs + STOP_GRACE_PERIOD to suddenly honor a last-moment reboot request, in order for that to be an issue.
it does make me think that if somehow we saw a successful-looking reboot request response from a propolis that had already consumed its configured acpi policy timeout and was into STOP_GRACE_PERIOD territory, we would disarm here, and maybe we shouldn't. i think per jordan's feedback, a more sophisticated timeout-handling mechanism that perhaps moves the timeout logic out of propolis's state driver could cover this case more robustly
| use crate::v1::instance::InstanceMigrationTargetParams; | ||
| use crate::v1::instance::VmmStateRequested as VmmStateRequestedV1; | ||
|
|
||
| /// Instance-stopping method choices |
There was a problem hiding this comment.
since this ends up being the public docs on InstanceShutdownPolicy we probably should say a bit more here..
There was a problem hiding this comment.
| pub use sled_agent_types_versions::v66::instance::{ | ||
| AcpiTimeoutSecs, InstanceShutdownPolicy, |
There was a problem hiding this comment.
re-using the client types from a different API here was very surprising - I was going to remark that we should leave a better comment on InstanceShutdownPolicy for ourselves reading later, that the doc comment on InstanceCreate::shutdown_policy seems great, and that we should probably just copy it before realizing that the sled-agent client type is the external API type.
I realize we've done this elsewhere (bgp_peer_src_addr does this with RouterPeerType in the external BgpPeer struct) but it definitely pokes a hole in my mental model that sled-agent's API is very internal!
There was a problem hiding this comment.
it was certainly a little strange for me too, but per #11565 (comment) they're at least both versioned.. though yeah, the difference in docstring audience is certainly a factor
| fn try_from(value: u64) -> Result<Self, Self::Error> { | ||
| if value < Self::MIN || value > Self::MAX { |
There was a problem hiding this comment.
I think this is because we're trying to keep schemars in line with what the column constraints actually are, right? since the constraint still has 3600 seconds as a maximum (as noted on the AcpiTimeoutSecs doc comment) at the very least we're checking different things in different places..
There was a problem hiding this comment.
whoops 💀 definitely wanna fix that once the whole waiting-for-buildomat situation clears up... but would want to get that in right away so as not to make a whole new schema revision about it, i think? maybe it's ok to eat the CI delay to not risk it
| /// CRDB. | ||
| pub fn shutdown_policy( | ||
| &self, | ||
| ) -> Result<instance_types::InstanceShutdownPolicy, Error> { |
There was a problem hiding this comment.
my AcpiTimeoutSecs comment is partially because I was at the bargaining stage of "Instance in the database is only fallibly convertible to API types" - I was hoping we could avoid having the fallible conversion by trusting the database has enforced the constraints we want on the API types. as-is, looking at a cursèd instance is impossible! paginated views including such an instance are also cursèd!
but I think even if we didn't check the timeout specifically, the relationship that "PowerButton implies a configured number" (and HardOff implies NULL, which we should probably check here too!) seems worth validating..
if the tyranny of |


This feature is intended to allow VM guest shutdowns to be triggered through the API and as part of sled evacuations for updates. An intent of this feature is to afford non-ephemeral instances a chance to cleanly unmount their filesystems, such that taking a new Oxide software release does not carry with it an unnecessary risk of filesystem corruption.
To quote RFD 622:
A note on terminology: ACPI calls the feature currently implemented on the Propolis side "Fixed Power Button". ACPI also specifies an altogether different feature that it calls "Graceful Shutdown Request" which is not presently implemented by Propolis. We may sometimes talk about "gracefully" stopping an instance, but for the sake of clarity, we may wish to be take care not to confuse operators and customers with the term "Graceful Shutdown" in documentation.
POST /v1/instancesnow takes ashutdown_policyparameter, eitherHardOfforPowerButtonwith a (mandatory) timeout between 1 and 3600 seconds. New instances with no shutdown policy specified in their create request will default to a policy of the pre-existing behavior: Hard-poweroff of the guest OS with no forewarning. Existing instances will also default tohard_offin the CRDB migration.PUT /v1/instances/{instance}will only change an instance'sshutdown_policyif one is explicitly included in the request, such that clients of an older API revision don't potentially clobber an existing instance's shutdown policy.POST /v1/instances/{instance}/stopandPOST /v1/instances/{instance}/rebootnow may be passedskip_os_shutdown=trueto force a hard-poweroff even if an instance has a soft shutdown policy. If an instance does have aPowerButtonshutdown policy, andskip_os_shutdownisn't passed, the instance will have its simulated Power Button pressed, and then following either the guest-initiated power event or the exhaustion of the configured timeout, the instance will be accordingly stopped or rebooted by Propolis.Further requests to stop or reboot an instance while it is still in the process of shutting down (i.e. the guest hasn't emitted a power event and the configured timeout hasn't expired) are permitted to change the instance's fate, i.e. whether or not Propolis will reboot the instance once it has stopped. However, changes to the shutdown policy's timeout will not be propagated to instances that are already shutting down; only the first timeout Propolis receives in a stop or reboot request is considered. Non-ACPI stop/reset requests, either through
skip_os_shutdownor changing to aHardOffpolicy, will be honored.The sled-agent's existing 10-minute propolis-zone-reaping timeout is extended by the configured timeout of a
PowerButton-type shutdown; that is to say, the zone will be forcibly terminated if propolis does not stop in the ACPI timeout plus 10 minutes. If a hard-stop request is sent during this timeout, and there are more than 10 minutes remaining, it will be replaced by a 10-minute timeout. If a reboot request is sent to propolis while it is still waiting for an ACPI shutdown, the zone-reaping timeout is dropped.