Skip to content

[Bug]: [Presets] dstack preset delete cannot delete interrupted or failed creations #4169

Description

@peterschmidt85

Steps to reproduce

  1. Start a preset creation: dstack apply -f preset.dstack.yml. Preset configurations do not support -d, so it holds the terminal.
  2. Stop it before it finishes: Ctrl+C and confirm the prompt, or dstack preset stop <id> -y from a second terminal. A creation that failed on its own works the same way.
  3. Confirm the session is listed: dstack preset list -a.
  4. Delete it: dstack preset delete <id> -y, or by name if it has one.

Actual behaviour

$ dstack preset list -a
 NAME           ID        BASE             CONSTRAINTS    BENCHMARK  STATUS             SUBMITTED
 smoke-qwen06b  8a2d894e  Qwen/Qwen3-0.6B  io=256/64 c=4             interrupted (0/2)  8 mins ago

$ dstack preset delete 8a2d894e
Preset '8a2d894e' does not exist

$ dstack preset delete smoke-qwen06b
Preset 'smoke-qwen06b' does not exist

Neither the ID nor the name works, and --base and --repo skip these sessions too. The only way to remove one is rm -rf ~/.dstack/presets/<id>, which leaves the /tmp/dpe-* workspace alias symlink behind (workspace.py:145-153, removed only by remove_agent_workspace).

Reproduced on master 0dee238d4 with an interrupted and a failed session.

Expected behaviour

Interrupted and failed creations delete like any other preset:

$ dstack preset delete 8a2d894e
Deleted preset 8a2d894e for Qwen/Qwen3-0.6B

A creation that is still trialing or verifying is refused because it is in use, the way dstack refuses to delete a fleet or volume that is in use, rather than reported as non-existent:

$ dstack preset delete 8a2d894e
Failed to delete preset 8a2d894e. Preset creation is in use. Stop it with `dstack preset stop 8a2d894e`

Cause

_delete resolves its argument with find_by_id_or_name (preset.py:363), which only matches ~/.dstack/presets/<id>/preset.yml (store.py:67-69, store.py:38). That file is written only after a creation is verified (create.py:682), so every creation that never got there is invisible to delete.

Two more places need attention for a complete fix:

  • PresetStore.delete refuses a directory without preset.yml (store.py:128-129), so resolving the reference is not enough on its own.
  • _delete ignores the return value of store.delete and prints Deleted ... unconditionally (preset.py:386-388), so a partial fix would report success while removing nothing.

For reference, dstack preset get already reads unfinished sessions through _get_unfinished_preset (preset.py:391-406), and stop, logs, and resume resolve them with resolve_session_ref (session.py:491-499).

The on-disk statuses are running, interrupted, success, and failed (models/preset_agent.py:69); trialing and verifying are display labels (output.py:14-19). "In use" therefore has to mean session_process_alive (session.py:359-367), the predicate list already uses, or a stale running left by a killed CLI would stay undeletable, which is this same bug again.

Open questions

  • Should delete reconcile first? _delete never calls _reconcile (unlike preset.py:222, 260, 333), so a detached session whose agent already finished would be deleted rather than finalized.
  • Should --base and --repo cover unfinished sessions? _list already matches them via _session_matches_model (preset.py:480-492).
  • Name precedence between a verified preset and a session's claimed name is unspecified.
  • Sessions interrupted without dstack preset stop (killed CLI, reboot) may still have runs on the server; _delete builds no API client today (preset.py:359-360).
  • mkdocs/docs/reference/cli/dstack/preset.md:135 and mkdocs/docs/concepts/presets.md:281-288 describe only the verified case and need updating with the fix.
  • Compounds with [Bug]: [Presets] Any failure during resume marks the creation permanently failed #4164: a resume-time exception marks a session failed, which is exactly a state this bug makes undeletable.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions