Skip to content

HPC-style architecture: design docs and component contracts - #126

Merged
blobcode merged 3 commits into
mainfrom
docs/hpc-style-architecture
Oct 28, 2026
Merged

blobcode merged 3 commits into
mainfrom
docs/hpc-style-architecture

Conversation

@BrandonwLii

Copy link
Copy Markdown
Collaborator

What this is

This PR writes down the architecture redesign agreed on October 26. Mist moves from container-image jobs to an HPC-style model, and docs only change; no code. The design:

  • Folders: every person gets a folder in each of their team's folders; teammates can read it, other teams can't see it.
  • Login: SSH sessions move files and manage jobs only; code runs only inside jobs.
  • Jobs: batch and short interactive jobs run as the researcher's own Linux user, in their own folder. No Dockerfiles, no mounts.
  • Scheduling: Mist's own scheduler decides what runs. Jobs are started through a pluggable executor: K3s first, a native systemd executor later.

Start with ARCHITECTURE.md. Each component has a contract doc in docs/agents/architecture/.

Files

File Contents
ARCHITECTURE.md Overview, proposed owners, core concepts, flows, who owns which data, code layout, milestones M1–M5, open questions
docs/agents/architecture/decisions.md 17 decisions, each with its context and consequences
identity-and-accounts.md Misty stub, UID/GID ranges, sign-up, SSH keys, provisioner
storage.md /mist layout and permissions, volumes, scratch, usage, backups
login-gateway.md Restricted SSH, identity relay, mist CLI
api.md Conventions, authentication, job request and response formats
scheduler.md Partitions, QoS, limits, states, scheduling cycle, priority and fairshare, backfill
executors.md Executor interface and guarantees, mist-job-entry, K3s and node executors
runtime-environment.md What a job sees: software, environment variables, caches, outputs, signals
portal.md, deployment-and-network.md Portal pages; hosts, networks, secrets, adding nodes
docs/agents/plans/jobs-and-scheduling.md Implementation plan for the jobs and scheduling layer
README.md Pointer to the new design

What reviewers should check

  • Owners: confirm or reassign your row in the components table. The login gateway has no owner yet.
  • Your component doc: read the "Work by milestone" and "Open questions" sections.
  • Decisions: push back by proposing a new entry in decisions.md that supersedes the old one, rather than editing it.

Not in this PR

  • Human docs: docs/content/ still describes the old platform and needs rewriting by its human owners. The "What researchers see" sections in storage.md and runtime-environment.md are meant as source material.
  • Hardware and security tasks:
    • move the direct link to the 10 GbE ports
    • rotate the shared host passwords and turn off password SSH
    • check what ollama and the NVIDIA box's second SSD (sdb) are used for

🤖 Generated with Claude Code

Describe the redesign agreed on October 26: per-team folders, restricted
SSH login, and jobs that run as the researcher's own UID under Mist's own
scheduler, with a pluggable executor (K3s first, native node executor
later). Adds the system overview, one contract doc per component, a
decision log, and the jobs/scheduling implementation plan. README points
to the new design and notes that docs/content describes the old platform.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

BrandonwLii and others added 2 commits October 26, 2026 14:20
ARCHITECTURE.md gains an owner-boundary diagram with a table of the
contracts between areas, and an end-to-end sequence of one researcher
going from sign-up to results. Each component doc gains one diagram of
how it fits: identity data flow, storage paths, gateway dispatch, API
routing, the scheduling cycle, both executors, the inside of a job,
portal pages to endpoints, and the network topology.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tighten component contracts (API, scheduler, executors, identity, storage,
login gateway, network) and add implementation-contracts.md with release
gates and invariants. Add decisions D18–D23 and update the jobs plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@blobcode
blobcode merged commit 49ce02a into main Oct 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants