Repository navigation
feat(orchestration): real plans, partial job runs, queue and timeline views and trigger details in the UI - #1400
Merged
Merged
Conversation
…bal unique-name check Publication now rejects a job, job group, sensor or calendar schedule whose name is already used by any other project resource or orchestration declaration, with the same D007 conflict discovery raises for models, sources and the rest. The narrower duplicate-job and duplicate-trigger checks are removed; the object-identity check stays. build_orchestration_snapshot and compile_snapshot take the discovered project inputs so publication can build the full name set. Fixtures whose jobs reused model names are renamed, and DuckShop gains a name-collision edge case.
A name conflict already names both declarations and their files. It now ends with how to resolve it, and when a job, job group, sensor or calendar schedule is involved, says where that name comes from: a job, group or sensor takes it from its first argument, a calendar schedule from the function it decorates.
…onflicts The name check now gathers every declaration as a named record and then scans them once, instead of threading a mutable map through a helper. The first conflict found is the same.
A person with view on every node of a target can ask the location to run sqb plan --json on a build command. The server freezes the command as it would an ad-hoc run and queues it as location work, so a plan never becomes a run, fires a sensor or takes a build slot. The location runs the plan as a child process on the frozen node keys and posts the document back; the server redacts warehouse text in it. Only the requester reads it, and Python identity diffs need view.sql.
A build command on the plan page is now planned by the location with sqb plan against the warehouse: each model's action and reason, backfills, migrations, sources, seeds and warnings, behind a planner spinner. Viewers can see plans; Run still needs run.adhoc and a plan of the exact command. An error stays under the command over the last good plan instead of replacing the page.
A plan only reads warehouse metadata, and naming a warehouse for a run needs run.warehouse. Plans now drop --warehouse when the server freezes the request, so a viewer cannot make the location use another warehouse.
Not yet wired into commands: attach_jobs has no caller outside the graph crate. Graph tests cover job:, bare job names, groups, intersections, exclusions, expansion and failures.
`--select job:<name>`, or a bare job name, selects what the job publishes: its own selection minus its own exclusion, resolved on the same graph, and a group selects its jobs' union. It works in intersections, with `+` expansion and in `--exclude`. Each command imports the project's orchestration modules only when a selector part names a job or a bare name the graph lacks. Path selector endpoints, Dagster DAG selection and Python-node check selection reject jobs. Still to do: the server's ad-hoc planner, lineage anchors for bare job names.
The server's planner receives each published job and group's resolved selection, read from the run's snapshot, and attaches it only when a selector names a job. No Python runs on the server, and the frozen run still records exact node keys, which authorization checks as before.
Like publication, the CLI's job catalog keeps a job or job group with targets only when the command's target is one of them.
A job run request can carry --select and --exclude selectors. The server resolves them with the ad-hoc planner over the same publication and keeps only the job's own nodes, so the run stays a run of that job with its concurrency groups and a write set limited to what it selects. Job groups, flag tokens and callers without a planner are refused. sqb admin request-run gains --select and --exclude.
Selectors resolve over the whole publication, so a person who narrows a job must be able to list it first, as for ad-hoc runs. Otherwise the refusals would tell them which nodes exist.
A run of part of a job reports the select and exclude selectors its requester typed, read from the authored request; nothing else of that request is returned.
Picking nodes on a job's page offers running only those or the rest of the job. The plan page keeps the job beside the command, plans the narrowed build on the location and starts a run of that job. Runs of part of a job are marked in the run lists and on the run page.
…k and the scoped work read Plan reads and ad-hoc checks both decided whether a person holds one action on every node of the target's current snapshot; plan and connection-test reads both scoped a work record to the requested location and target. Each now has one owner. The plan routes join the other person route handlers in the duplicate-code allowlist.
The narrowing is frozen with the run request, which a rerun copies, so a rerun of part of a job lists the same selectors as the run it repeats.
Publication records each job's selectors as written, its targets and the project-relative file and line that declared it; declarations outside the project keep their location private. The person job read returns them with the cursor type its selected nodes share, and the job page and list show the selection, excludes, targets and source location.
… defines it location_command only imported declaration_site to register it. The native host registration now adds it next to the host table, on the same _native module.
…as a person A person who may list the target can read a sensor's committed cursor, its recent checks projected from their recorded phases, and the runs it requested by run key. Custom cursor values and run keys pass through secret redaction. The trigger page shows where the sensor is up to, its recent checks and the runs it started, and flags code changed since the last check.
…ence A sensor's custom cursor values and run keys are computed from data its code read, so the sensor read returns them only to people who hold view.evidence on every current node; others still see positions, checks and runs. The unaudited coverage check that change notices use now takes the action to check.
…s past the newest 1,000 A rerun can name nodes of the source run; it needs view on every current node before the names resolve, refuses names outside the source selection, and cannot be combined with rerun from failure. The runs listing takes a before cursor (queue time and run ID), and the UI loads older windows of 1,000 when paging reaches the end of the loaded ones. The runs page's Active and Queued tabs can cancel every matching run, and a run's detail page reruns the nodes selected on its timeline.
The person runs read takes queued=true and returns the location and target's queued runs in claim order, each with its position and waits: its group run or an upstream job of the group, a running run of the same job, write claims, held concurrency limits and earlier runs on the same limit. The waits come from the scheduler's own admission helpers, and a run in another location or target is counted, never named. The Runs page's Queue view and a queued run's detail page show them.
The Runs page's Timeline view and the overview's timeline draw top-level job runs overlapping the window, each running schedule's next due tick as long as the job's median successful run, and ticks an existing run covered. Runs load back to the window's start, at most ten windows of 1,000.
The job page's launch form offers a cursor window only when the job has incremental models or loaders, shows examples for the job's cursor type, and keeps a window and a full refresh apart, as the server requires.
Runs read their backfill window from the frozen request, as they read a narrowing, and nothing else of the request. The run header shows it as "window · start → end" and the runs list marks the row.
… triggers in the sidebar The sidebar badges come from the existing listings, read once per location target. The nodes listing goes first, so a person who can't list the whole target gets one refusal and no badges.
The repository allows only doc comments.
GET /api/control/platform/events returns stored platform events newest first, without heartbeats. Server-wide events are visible to anyone signed in; an event about a location target (a broken location, an overdue or low-horizon tick, a failed launch) only to people who may list that target, checked as the problems read does. The Locations page's server status and a location's History tab list them, with an open problem marking its newest stored event as ongoing.
…not see Each page of /api/control/platform/events now carries the storage ID the next scan continues before, so a page whose events are all hidden no longer ends the history. The UI keeps scanning until enough of the location's events match. A trigger's or run's event whose row is gone is left out instead of being shown as server-wide.
…h coverage on catalog nodes The Warehouse tab reads the node's physical catalog observation, its statistics samples and, for microbatch models, its recorded coverage. Samples are cached per relation, so a refresh reads only newer ones. The sync records no declared types or creation times, so the tab lists warehouse types without comparing them and leaves those times out.
The server pages statistics oldest first from its retention cutoff, so a first read that stopped after ten pages could show a row count weeks old. It now starts 900 hours back, which ten pages always cover at one sample an hour.
The banner said the location was healthy while the overview was still loading, then switched to its problems. It now waits, and the group panel no longer shows and then disappears.
…g bar A click kept the current page for up to 1.5 seconds while the next one loaded, with a bar along the top. Pages now switch after at most 200 ms and show remembered data or their own loading state, and the bar is gone.
Connections shows every location target the person may list whole, with its newest connection check, a manual test, where its state tables are read from and each table's newest sync. Locations publish no connection settings, so the page shows what their checks and syncs report.
Each refused health read is audited, so Connections no longer asks again on every reload, signal or manual test for a target the person may not list whole.
GET /api/admin/server reports the server version, storage, when the installation and its schema were created, the sign-in mode, whether this server runs the scheduler, and how long each kind of record is kept. It needs user.manage everywhere, like the people list. The Settings page shows it to administrators and only the person's own access to others.
The people's problems read now lists running calendar schedules whose last date is at most a day away, by the watchdog's own threshold, for targets the person lists. The Locations problems show them per location.
The person trigger listing now carries up to five due times from next_due_at on, worked out by the scheduler's own cron and calendar rules, so the UI no longer guesses later cron times. The trigger page also states the catch-up policy: one tick for the latest missed time.
Publication records the project-relative file and line of each schedule, calendar and sensor, and the targets it is limited to; a job's schedule takes its job's. A sensor's fingerprint leaves both out, so moving its code does not flag the definition as changed. The trigger page shows them.
… path Links and navigations now go through $lib/paths, which adds the base path the UI is built for. sqb server serves it at the root as before; a build with SQB_UI_BASE set, such as a static demo at /demo, keeps every link under that path.
…ayout A tick's outcome shows as a coloured dot beside plain text, so a column of runs that started normally no longer reads as a wall of green. Digits are equal width everywhere, relative times under a minute move in steps of ten seconds, and the sensor's cursor age keeps room for its longest value, so the controls beside a ticking time stay put.
…s-ui # Conflicts: # crates/sqlbuild-python/src/bindings/_helpers/native_host/constants.rs # crates/sqlbuild-python/src/bindings/_helpers/native_host/orchestration_declarations.rs
kvlonge
enabled auto-merge (squash)
November 10, 2026 02:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The UI guessed at plans, could not run part of a job, hid sensor and trigger state, and had no queue or timeline view.
Changes
--warehousenever reaches it.job:<name>selectors, partial runs and reruns (view on every node).before_queued_atpaging, queue, timeline, backfill window (RunRecord.window).API changes are additive; TS client regenerated. Upgrade the server before its locations and do not downgrade it.
Verification
make check-cipasses on this tree before the last main merge, which adds only a dense guard constant.DuckShop: no compiler or project shape is involved.
BEGIN_COMMIT_OVERRIDE
feat(ui): the plan page shows the location's real sqb plan
feat(selectors): select jobs and job groups with
job:<name>or a bare namefeat(runs): run part of a published job; reruns keep the narrowing
feat(orchestration): job pages show selectors, targets and declaring file and line
feat(orchestration): sensor pages show the cursor, recent checks and runs started
feat(runs): rerun selected nodes, cancel matching runs, page past 1,000
feat(ui): queue, runs timeline and backfill windows
feat(ui): sidebar counts, catalog warehouse objects and microbatch coverage
feat(platform): platform events, server settings, low calendars and connections
feat(triggers): show each trigger's declaration site, targets and next due times
fix(ui): tick outcomes show as a coloured dot; relative times no longer shift the layout
END_COMMIT_OVERRIDE