Repository navigation
Description
Question
Run submission has never completed end to end against a live Ensemble Manager. #85 and #88 each
fixed a defect on this path, and neither fix has been exercised against a real server. This is
the last item of #76's checklist that #76 did not close.
It is not the SOW acceptance criterion — §8 is the single-form model input config, which
passes (#76). This is about not shipping an unproven path.
What is already settled, so this is a short walk
Measured while resolving #76, signed in against TACC:
- The routes are exactly what ui-react submits runs to /executionEngines/localex, a route that does not exist #88 predicted. TACC's Ensemble Manager publishes
/executionEngines/tapis,/executionsand/executionsLocal— three separate mounts, no
shared prefix.executionEnginePathmaps them correctly. EXECUTION_ENGINE: tapisis right for TACC. Its live Lit config at
mint.tacc.utexas.edu/config.jssetsREACT_APP_EXECUTION_ENGINE = "tapis". The constraint
ui-react submits runs to /executionEngines/localex, a route that does not exist #88 put on Flip mint.tacc.utexas.edu to ui-react #77 stands.- The Tapis token attaches. Every Hasura call from the signed-in app carried
Authorization: Bearer …, so the Ensemble Manager calls send no Authorization header — localStorage key mismatch #85 fix holds in the browser. - The thread wizard renders against TACC's real problem statements, tasks and sub-tasks.
The one caveat worth knowing before you run it
TACC holds 264 executions: 263 localex and 1 tapis. So the Tapis submission path has
essentially never been used there, by Lit or anything else. A failure may well be TACC's Tapis
configuration rather than ui-react.
Why this is not done yet
Submitting creates a real job on TACC's cluster. That is a bigger side effect than the catalog
test rows #76 wrote and removed, and it was not covered by the go-ahead for those. It needs an
explicit decision on whether to fire a real run at the client's production deployment, and which
model to fire.
End to end testing / hello world file input test is the obvious candidate — it is a
sleep-and-echo fixture, it is the config #76 used, and it has a required and an optional input.
What to check when it runs
- Submit returns 2xx, not the 500 that Ensemble Manager calls send no Authorization header — localStorage key mismatch #85 showed is how an anonymous submit surfaces.
- The execution row appears with
execution_engine: tapis. - The Runs step polls status without a 401 — the other half of the Ensemble Manager calls send no Authorization header — localStorage key mismatch #85 fix.
- The Results step resolves the output location. Note
MintResultsreads the engine off the
execution row rather than config, and itslocalex/wingspath rewrite is dead code, so it
should need nothing.
Activity
mosoriob commentedon Aug 9, 2026
Blocked. No run was submitted. The wizard cannot reach the Runs step, so the submit path is
still unproven. Now blocked by #94.
Run signed in as
mosorio@portalsonlocalhost:3000, branchdevelop(4f7debf+1d1ada9),with
ENSEMBLE_MANAGER_APIpointed at TACC.Why it is blocked
The Datasets step cannot be satisfied, and Runs stays locked until every required input has a
dataset. Two separate faults, and only the second is ours.
1. The chosen model has no data at TACC.
hello world file input testneedsland_surface_terrain~left_channel~stream_reach_water__volume_flux. TACC's CKAN holds 215packages; 33 carry annotations, covering 40 distinct standard variables. None is that one. So
picking a different model was needed anyway — this ticket's candidate cannot run at TACC.
2. The dataset lookup is broken for almost every variable — #94.
ui-reactsearches CKANfree text for the variable name. CKAN does not index the
mint_standard_variablesresource field.Lit filters on that field instead and never sends
q. Measured:ui-reactreturns exactly theannotated datasets for 3 of 40 variables. 28 return zero. Some return only false positives.
Full evidence on #94.
Fault 2 means changing the model would not have rescued this ticket. Both faults had to be
found before the run could be attempted.
What was confirmed, signed in against TACC
These held, so the #85 and #88 fixes look right up to the point the run stops:
/executionEngines/tapis→ 401(exists, wants auth),
/executionEngines/localex→ 404,/executionsLocal→ 400,/executions→ 400. The pre-ui-react submits runs to /executionEngines/localex, a route that does not exist #88 code posted to the 404.EXECUTION_ENGINE: tapisis right for TACC. Its live Lit config atmint.tacc.utexas.edu/config.jssetsREACT_APP_EXECUTION_ENGINE = "tapis". The EnsembleManager is
https://ensemble-manager.mint.tacc.utexas.edu/v1.localhost:3000. The OAuth redirect carriesclient_id=mint-localhost-3000and
redirect_uri=http://localhost:3000/oauth2/callback. Tapis now enforces MFA.model picker finds
hello world file input test.Still unproven
Everything this ticket exists to check: the 2xx on submit, the execution row with
execution_engine: tapis, the Runs poll without a 401, and the Results output location.Note for whoever takes this next
Pick a model whose input variable is annotated at TACC. Candidates with real data:
groundwater__hydraulic_head(6 datasets),groundwater__recharge_volume_flux(5),aquifer__hydraulic_conductivity(4). Do this after #94, or the wizard will not find them.Also worth knowing: the region filter defaults to California, and widening it to "any region"
changes nothing here — the constraint is the annotation, not geography.
State left behind
I created sub-task
wayfinder-92 tapis submit checkunder problem statementtestto avoidediting Max's existing sub-tasks, then deleted it. TACC's catalog is back to its prior state. No
job was submitted to the cluster.
mosoriob commentedon Aug 9, 2026
A local MicroK8s dev cluster is available as a rehearsal target for this ticket.
Its Ensemble Manager runs
execution_engine: tapis, the same backend TACC runs. So it exercises the same submission route. Its data catalog is also TACC's CKAN, not a local one.It also confirms #88 on real infrastructure.
/v1/executionEngines/localexreturns 404./v1/executionsLocal,/v1/executionsand/v1/executionEngines/tapisall exist.It does not unblock this ticket. #94 is a client-side matching fault, so the Datasets step stays empty against any cluster. #94 must land first.
One limit on the rehearsal. The dev cluster hostnames are not publicly resolvable, so Tapis refuses to create job subscriptions against them. The values file sends callbacks to a webhook inspector instead. You can rehearse submission. Job status callbacks do not return to the Ensemble Manager.
mosoriob commentedon Aug 9, 2026
Blocked again. No run was submitted. The wizard cannot reach the Runs step. This time the
Models step is the wall, not the Datasets step.
Go-ahead from Max was: rehearse on the mint.local dev cluster first, then repeat at TACC.
Model: HAND Setup for Hays catchment. Neither happened, for the reason below.
Run signed in as
mosorio@portalsonlocalhost:3000, branchdevelop(7da6df3), pointed atthe dev cluster over
http://— its ingress serves a self-signed certificate that Chrome refusesfor XHR, and its
httplistener answers the same routes with no redirect.Why it is blocked: #98
The Models step cannot see a single model that has data.
The wizard's model tree query filters the software table on
type _eq sdm#Model. That is thegeneric type. The catalog mostly uses the exact subtypes —
EmpiricalModel,Theory-GuidedModel,CoupledModel. HAND is anEmpiricalModel, so all 44 of itsconfigurations at TACC are hidden.
Measured at TACC:
So the choice of HAND was not the problem — no wizard-visible model could have run. Full evidence
and the three affected queries are on
#98, which now blocks this ticket and
#77.
Which model to use once #98 lands
HAND Setup for Hays catchmentstill holds. One required input,input-dem, formattif,standard variable
land_surface__elevation.TACC has 2 annotated DEM packages, and neither is a clean fit:
ntgam-trinity-woodbine-v301— Texas, format ZIPbethel-elevation— Alaska, formatgeotifMintDatasetsmatches datatype with===, so both land in the "does not match the inputdatatype" list rather than the main one. They stay selectable. Under a Texas region only the
first survives
ext_bbox. The run is scientifically meaningless. It exercises the submit path,which is all this ticket asks for.
Second defect found: #99
Deleting a problem statement in the app removes it from the list but leaves the row in the
database, with its task and thread. The three delete mutations delete the provenance before the
row, and the
userdelete permission depends on that provenance. So the delete matches 0 rowsand returns no error. Same class as #91: a
write that is valid against the admin schema and fails under the
userrole.Filed as #99, blocking
#77.
What was confirmed
/v1/executionEngines/tapis→ 401,/executionEngines/localex→ 404,/executionsand/executionsLocal→ 415. Same threeseparate mounts #88 predicted, on real
infrastructure.
problem statements as TACC.
Still unproven
Everything this ticket exists to check: the 2xx on submit, the execution row with
execution_engine: tapis, the Runs poll without a 401, and the Results output location.State left behind
Nothing at TACC. All reads only.
On the dev cluster, one orphan from #99 that I could not remove — the
userrole has nopermission left for it, and reading the cluster admin secret was denied:
ui-react/public/env-config.jswas pointed at the dev cluster during the run and is restored.mosoriob commentedon Aug 9, 2026
Unblocked by #98 — plus one constraint the fix exposed
#98 is fixed (PR #101). The Models step now offers 175 leaf configurations instead of 18, and all 44 HAND configurations are visible.
The Models step has a second filter, and it can still show you nothing.
Beyond the model tree,
ModelsStepkeeps only rows whose outputs carry the thread'sresponse_variable_id. Measured across the now-visible catalog:A thread whose indicator is not one of those 146 renders
0 of 175 modelsandNo models found.with no explanation. That is exactly what a live check hit: threadnzGQkhtRsudfSEHkDGct(Maximize Pumping, Texas) carriestotal_water_storage, a legacy lowercase id no configuration at TACC produces. Every live indicator uses CSDMS-style names.Pick the indicator deliberately. These three each land on all 44 HAND configurations:
RECHARGE_VOLUME_FLUXDRAWDOWNSTORAGE_VOLUMEAlso still standing, from #94:
ext_bboxhides the third of TACC's annotated catalog that has no spatial extent, and the California default keeps 0 of 33. Choose Texas.mosoriob commentedon Aug 9, 2026
Correction to the note above: the indicator constraint now has its own ticket, #103 — the Variables step offers 668 standard variables where only 147 are produced by any configuration. #103 blocks #77, not this ticket: #92 can steer around it by picking an indicator from the 147. The three named above still hold.
mosoriob commentedon Aug 9, 2026
Blocked again. No run was submitted. This is the fourth wall, and the first one that
is not a wrong query. The wizard dead-ends at Parameters, one step further in than
last time.
Target was TACC directly, not the dev cluster. Max approved that: the dev-cluster
rehearsal cannot prove the Runs poll or Results, because Tapis refuses job subscriptions
against
*.mint.local.Run signed in as
mosorio@portalsonlocalhost:3000, branchdevelop(5f637d8),ENSEMBLE_MANAGER_API=https://ensemble-manager.mint.tacc.utexas.edu/v1,EXECUTION_ENGINE=tapis. No MFA was needed — a valid token was still in theautomation profile.
Why it is blocked: #104
The Parameters, Runs and Results steps are never fed data.
threadExecutionDatainMintThread.tsxstartsnull, no query loads it, and both setters bail out onnull.So
execData.modelsandexecData.execution_summaryare always empty. Parameters showsPlease select model(s) first.and has no Continue button. Runs and Results stay locked.This is independent of #85 and #88. Those fixed the submit call. The wizard never
reaches it. Full evidence on #104, which now blocks this ticket and #77.
What was confirmed, and it is a lot
The three earlier fixes all hold against TACC's live catalog:
HAND Setup for Hays catchmentneeds 1 input,land_surface__elevation, as predicted.packages this ticket named, both badged
[partial]on datatype. The wizard clearedthe Datasets step for the first time:
1 / 1 inputs.1 / 1again. The write path is fine.So the route from Framing to Datasets is now clean end to end at TACC.
Two smaller things seen on the way
Bethel Elevationis an Alaska package, and it was offered under a Texas framing.Thread datasets: ui-react searches CKAN free text instead of the mint_standard_variables field #94 made
findDatasetsread the whole catalog, so the region no longer narrows theDatasets step. That is the opposite failure to the one Datasets step: show the datasets the region filter hides, instead of dropping them #97 describes. Worth folding into
Datasets step: show the datasets the region filter hides, instead of dropping them #97, which owns the region rule.
storage_volume,drawdown,recharge_volume_flux, repeated). Cosmetic. Not filed.Still unproven
Everything this ticket exists to check: the 2xx on submit, the execution row with
execution_engine: tapis, the Runs poll without a 401, and the Results output location.State left behind at TACC — needs your hand
Max approved creating a fresh problem statement and accepting the orphan. I created it.
I could not delete it: the safety classifier blocks delete clicks against production,
which is correct. So it is still visible in TACC's Texas list. Please remove these rows:
Note #99 means a delete from the app will not actually remove them. Use the admin secret.
Nothing else at TACC was written.
ui-react/public/env-config.jswas pointed at TACC'sEnsemble Manager during the run and is restored.
mosoriob commentedon Aug 9, 2026
Unblocked by #104 (PR #105). One constraint lands here.
The wizard can now reach Parameters, Runs and Results:
GetThreadExecutionloads themodels, bindings and execution summaries, and the Datasets and Parameters steps persist
what they collect.
#104 could not verify the two writes through the UI, so this ticket must. Both
documents are proven valid for the
userrole at TACC —UpdateThreadDatais the one Litalready uses, and
UpdateThreadParameterswas probed toconstraint-violationrather thanvalidation-failed, so it validates and executes. What is unproven is the app drivingthem. Confirm on the way through:
n / n inputsand the rail must readAll inputs assigned. Before fix(ui-react): load the thread execution pipeline from Hasura (#104) #105 it read0 / nand
Pending, because Continue deleted the bindings instead of writing them.thread_model_execution_summaryrow. If Runs saysPlease setup some models first.after saving parameters, the write did not land.
Two things to expect while walking it:
total_runsis now input resources x parameter values, not parameter values alone.A dataslice of 6 files with no adjustable parameters is 6 runs. That is what TACC's own
summaries hold, so a smaller number on the Runs step means the resource count is wrong.
deliberately (Thread datasets: ui-react searches CKAN free text instead of the mint_standard_variables field #94):
ext_bboxhides the third of TACC's annotated catalog with nospatial extent.
Resource rows are now keyed by MD5 of the URL, matching what Lit wrote — so re-saving a
binding upserts the existing row rather than adding a duplicate.
mosoriob commentedon Aug 9, 2026
The submit call is proven. The model was not runnable.
ui-reactsubmitted correctly. Measured at TACC, signed in asmosorio@portals, branchdevelop(d78aaa6),EXECUTION_ENGINE = tapis:Route correct, so #88 holds. Token
attached and accepted, no 401, so #85
holds. The server answered 500.
The 500 is not ours.
HAND Setup for Hays catchmentcarries a localex ZIP component, andthe Ensemble Manager's Tapis path parses that URL as JSON. Filed as
#108. Proved by negative control: the
submit was repeated with a GeoTIFF input instead of the ZIP input and the response was
identical byte for byte, so the input is not involved.
The two writes #104 could not prove: both pass
1 / 1 inputs, and areload still read
1 / 1andAll inputs assigned. Athread_model_iorow carries adataslice id.
thread_model_execution_summaryappeared withtotal_runs, and the Runs step then listed the model and offeredSend Runs.The wizard reached the Runs step for the first time.
Pick a Tapis-ready model — this is the constraint that decides the ticket
Of TACC's 315 configurations, 17 point at a Tapis app. Crossed with CKAN annotations, on
the
portalstenant:corpus_nlpcorpus_nlphello world file input testland_surface_terrain~left_channel~stream_reach_water__volume_fluxThis ticket's original candidate still cannot run. No TACC dataset carries its input
variable, exactly as the first attempt found.
Next attempt uses
SUBSIDE Allocation Specification, on a new thread. A new thread isneeded because #107 blocks changing the
model on a thread that already holds bindings.
Two defects found on the way
any indicator. It writes a standard-variable URI into a column whose foreign key points at
the legacy
variabletable. 0 of 668 options are valid. Steered around: with noindicator every model stays available.
nothing and swallows the error.
delete_thread_modelis refused by fourON DELETE RESTRICTforeign keys once the thread holds any execution data.
Both block #77.
Smaller things seen
2 runs (1 input resources x 1 parameters). The arithmetic does notmatch its own factors. The dataslice held 2 resources, one of them with an empty URL and
id
d41d8cd98f00b204e9800998ecf8427e, the MD5 of the empty string. Every URL-less resourcein the catalog collides on that one id.
9cf3607, identical tomint-ensemble-managermain. TACC's Hasura is current too:0d74aa1, identical tographql_enginemain. Soneither wall is a stale deployment.
mosoriob commentedon Aug 9, 2026
A run went from
ui-reactto TACC's cluster and finished. The path is proven.Model SUBSIDE Allocation Specification, thread
jx8ibyilmsmg1pvt, signed in asmosorio@portals, branchdevelop(d78aaa6),EXECUTION_ENGINE = tapis.One of the two runs ran to completion on the cluster. Its log ends:
That is the first time a
ui-reactrun has produced output at TACC.The four checks
execution_engine: tapislocalex. #109What the two failures are
#109 — the engine label.
execution_engineis a hardcoded"localex"literal inExecutionCreation.createExecutionMetadata, so every row carries it whatever engine ran. Therow's
run_idis a real Tapis job id. The server does not read the column — it readsprefs.execution_engine— so nothing breaks today, but every consumer of the column is wrong.#110 — the Results step. Two independent causes.
MintResultsguards ononPublishResults, a propMintThreadnever passes, so Fetch results makes no request atall. And the route it should call fails anyway: the successful execution returns 500 wrapping a
403 from Tapis or CKAN. Both need a fix. This blocks #77.
The first attempt failed on data, not on code
The first run used
Subside & Oil Spill Allocation Details, whose resource is an.m4a. Themodel is a narrative-analysis tool that reads JSON. My dataset choice, not a defect.
The retry used
nunap-interview-data, whose resources are whisper-transcription JSON. One ofits two files is invalid JSON — a raw newline at byte 25, inside a string — which is exactly
what the log reported:
So one run failed on bad catalog data and one succeeded. Both diagnoses came from the
Ensemble Manager log endpoint, which returned 200 with the bearer token.
Confirmed on the way
Continue worked with no error. The workaround in the brief is sound.
Also filed
2 runs (1 input resources × 1 parameters). Reproduced on twothreads. The count is right, the printed factor is wrong.
d41d8cd98f00b204e9800998ecf8427e,the MD5 of the empty string.
Left behind at TACC
Problem statement
swi983qfmsmbp8x6, task1hdzfg5zmsmbp98f, threadsjx8ibyilmsmg1pvt,xoqh3tismsmfgivbandx4gbmq00msmbp9ib. #99 means they cannot be deleted from the app.Three real Tapis jobs were run.
mosoriob commentedon Aug 9, 2026
Resolved. The submit path is proven end to end: a run reached Tapis from ui-react and finished on TACC's cluster. Two of the four checks failed and are now #109 and #110; #110 blocks #77.