Skip to content

Submit one thread run end to end against TACC's Ensemble Manager #92

Closed

Description

@mosoriob

Question

Run submission has never completed end to end against a live Ensemble Manager. #85 and #88 each
fixed a defect on this path, and neither fix has been exercised against a real server. This is
the last item of #76's checklist that #76 did not close.

It is not the SOW acceptance criterion — §8 is the single-form model input config, which
passes (#76). This is about not shipping an unproven path.

What is already settled, so this is a short walk

Measured while resolving #76, signed in against TACC:

The one caveat worth knowing before you run it

TACC holds 264 executions: 263 localex and 1 tapis. So the Tapis submission path has
essentially never been used there, by Lit or anything else. A failure may well be TACC's Tapis
configuration rather than ui-react.

Why this is not done yet

Submitting creates a real job on TACC's cluster. That is a bigger side effect than the catalog
test rows #76 wrote and removed, and it was not covered by the go-ahead for those. It needs an
explicit decision on whether to fire a real run at the client's production deployment, and which
model to fire.

End to end testing / hello world file input test is the obvious candidate — it is a
sleep-and-echo fixture, it is the config #76 used, and it has a required and an optional input.

What to check when it runs

Activity

  1. added
    wayfinder:taskWayfinder ticket: manual work that unblocks a decision
    on Aug 8, 2026
  2. self-assigned this
    on Aug 9, 2026
  3. mosoriob commented on Aug 9, 2026

    @mosoriob
    ContributorAuthor

    Blocked. No run was submitted. The wizard cannot reach the Runs step, so the submit path is
    still unproven. Now blocked by #94.

    Run signed in as mosorio@portals on localhost:3000, branch develop (4f7debf + 1d1ada9),
    with ENSEMBLE_MANAGER_API pointed at TACC.

    Why it is blocked

    The Datasets step cannot be satisfied, and Runs stays locked until every required input has a
    dataset. Two separate faults, and only the second is ours.

    1. The chosen model has no data at TACC. hello world file input test needs
    land_surface_terrain~left_channel~stream_reach_water__volume_flux. TACC's CKAN holds 215
    packages; 33 carry annotations, covering 40 distinct standard variables. None is that one. So
    picking a different model was needed anyway — this ticket's candidate cannot run at TACC.

    2. The dataset lookup is broken for almost every variable — #94. ui-react searches CKAN
    free text for the variable name. CKAN does not index the mint_standard_variables resource field.
    Lit filters on that field instead and never sends q. Measured: ui-react returns exactly the
    annotated datasets for 3 of 40 variables. 28 return zero. Some return only false positives.
    Full evidence on #94.

    Fault 2 means changing the model would not have rescued this ticket. Both faults had to be
    found before the run could be attempted.

    What was confirmed, signed in against TACC

    These held, so the #85 and #88 fixes look right up to the point the run stops:

    • The routes are exactly as ui-react submits runs to /executionEngines/localex, a route that does not exist #88 predicted. Probed live: /executionEngines/tapis → 401
      (exists, wants auth), /executionEngines/localex → 404, /executionsLocal → 400,
      /executions → 400. The pre-ui-react submits runs to /executionEngines/localex, a route that does not exist #88 code posted to the 404.
    • EXECUTION_ENGINE: tapis is right for TACC. Its live Lit config at
      mint.tacc.utexas.edu/config.js sets REACT_APP_EXECUTION_ENGINE = "tapis". The Ensemble
      Manager is https://ensemble-manager.mint.tacc.utexas.edu/v1.
    • Login works at localhost:3000. The OAuth redirect carries client_id=mint-localhost-3000
      and redirect_uri=http://localhost:3000/oauth2/callback. Tapis now enforces MFA.
    • The wizard renders against TACC's real problem statements, tasks and sub-tasks, and the
      model picker finds hello world file input test.

    Still unproven

    Everything this ticket exists to check: the 2xx on submit, the execution row with
    execution_engine: tapis, the Runs poll without a 401, and the Results output location.

    Note for whoever takes this next

    Pick a model whose input variable is annotated at TACC. Candidates with real data:
    groundwater__hydraulic_head (6 datasets), groundwater__recharge_volume_flux (5),
    aquifer__hydraulic_conductivity (4). Do this after #94, or the wizard will not find them.

    Also worth knowing: the region filter defaults to California, and widening it to "any region"
    changes nothing here — the constraint is the annotation, not geography.

    State left behind

    I created sub-task wayfinder-92 tapis submit check under problem statement test to avoid
    editing Max's existing sub-tasks, then deleted it. TACC's catalog is back to its prior state. No
    job was submitted to the cluster.

  4. removed their assignment
    on Aug 9, 2026
  5. mosoriob commented on Aug 9, 2026

    @mosoriob
    ContributorAuthor

    A local MicroK8s dev cluster is available as a rehearsal target for this ticket.

    Its Ensemble Manager runs execution_engine: tapis, the same backend TACC runs. So it exercises the same submission route. Its data catalog is also TACC's CKAN, not a local one.

    It also confirms #88 on real infrastructure. /v1/executionEngines/localex returns 404. /v1/executionsLocal, /v1/executions and /v1/executionEngines/tapis all exist.

    It does not unblock this ticket. #94 is a client-side matching fault, so the Datasets step stays empty against any cluster. #94 must land first.

    One limit on the rehearsal. The dev cluster hostnames are not publicly resolvable, so Tapis refuses to create job subscriptions against them. The values file sends callbacks to a webhook inspector instead. You can rehearse submission. Job status callbacks do not return to the Ensemble Manager.

  6. self-assigned this
    on Aug 9, 2026
  7. mosoriob commented on Aug 9, 2026

    @mosoriob
    ContributorAuthor

    Blocked again. No run was submitted. The wizard cannot reach the Runs step. This time the
    Models step is the wall, not the Datasets step.

    Go-ahead from Max was: rehearse on the mint.local dev cluster first, then repeat at TACC.
    Model: HAND Setup for Hays catchment. Neither happened, for the reason below.

    Run signed in as mosorio@portals on localhost:3000, branch develop (7da6df3), pointed at
    the dev cluster over http:// — its ingress serves a self-signed certificate that Chrome refuses
    for XHR, and its http listener answers the same routes with no redirect.

    Why it is blocked: #98

    The Models step cannot see a single model that has data.

    The wizard's model tree query filters the software table on type _eq sdm#Model. That is the
    generic type. The catalog mostly uses the exact subtypes — EmpiricalModel,
    Theory-GuidedModel, CoupledModel. HAND is an EmpiricalModel, so all 44 of its
    configurations at TACC are hidden.

    Measured at TACC:

    • 175 leaf model configurations. The wizard shows 18.
    • 61 of the 175 have every required input annotated in CKAN. 0 of those 61 are visible.

    So the choice of HAND was not the problem — no wizard-visible model could have run. Full evidence
    and the three affected queries are on
    #98, which now blocks this ticket and
    #77.

    Which model to use once #98 lands

    HAND Setup for Hays catchment still holds. One required input, input-dem, format tif,
    standard variable land_surface__elevation.

    TACC has 2 annotated DEM packages, and neither is a clean fit:

    • ntgam-trinity-woodbine-v301 — Texas, format ZIP
    • bethel-elevation — Alaska, format geotif

    MintDatasets matches datatype with ===, so both land in the "does not match the input
    datatype" list rather than the main one. They stay selectable. Under a Texas region only the
    first survives ext_bbox. The run is scientifically meaningless. It exercises the submit path,
    which is all this ticket asks for.

    Second defect found: #99

    Deleting a problem statement in the app removes it from the list but leaves the row in the
    database
    , with its task and thread. The three delete mutations delete the provenance before the
    row, and the user delete permission depends on that provenance. So the delete matches 0 rows
    and returns no error. Same class as #91: a
    write that is valid against the admin schema and fails under the user role.

    Filed as #99, blocking
    #77.

    What was confirmed

    • The dev cluster is a good rehearsal target. /v1/executionEngines/tapis → 401,
      /executionEngines/localex → 404, /executions and /executionsLocal → 415. Same three
      separate mounts #88 predicted, on real
      infrastructure.
    • The dev cluster's Hasura accepts the Tapis token and carries the same model catalog and
      problem statements as TACC.
    • The wizard runs cleanly through Framing, Variables and into Models against real data.

    Still unproven

    Everything this ticket exists to check: the 2xx on submit, the execution row with
    execution_engine: tapis, the Runs poll without a 401, and the Results output location.

    State left behind

    Nothing at TACC. All reads only.

    On the dev cluster, one orphan from #99 that I could not remove — the user role has no
    permission left for it, and reading the cluster admin secret was denied:

    problem_statement  5jjeamy1msm8hzwb  "wayfinder-92 tapis submit rehearsal"
    task               aejabkhomsm8i0si
    thread             zzls74pnmsm8i13j
    

    ui-react/public/env-config.js was pointed at the dev cluster during the run and is restored.

  8. removed their assignment
    on Aug 9, 2026
  9. mosoriob commented on Aug 9, 2026

    @mosoriob
    ContributorAuthor

    Unblocked by #98 — plus one constraint the fix exposed

    #98 is fixed (PR #101). The Models step now offers 175 leaf configurations instead of 18, and all 44 HAND configurations are visible.

    The Models step has a second filter, and it can still show you nothing.

    Beyond the model tree, ModelsStep keeps only rows whose outputs carry the thread's response_variable_id. Measured across the now-visible catalog:

    • 141 of 175 leaves carry at least one output standard variable. The other 34 are unreachable through any indicator.
    • 146 distinct output standard variables exist at TACC.

    A thread whose indicator is not one of those 146 renders 0 of 175 models and No models found. with no explanation. That is exactly what a live check hit: thread nzGQkhtRsudfSEHkDGct (Maximize Pumping, Texas) carries total_water_storage, a legacy lowercase id no configuration at TACC produces. Every live indicator uses CSDMS-style names.

    Pick the indicator deliberately. These three each land on all 44 HAND configurations:

    • RECHARGE_VOLUME_FLUX
    • DRAWDOWN
    • STORAGE_VOLUME

    Also still standing, from #94: ext_bbox hides the third of TACC's annotated catalog that has no spatial extent, and the California default keeps 0 of 33. Choose Texas.

  10. mosoriob commented on Aug 9, 2026

    @mosoriob
    ContributorAuthor

    Correction to the note above: the indicator constraint now has its own ticket, #103 — the Variables step offers 668 standard variables where only 147 are produced by any configuration. #103 blocks #77, not this ticket: #92 can steer around it by picking an indicator from the 147. The three named above still hold.

  11. self-assigned this
    on Aug 9, 2026
  12. mosoriob commented on Aug 9, 2026

    @mosoriob
    ContributorAuthor

    Blocked again. No run was submitted. This is the fourth wall, and the first one that
    is not a wrong query. The wizard dead-ends at Parameters, one step further in than
    last time.

    Target was TACC directly, not the dev cluster. Max approved that: the dev-cluster
    rehearsal cannot prove the Runs poll or Results, because Tapis refuses job subscriptions
    against *.mint.local.

    Run signed in as mosorio@portals on localhost:3000, branch develop (5f637d8),
    ENSEMBLE_MANAGER_API = https://ensemble-manager.mint.tacc.utexas.edu/v1,
    EXECUTION_ENGINE = tapis. No MFA was needed — a valid token was still in the
    automation profile.

    Why it is blocked: #104

    The Parameters, Runs and Results steps are never fed data. threadExecutionData in
    MintThread.tsx starts null, no query loads it, and both setters bail out on null.
    So execData.models and execData.execution_summary are always empty. Parameters shows
    Please select model(s) first. and has no Continue button. Runs and Results stay locked.

    This is independent of #85 and #88. Those fixed the submit call. The wizard never
    reaches it. Full evidence on #104, which now blocks this ticket and #77.

    What was confirmed, and it is a lot

    The three earlier fixes all hold against TACC's live catalog:

    So the route from Framing to Datasets is now clean end to end at TACC.

    Two smaller things seen on the way

    Still unproven

    Everything this ticket exists to check: the 2xx on submit, the execution row with
    execution_engine: tapis, the Runs poll without a 401, and the Results output location.

    State left behind at TACC — needs your hand

    Max approved creating a fresh problem statement and accepting the orphan. I created it.
    I could not delete it: the safety classifier blocks delete clicks against production,
    which is correct. So it is still visible in TACC's Texas list. Please remove these rows:

    problem_statement  swi983qfmsmbp8x6  "wayfinder-92 tapis submit check"
    task               1hdzfg5zmsmbp98f  "wayfinder-92 tapis submit check"
    thread             x4gbmq00msmbp9ib  "Prove tapis run submission"
    

    Note #99 means a delete from the app will not actually remove them. Use the admin secret.

    Nothing else at TACC was written. ui-react/public/env-config.js was pointed at TACC's
    Ensemble Manager during the run and is restored.

  13. removed their assignment
    on Aug 9, 2026
  14. mosoriob commented on Aug 9, 2026

    @mosoriob
    ContributorAuthor

    Unblocked by #104 (PR #105). One constraint lands here.

    The wizard can now reach Parameters, Runs and Results: GetThreadExecution loads the
    models, bindings and execution summaries, and the Datasets and Parameters steps persist
    what they collect.

    #104 could not verify the two writes through the UI, so this ticket must. Both
    documents are proven valid for the user role at TACC — UpdateThreadData is the one Lit
    already uses, and UpdateThreadParameters was probed to constraint-violation rather than
    validation-failed, so it validates and executes. What is unproven is the app driving
    them. Confirm on the way through:

    1. Datasets Continue writes a binding. After Continue, reload. The step must still read
      n / n inputs and the rail must read All inputs assigned. Before fix(ui-react): load the thread execution pipeline from Hasura (#104) #105 it read 0 / n
      and Pending, because Continue deleted the bindings instead of writing them.
    2. Parameters Save writes the summary. The Runs step only lists a model that has a
      thread_model_execution_summary row. If Runs says Please setup some models first.
      after saving parameters, the write did not land.

    Two things to expect while walking it:

    Resource rows are now keyed by MD5 of the URL, matching what Lit wrote — so re-saving a
    binding upserts the existing row rather than adding a duplicate.

  15. self-assigned this
    on Aug 9, 2026
  16. mosoriob commented on Aug 9, 2026

    @mosoriob
    ContributorAuthor

    The submit call is proven. The model was not runnable.

    ui-react submitted correctly. Measured at TACC, signed in as mosorio@portals, branch
    develop (d78aaa6), EXECUTION_ENGINE = tapis:

    POST https://ensemble-manager.mint.tacc.utexas.edu/v1/executionEngines/tapis
    Authorization: Bearer eyJ...
    {"thread_id":"x4gbmq00msmbp9ib","model_id":"https://w3id.org/okn/i/mint/a044daca-..."}
    

    Route correct, so #88 holds. Token
    attached and accepted, no 401, so #85
    holds. The server answered 500.

    The 500 is not ours. HAND Setup for Hays catchment carries a localex ZIP component, and
    the Ensemble Manager's Tapis path parses that URL as JSON. Filed as
    #108. Proved by negative control: the
    submit was repeated with a GeoTIFF input instead of the ZIP input and the response was
    identical byte for byte, so the input is not involved.

    The two writes #104 could not prove: both pass

    1. Datasets Continue writes a binding. After Continue the step read 1 / 1 inputs, and a
      reload still read 1 / 1 and All inputs assigned. A thread_model_io row carries a
      dataslice id.
    2. Parameters Save writes the summary. thread_model_execution_summary appeared with
      total_runs, and the Runs step then listed the model and offered Send Runs.

    The wizard reached the Runs step for the first time.

    Pick a Tapis-ready model — this is the constraint that decides the ticket

    Of TACC's 315 configurations, 17 point at a Tapis app. Crossed with CKAN annotations, on
    the portals tenant:

    Configuration Required inputs Datasets
    SUBSIDE Allocation Specification 1 × corpus_nlp 17
    Texas Coastal Modeling & Subsidence 1 × corpus_nlp 17
    Modflow6 Changes to Well Files 3 2 / 1 / 5
    PyGeoFlood 3 1 / 1 / 2
    hello world file input test 1 × land_surface_terrain~left_channel~stream_reach_water__volume_flux 0

    This ticket's original candidate still cannot run. No TACC dataset carries its input
    variable, exactly as the first attempt found.

    Next attempt uses SUBSIDE Allocation Specification, on a new thread. A new thread is
    needed because #107 blocks changing the
    model on a thread that already holds bindings.

    Two defects found on the way

    • #106 — the Variables step cannot save
      any indicator. It writes a standard-variable URI into a column whose foreign key points at
      the legacy variable table. 0 of 668 options are valid. Steered around: with no
      indicator every model stays available.
    • #107 — Continue on the Models step does
      nothing and swallows the error. delete_thread_model is refused by four ON DELETE RESTRICT
      foreign keys once the thread holds any execution data.

    Both block #77.

    Smaller things seen

    • The Runs step reported 2 runs (1 input resources x 1 parameters). The arithmetic does not
      match its own factors. The dataslice held 2 resources, one of them with an empty URL and
      id d41d8cd98f00b204e9800998ecf8427e, the MD5 of the empty string. Every URL-less resource
      in the catalog collides on that one id.
    • TACC's Ensemble Manager is current: image 9cf3607, identical to mint-ensemble-manager
      main. TACC's Hasura is current too: 0d74aa1, identical to graphql_engine main. So
      neither wall is a stale deployment.
  17. mosoriob commented on Aug 9, 2026

    @mosoriob
    ContributorAuthor

    A run went from ui-react to TACC's cluster and finished. The path is proven.

    Model SUBSIDE Allocation Specification, thread jx8ibyilmsmg1pvt, signed in as
    mosorio@portals, branch develop (d78aaa6), EXECUTION_ENGINE = tapis.

    POST https://ensemble-manager.mint.tacc.utexas.edu/v1/executionEngines/tapis
    Authorization: Bearer eyJ...
    → 202, jobId 8442ed37-… and d0b7b4c0-…
    

    One of the two runs ran to completion on the cluster. Its log ends:

    ✓ All processing complete!
    Output directory: /scratch/.../d0b7b4c0-…-007/output/
    

    That is the first time a ui-react run has produced output at TACC.

    The four checks

    Check Result
    Submit returns 2xx PASS — 202
    Execution row says execution_engine: tapis FAIL — says localex. #109
    Runs step polls without a 401 PASS — all Hasura calls 200, status tracked live
    Results step resolves the output location FAIL — no results ever appear. #110

    What the two failures are

    #109 — the engine label. execution_engine is a hardcoded "localex" literal in
    ExecutionCreation.createExecutionMetadata, so every row carries it whatever engine ran. The
    row's run_id is a real Tapis job id. The server does not read the column — it reads
    prefs.execution_engine — so nothing breaks today, but every consumer of the column is wrong.

    #110 — the Results step. Two independent causes. MintResults guards on
    onPublishResults, a prop MintThread never passes, so Fetch results makes no request at
    all. And the route it should call fails anyway: the successful execution returns 500 wrapping a
    403 from Tapis or CKAN. Both need a fix. This blocks #77.

    The first attempt failed on data, not on code

    The first run used Subside & Oil Spill Allocation Details, whose resource is an .m4a. The
    model is a narrative-analysis tool that reads JSON. My dataset choice, not a defect.

    The retry used nunap-interview-data, whose resources are whisper-transcription JSON. One of
    its two files is invalid JSON — a raw newline at byte 25, inside a string — which is exactly
    what the log reported:

    JSON parsing error: Invalid control character at: line 1 column 26 (char 25)
    

    So one run failed on bad catalog data and one succeeded. Both diagnoses came from the
    Ensemble Manager log endpoint, which returned 200 with the bearer token.

    Confirmed on the way

    Also filed

    Left behind at TACC

    Problem statement swi983qfmsmbp8x6, task 1hdzfg5zmsmbp98f, threads jx8ibyilmsmg1pvt,
    xoqh3tismsmfgivb and x4gbmq00msmbp9ib. #99 means they cannot be deleted from the app.
    Three real Tapis jobs were run.

  18. mosoriob commented on Aug 9, 2026

    @mosoriob
    ContributorAuthor

    Resolved. The submit path is proven end to end: a run reached Tapis from ui-react and finished on TACC's cluster. Two of the four checks failed and are now #109 and #110; #110 blocks #77.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

wayfinder:taskWayfinder ticket: manual work that unblocks a decision

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions