Skip to content

Fix execution pages stuck loading and Bad Request on entity pages - #948

Open
1fanwang wants to merge 1 commit into
flyteorg:masterfrom
1fanwang:1fanwang/fix-created-at-sort-key
Open

1fanwang wants to merge 1 commit into
flyteorg:masterfrom
1fanwang:1fanwang/fix-created-at-sort-key

Conversation

@1fanwang

Copy link
Copy Markdown
Contributor

TL;DR

Since #942, opening an execution in the console against flyteadmin shows a spinner that never stops, so its nodes and the Relaunch button never appear. The workflow, task and launch plan pages also show "Failed to fetch data / Bad Request" for their versions and descriptions. flyteadmin rejects each of those requests with:

{"code":3,"message":"invalid sort key 'execution_created_at'","details":[]}

The change is not in a release yet; v1.23.0 predates it.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

The pull request above changed the shared creation-time sort so that execution lists sort by when the execution was created. The console uses that same sort for node executions, workflows, tasks, launch plans and descriptions, and flyteadmin understands the execution-specific key only on the executions list. For every other list it refuses the request as an invalid sort key.

This gives execution lists their own sort and puts every other list back on its own creation time, so the start-time filter behavior from the earlier change stays.

Tracking Issue

N/A. The regression comes from #942.

Follow-up issue

NA

Testing Done

I ran the console dev server against a Flyte v1.16.4 sandbox and opened the execution, workflow, task, launch plan and project executions pages with Playwright, on master at 73d8292 and on this branch at 460a967. A small proxy serves the console and admin from one origin, as a deployment does.

Setup:

gh release download flytectl/v0.9.8 --repo flyteorg/flyte -p flytectl_Darwin_arm64.tar.gz -p checksums.txt
grep Darwin_arm64 checksums.txt | shasum -a 256 -c -
tar xzf flytectl_Darwin_arm64.tar.gz flytectl
./flytectl demo start --version v1.16.4
uv venv -q --python 3.11 .venv && VIRTUAL_ENV=$PWD/.venv uv pip install -q flytekit==1.16.10 playwright==1.57.0
.venv/bin/playwright install chromium
export FLYTECTL_CONFIG=$HOME/.flyte/config-sandbox.yaml
.venv/bin/pyflyte run --remote --project flytesnacks --domain development wf_8105.py wf --params '{"name": "vol", "volume_size": 10737418240}'
./flytectl --config ~/.flyte/config-sandbox.yaml get execution -p flytesnacks -d development axxqf52fctjt47dmbcds

The seed execution axxqf52fctjt47dmbcds succeeded. Console, from the repository root, with a self-signed pair because the dev server exits without one:

mkdir -p scripts/certificate && openssl req -x509 -nodes -newkey rsa:2048 -days 2 -subj /CN=localhost -keyout scripts/certificate/server.key -out scripts/certificate/server.crt
unset ADMIN_API_URL; BASE_URL=/console LOCAL_DEV_HOST=localhost NODE_OPTIONS=--max-old-space-size=8192 node .yarn/releases/yarn-3.2.1.cjs workspace @clients/console run start --no-open
node proxy.js

The admin rejects the key directly, and accepts created_at on the same request:

$ curl -s -w '\nHTTP %{http_code}\n' "http://localhost:30080/api/v1/node_executions/flytesnacks/development/axxqf52fctjt47dmbcds?limit=5&sort_by.direction=DESCENDING&sort_by.key=execution_created_at"
{"code":3,"message":"invalid sort key 'execution_created_at'","details":[]}
HTTP 400
$ curl -s -w '\nHTTP %{http_code}\n' "http://localhost:30080/api/v1/node_executions/flytesnacks/development/axxqf52fctjt47dmbcds?limit=5&sort_by.direction=DESCENDING&sort_by.key=created_at" | cut -c1-200
{"nodeExecutions":[{"id":{"nodeId":"end-node","executionId":{"project":"flytesnacks","domain":"development","name":"axxqf52fctjt47dmbcds","org":""}},"inputUri":"s3://my-s3-bucket/metadata/propeller/fl
HTTP 200

Master, 73d8292:

$ .venv/bin/python crawl_sort.py master
== execution details: http://localhost:8095/console/projects/flytesnacks/domains/development/executions/axxqf52fctjt47dmbcds
   400 node_executions sort_by.key=execution_created_at
   Relaunch buttons: 0, loading spinners: 1
== workflow details: http://localhost:8095/console/projects/flytesnacks/domains/development/workflows/wf_8105.wf
   200 executions sort_by.key=execution_created_at
   200 launch_plans sort_by.key=created_at
   400 description_entities sort_by.key=execution_created_at
   400 workflows sort_by.key=execution_created_at
   page shows: Failed to fetch data
   page shows: Bad Request
== task details: http://localhost:8095/console/projects/flytesnacks/domains/development/tasks/wf_8105.show
   200 executions sort_by.key=execution_created_at
   400 description_entities sort_by.key=execution_created_at
   400 tasks sort_by.key=execution_created_at
   page shows: Failed to fetch data
   page shows: Bad Request
   page shows: No executions found
== launch plan details: http://localhost:8095/console/projects/flytesnacks/domains/development/launchPlans/wf_8105.wf
   200 executions sort_by.key=execution_created_at
   200 launch_plans sort_by.key=created_at
   400 launch_plans sort_by.key=execution_created_at
   page shows: Failed to fetch data
   page shows: Bad Request
== project executions: http://localhost:8095/console/projects/flytesnacks/domains/development/executions
   200 executions sort_by.key=execution_created_at

This branch, 460a967:

$ .venv/bin/python crawl_sort.py fix
== execution details: http://localhost:8095/console/projects/flytesnacks/domains/development/executions/axxqf52fctjt47dmbcds
   200 node_executions sort_by.key=created_at
   Relaunch buttons: 1, loading spinners: 0
== workflow details: http://localhost:8095/console/projects/flytesnacks/domains/development/workflows/wf_8105.wf
   200 description_entities sort_by.key=created_at
   200 executions sort_by.key=execution_created_at
   200 launch_plans sort_by.key=created_at
   200 workflows sort_by.key=created_at
== task details: http://localhost:8095/console/projects/flytesnacks/domains/development/tasks/wf_8105.show
   200 description_entities sort_by.key=created_at
   200 executions sort_by.key=execution_created_at
   200 tasks sort_by.key=created_at
   page shows: No executions found
== launch plan details: http://localhost:8095/console/projects/flytesnacks/domains/development/launchPlans/wf_8105.wf
   200 executions sort_by.key=execution_created_at
   200 launch_plans sort_by.key=created_at
== project executions: http://localhost:8095/console/projects/flytesnacks/domains/development/executions
   200 executions sort_by.key=execution_created_at

Execution lists keep execution_created_at, and every other list is back on created_at. The task page shows "No executions found" in both runs because the seed launched the workflow, not the task directly.

Reproducer source: wf_8105.py
from dataclasses import dataclass
from flytekit import task, workflow

@dataclass
class Params:
    name: str
    volume_size: int   # value will exceed int32

@task
def show(params: Params) -> str:
    return f"{params.name}: {params.volume_size}"

@workflow
def wf(params: Params) -> str:
    return show(params=params)
Reproducer source: proxy.js
// Serves the local console and the sandbox admin from one origin, as a Flyte deployment does.
const http = require('http');
const route = (url) => (url.startsWith('/console') ? { host: '::1', port: 8080 } : { host: '127.0.0.1', port: 30080 });
http
  .createServer((req, res) => {
    const target = route(req.url);
    const upstream = http.request(
      { ...target, method: req.method, path: req.url, headers: { ...req.headers, host: `localhost:${target.port}` } },
      (up) => {
        res.writeHead(up.statusCode, up.headers);
        up.pipe(res);
      },
    );
    upstream.on('error', (e) => {
      res.writeHead(502);
      res.end(String(e));
    });
    req.pipe(upstream);
  })
  .listen(8095, 'localhost', () => console.log('proxy on http://localhost:8095'));
Reproducer source: crawl_sort.py
#!/usr/bin/env python3
import sys
from urllib.parse import parse_qs, urlparse

from playwright.sync_api import sync_playwright

BASE = "http://localhost:8095/console/projects/flytesnacks/domains/development"
PAGES = {
    "execution details": f"{BASE}/executions/axxqf52fctjt47dmbcds",
    "workflow details": f"{BASE}/workflows/wf_8105.wf",
    "task details": f"{BASE}/tasks/wf_8105.show",
    "launch plan details": f"{BASE}/launchPlans/wf_8105.wf",
    "project executions": f"{BASE}/executions",
}

label = sys.argv[1]
with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context(ignore_https_errors=True, viewport={"width": 1440, "height": 1200})
    for name, url in PAGES.items():
        page = context.new_page()
        sorted_calls = []

        def on_response(resp, sorted_calls=sorted_calls):
            parsed = urlparse(resp.url)
            if not parsed.path.startswith("/api/v1/"):
                return
            key = parse_qs(parsed.query).get("sort_by.key")
            if key:
                endpoint = "/".join(parsed.path.split("/")[3:4])
                sorted_calls.append((resp.status, endpoint, key[0]))

        page.on("response", on_response)
        page.goto(url, wait_until="networkidle", timeout=120000)
        page.wait_for_timeout(4000)
        page.screenshot(path=f"runs/sort-{label}/{name.replace(' ', '_')}.png", full_page=True)
        print(f"== {name}: {url}")
        for status, endpoint, key in sorted(set(sorted_calls)):
            print(f"   {status} {endpoint} sort_by.key={key}")
        text = page.locator("body").inner_text()
        for marker in ("Failed to fetch data", "Bad Request", "Something went wrong", "No executions found"):
            if marker in text:
                print(f"   page shows: {marker}")
        if name == "execution details":
            relaunch = page.get_by_role("button", name="Relaunch").count()
            spinner = page.locator('[role="progressbar"]').count()
            print(f"   Relaunch buttons: {relaunch}, loading spinners: {spinner}")
        page.close()
    browser.close()

The start-time filter change made executionSortFields.createdAt
execution_created_at. Only the executions list accepts that key, so
flyteadmin rejected the node execution, workflow, task, launch plan and
description entity requests with "invalid sort key". Execution lists now
use executionCreatedAt and everything else is back on created_at.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant