Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,3 @@ $(ICON_LIGHT_BG): $(ICON_MASTER) scripts/make-app-icon.py

$(ICON_ADAPTIVE): $(ICON_MASTER) scripts/make-app-icon.py
uv run python scripts/make-app-icon.py --adaptive $(ICON_MASTER) $@

2 changes: 1 addition & 1 deletion docs/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= uv run --project .. sphinx-build
SPHINXBUILD ?= uv run --project .. --extra docs sphinx-build
SOURCEDIR = source
BUILDDIR = build

Expand Down
108 changes: 108 additions & 0 deletions docs/source/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,9 @@ experimaestro jobs log -f mymodule.MyTask/abc123def

# Get job directory path
experimaestro jobs path mymodule.MyTask/abc123def

# Inspect job parameters in interactive TUI
experimaestro inspect mymodule.MyTask/abc123def
```

## Filter Expressions
Expand Down Expand Up @@ -474,3 +477,108 @@ the skill to the latest version.
## (Sphinx) Checking undocumented configurations

Use `experimaestro check-documentation objects.inv my_package` to check for undocumented members. The `objects.inv` file should be generated by sphinx

## Experiment DAG Visualization (`experimaestro dag`)

Experimaestro provides dependency graph visualization and export capabilities via the `dag` command. It supports interactive browser rendering, standalone HTML reports, standardized JSON graphs, visual plots, and console text trees.

![Experimaestro DAG Visualization](img/DAG_example.png)

Principal Features:
- Two views available - a hiearchical-aligned heuristic (Aligned) and a more advanced using [Dagre JS](https://github.com/dagrejs/dagre)
- Optionally display intermediate configs (here `ParameterLoader` is just the output of `Learner` passed to `Evaluate`, not a task) or only the Tasks that are run.
- Shows the status of Tasks
- Wired with the [Grid Search](experiments/grid_search.md) feature to only show by default the first config - you can still choose to display all of them.

Example usage:
```bash
# Launch interactive web viewer in browser (default mode)
experimaestro dag my_experiment

# Export standalone interactive HTML report (zero-server requirement)
experimaestro dag my_experiment --save-as dag.html

# Print terminal ASCII tree
experimaestro dag my_experiment --tree

# Display interactive matplotlib plot window (requires matplotlib)
experimaestro dag my_experiment --plot

# Save plot image directly without GUI window
experimaestro dag my_experiment --save-as dag.png

# List hyperparameter trial configurations
experimaestro dag my_experiment --list-configs

# Inspect a single trial configuration
experimaestro dag my_experiment -c 0
experimaestro dag my_experiment --filter "lr=0.001,kernel_size=3"

# Export standardized JSON graph
experimaestro dag my_experiment --save-as dag.json
experimaestro dag my_experiment --json
```

### Options

| Option | Description |
|--------|-------------|
| `EXPERIMENT_ID` | Experiment identifier (positional or via `-e`, `--experiment`, `--experiment-id`). Auto-selected if only one experiment exists in the workspace |
| `-r, --run, --run-id` | Run timestamp ID (default: most recent run) |
| `-w, --workspace` | Workspace ID or path (default: auto-detected) |
| `-W, --web / -nw, --no-web` | Interactive web viewer in browser (default: enabled unless `--tree`, `--plot`, or file export is specified) |
| `-t, --tree` | Print text tree representation of the DAG to terminal |
| `--plot` | Display matplotlib interactive plot window (requires matplotlib) |
| `--port` | Port for web server (default: random available port) |
| `--no-open` | Do not automatically open browser when launching web viewer |
| `--save-as` | Save output file (`.html`, `.json`, `.png`, `.svg`, `.pdf`) |
| `--html` | Save standalone interactive HTML page |
| `--json` | Export DAG as JSON to standard output |
| `--json-file` | Export DAG as JSON to specified file |
| `--tasks-only` | Only display Task and Action nodes, skipping static config nodes |
| `--list-configs` | List all unique configuration parameter sets found in the experiment |
| `--single-config` | Render DAG for a single grid configuration (defaults to trial 0) |
| `-c, --config-index` | Index of configuration to display |
| `--filter` | Filter DAG by tag values (e.g. `lr=0.001,kernel_size=3`) |
| `--include-actions` | Include downstream Action and EvaluatedModel nodes when filtering configs |
| `--layout` | Web viewer layout algorithm: `dagre` (default) or `aligned` |

## Job Parameters Inspector (`experimaestro inspect`)

Experimaestro provides an interactive terminal-based (TUI) parameters inspector built with Textual. It visualizes the hierarchical configuration graph of jobs, with support for cycle detection, relative path resolution, Python code synthesis, and remote workspace job inspection via SSH.

```bash
# Inspect current directory / job
experimaestro inspect

# Inspect a specific job by task name or hash prefix
experimaestro inspect mymodule.MyTask/1822b2b

# Inspect a specific params.json directly
experimaestro inspect /path/to/params.json

# Inspect a job in a remote SSH workspace
experimaestro inspect -w my_cluster mymodule.MyTask/1822b2b
```

### Options

| Option | Description |
|--------|-------------|
| `TARGET` | Job ID or hash prefix, path to `params.json` / `experimaestro.json`, or job directory (default: `.`) |
| `-w, --workspace` | Workspace ID or directory path (defaults to current Experimaestro workspace; supports remote SSH workspaces) |

### Keybindings

| Key | Action |
|-----|--------|
| `q` / `Esc` | Quit / close active search or modal |
| `/` | Focus search bar to filter nodes by parameter name or value |
| `g` | Toggle parameter grouping (alphabetical vs categorized) |
| `p` | Toggle path display (relative vs absolute) |
| `c` | Copy synthesized Python code snippet to clipboard |
| `s` | Copy selected parameter value or resolved path to clipboard |
| `J` | Toggle raw JSON view |
| `e` | Expand all nodes |
| `c` (on parent) | Collapse all child nodes |
| `?` | Show interactive keyboard shortcuts and help |
6 changes: 3 additions & 3 deletions docs/source/experiments/grid_search.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,15 +140,15 @@ class MyConfig(ConfigurationBase):
def run(helper: ExperimentHelper, cfg: MyConfig):
# Generate all permutations and their corresponding tags
configurations, all_tags = generate_grid(cfg)

for config, tags in zip(configurations, all_tags):
# Each permutation is a finalized MyConfig object
# 'tags' contains the specific values used (e.g., {'learning_rate': 0.1})

# Use tags to create a descriptive experiment ID
tag_suffix = ".".join(f"{k}={v}" for k, v in tags.items())
sub_id = f"{config.id}.{tag_suffix}" if tag_suffix else config.id

# Run your experiment logic for this config
...
```
Expand Down
Binary file added docs/source/img/DAG_example.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/source/img/icon-adaptive.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
17 changes: 17 additions & 0 deletions docs/source/interfaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -428,3 +428,20 @@ This is needed when:
- Jobs were created before the database was introduced
- Jobs were created by external processes
- The database is out of sync with the filesystem

## Experiment DAG Visualization

Experimaestro provides an interactive web-based DAG viewer via `experimaestro dag` to explore task dependency graphs, configuration nodes, and trial hyperparameters.

```{figure} img/DAG_example.png
:alt: Experimaestro DAG Web Viewer

Interactive DAG viewer: inspect task dependencies, parameters, execution states, and filter by hyperparameter configuration.
```

```bash
# Launch interactive web viewer
experimaestro dag my_experiment
```

See [Experiment DAG Visualization](cli.md#experiment-dag-visualization-experimaestro-dag) in the CLI reference for details and options.
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ dependencies = [
"typing-extensions >=4.2; python_version < \"3.12\"",
"uvicorn[standard] >=0.27,<1",
"watchdog >=2",
"networkx>=3.6.1",
]

[project.optional-dependencies]
Expand Down
13 changes: 13 additions & 0 deletions src/experimaestro/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,16 @@ def get_command(self, ctx, name):

cli.add_command(workspaces_cli)

# Import and add dag command
from .dag import dag as dag_cli # noqa: E402

cli.add_command(dag_cli)

# Import and add inspect command
from .inspect import inspect_command # noqa: E402

cli.add_command(inspect_command, "inspect")


@cli.group()
def migrate():
Expand Down Expand Up @@ -445,6 +455,9 @@ def experiments(ctx, workdir, workspace):
ctx.obj = None


experiments.add_command(dag_cli, "dag")


@experiments.command("list")
@pass_cfg
def list_experiments(workdir: Path):
Expand Down
Loading
Loading