Skip to content

fix: add blueprint_id index to _devlake_pipelines - #9059

Open
dncrews wants to merge 1 commit into
apache:mainfrom
dncrews:fix/pipeline-index
Open

fix: add blueprint_id index to _devlake_pipelines#9059
dncrews wants to merge 1 commit into
apache:mainfrom
dncrews:fix/pipeline-index

Conversation

@dncrews

@dncrews dncrews commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Summary

GET /blueprints/:blueprintId/pipelines runs a COUNT(*) and a filtered
SELECT on _devlake_pipelines.blueprint_id, but that column was not indexed.
On instances with a large pipeline history this forces a full table scan on
every request.

On a v1.0.3-beta15 instance, EXPLAIN ANALYZE of the count showed a full scan
of ~34k rows taking ~30s to return the 3 matching pipelines:

-> Aggregate: count(0) (actual time=30173..30173 rows=1 loops=1)
   -> Filter: (_devlake_pipelines.blueprint_id = 876) (actual time=26756..30173 rows=3 loops=1)
      -> Table scan on _devlake_pipelines (cost=16488 rows=27609) (actual time=545..30170 rows=34008 loops=1)

That ~30s scan exceeds the gateway timeout and surfaces as an upstream request timeout in config-ui when opening a blueprint's pipeline history.

This PR adds an index on _devlake_pipelines.blueprint_id:

  • a gorm:"index" tag on Pipeline.BlueprintId so fresh installs create the
    index at table creation, and
  • a migration script (20260818_add_blueprint_id_index_to_pipelines.go) so
    existing installs get the index via AutoMigrateTables, following the same
    pattern as addPullRequestIdIndexToPullRequestComments.

With the index, the blueprint_id count/lookup becomes an index seek instead of
a full table scan; the same EXPLAIN ANALYZE returns in sub-millisecond time.

Does this close any open issues?

N/A

Screenshots

N/A (backend performance change, no UI change).

Other Information

  • The change is additive and non-destructive: it only adds a secondary index,
    with no schema or data semantics change.
  • The AutoMigrateTables path is idempotent by index name, so re-running the
    migration is safe.

GET /blueprints/:blueprintId/pipelines runs COUNT(*) and a filtered
SELECT on _devlake_pipelines.blueprint_id, but the column was
unindexed. On instances with tens of thousands of pipeline rows this
forces a full table scan on every request; observed ~30s for a table
of 34k rows, causing upstream request timeouts in config-ui.

Add a gorm index tag on Pipeline.BlueprintId for fresh installs and a
migration script to add the index to existing installs.

Signed-off-by: Dan Crews <crewsd@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant