Skip to content

Add opt-in scale comparison framework - #73

Open
jsturtevant wants to merge 9 commits into
observability/grafana-dashboardfrom
observability/scale-framework
Open

jsturtevant wants to merge 9 commits into
observability/grafana-dashboardfrom
observability/scale-framework

Conversation

@jsturtevant

@jsturtevant jsturtevant commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Adds an opt-in same-source Lampyr and Kubernetes comparison with JSON and Markdown results. It provides repeatable measurements for #1 without running scale benchmarks in CI.

@jsturtevant
jsturtevant added this pull request to stack #74 November 20, 2026 02:15
@jsturtevant
jsturtevant force-pushed the observability/scale-framework branch from a38e2bf to 325855d Compare November 20, 2026 02:19
Copilot AI balanced review requested due to automatic review settings November 20, 2026 02:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fixed Deployment and Service names and unsafe cleanup can overwrite or delete resources this invocation does not own, while traffic-routing and metric-calculation issues can distort benchmark comparisons.

4 open issues
What changed in this PR

Adds an opt-in framework for comparing Lampyr throughput and resource usage with the same Go workload in Kubernetes, supporting issue #1 without adding scale runs to CI.

Changes:

  • Adds a concurrency sweep with client measurements, platform metrics, and JSON/Markdown reports.
  • Provides a same-source Kubernetes baseline image and load-generator installation recipe.
  • Documents the workflow and network-path differences, with unit tests for runner helpers and reporting.
File Description
scripts/​tests/​test_scale_test.py Tests reporting, metric queries, validation, and cleanup.
scripts/​scale-test.py Implements workload orchestration, measurement, and reporting.
scripts/​fixtures/​scale-workload/​Dockerfile Builds the Kubernetes baseline from the Go example.
k8s/​observability.yaml Exposes revision labels for resource attribution.
Justfile Adds load-generator installation and comparison recipes.
docs/​scale-testing.md Explains setup, execution, and measurement caveats.
docs/​observability.md Links observability documentation to the scale framework.

Review effort: Balanced

Comment thread scripts/scale-test.py Resolved
Comment thread scripts/scale-test.py Resolved
Comment thread scripts/scale-test.py Resolved
Comment thread scripts/scale-test.py Resolved
@jsturtevant
jsturtevant force-pushed the observability/scale-framework branch from 325855d to 6903523 Compare November 20, 2026 02:25
Copilot AI balanced review requested due to automatic review settings November 20, 2026 02:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fixed-name Kubernetes resources lack ownership tracking, so cleanup can delete a pre-existing baseline or another run’s resources even after deployment fails.

7 open issues

Review effort: Balanced

Comment thread scripts/scale-test.py Resolved
Comment thread scripts/scale-test.py Resolved
Comment thread scripts/scale-test.py Resolved
@jsturtevant
jsturtevant force-pushed the observability/scale-framework branch from 6903523 to 67741f6 Compare November 20, 2026 19:41
Copilot AI balanced review requested due to automatic review settings November 20, 2026 19:41

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread scripts/scale-test.py Resolved
Comment thread scripts/scale-test.py Resolved
@jsturtevant
jsturtevant force-pushed the observability/scale-framework branch from 67741f6 to b6f20fe Compare November 20, 2026 20:24
@jsturtevant
jsturtevant force-pushed the observability/scale-framework branch from b6f20fe to 05a4291 Compare November 20, 2026 21:05
Copilot AI balanced review requested due to automatic review settings November 20, 2026 21:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsturtevant
jsturtevant force-pushed the observability/scale-framework branch from 05a4291 to 8f5b7a1 Compare November 20, 2026 21:39
@jsturtevant
jsturtevant force-pushed the observability/scale-framework branch from 8f5b7a1 to ca2157f Compare November 20, 2026 21:46
Copilot AI balanced review requested due to automatic review settings November 20, 2026 21:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsturtevant
jsturtevant force-pushed the observability/scale-framework branch from ca2157f to b5dbb77 Compare November 20, 2026 22:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread docs/scale-testing.md Resolved
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
@jsturtevant
jsturtevant force-pushed the observability/scale-framework branch from 150910d to b722631 Compare November 21, 2026 01:30
Copilot AI balanced review requested due to automatic review settings November 21, 2026 01:30
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown

Comment thread scripts/tests/test_scale_test.py Resolved
Copilot AI balanced review requested due to automatic review settings November 21, 2026 01:36

Copilot AI left a comment

Copy link
Copy Markdown

Comment thread scripts/scale-test.py Resolved
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 21, 2026 01:50

Copilot AI left a comment

Copy link
Copy Markdown

Comment thread scripts/scale-test.py Resolved
Comment thread Justfile Resolved
Comment thread scripts/scale-test.py Resolved
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 21, 2026 02:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved issues overwrite JSON results with Markdown, accept stale resource measurements, and cause concurrent installer failures, with corresponding regression tests failing.

3 open issues

Review effort: Balanced

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 21, 2026 02:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because mismatched CPU and memory container coverage can pass freshness checks while silently undercounting resources.

No open issues 🎉

3 resolved since last review

Review effort: Balanced

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because telemetry checks can miss running Pods absent at both endpoints, potentially understating resource usage and request counts.

No open issues 🎉

Review effort: Balanced

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because mismatched request paths and queries can cause the benchmark to compare different workloads, undermining the comparison’s validity.

No open issues 🎉

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants