Repository navigation
Add opt-in scale comparison framework - #73
jsturtevant wants to merge 9 commits into
Conversation
a38e2bf to
325855d
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Fixed Deployment and Service names and unsafe cleanup can overwrite or delete resources this invocation does not own, while traffic-routing and metric-calculation issues can distort benchmark comparisons.
4 open issues
What changed in this PR
Adds an opt-in framework for comparing Lampyr throughput and resource usage with the same Go workload in Kubernetes, supporting issue #1 without adding scale runs to CI.
Changes:
- Adds a concurrency sweep with client measurements, platform metrics, and JSON/Markdown reports.
- Provides a same-source Kubernetes baseline image and load-generator installation recipe.
- Documents the workflow and network-path differences, with unit tests for runner helpers and reporting.
| File | Description |
|---|---|
scripts/tests/test_scale_test.py |
Tests reporting, metric queries, validation, and cleanup. |
scripts/scale-test.py |
Implements workload orchestration, measurement, and reporting. |
scripts/fixtures/scale-workload/Dockerfile |
Builds the Kubernetes baseline from the Go example. |
k8s/observability.yaml |
Exposes revision labels for resource attribution. |
Justfile |
Adds load-generator installation and comparison recipes. |
docs/scale-testing.md |
Explains setup, execution, and measurement caveats. |
docs/observability.md |
Links observability documentation to the scale framework. |
Review effort: Balanced
325855d to
6903523
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Fixed-name Kubernetes resources lack ownership tracking, so cleanup can delete a pre-existing baseline or another run’s resources even after deployment fails.
7 open issues
Use run-specific resources and ownership-scoped cleanup Requests without timeouts can block scenarios indefinitely · New Port-forwarding bypasses baseline replica load balancing · New CPU evaluation can miss load-period usage · New Port-forwarding bypasses Service load balancing Summed component peaks overstate combined peak memory Counter resets cause scenario metrics to be undercounted
Review effort: Balanced
6903523 to
67741f6
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Unbounded requests can hang the runner indefinitely, while shared baseline resource names allow concurrent runs to modify or delete each other’s targets.
9 open issues
Use run-specific resources and ownership-scoped cleanup Measure combined memory peak from simultaneous component usage · New Reject multiple baseline replicas when port-forwarding to one Pod · New CPU evaluation can miss load-period usage Port-forwarding bypasses baseline replica load balancing Requests without timeouts can block scenarios indefinitely Port-forwarding bypasses Service load balancing Summed component peaks overstate combined peak memory Counter resets cause scenario metrics to be undercounted
Review effort: Balanced
67741f6 to
b6f20fe
Compare
b6f20fe to
05a4291
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
scripts/scale-test.py reuses fixed Kubernetes resource names and deletes them unconditionally, risking modification or deletion of pre-existing resources and concurrent runs’ baselines.
9 open issues
Use run-specific resources and ownership-scoped cleanup Reject multiple baseline replicas when port-forwarding to one Pod Measure combined memory peak from simultaneous component usage CPU evaluation can miss load-period usage Port-forwarding bypasses baseline replica load balancing Requests without timeouts can block scenarios indefinitely Port-forwarding bypasses Service load balancing Summed component peaks overstate combined peak memory Counter resets cause scenario metrics to be undercounted
Review effort: Balanced
05a4291 to
8f5b7a1
Compare
8f5b7a1 to
ca2157f
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Short runs can incorrectly report zero CPU usage, while multi-replica baselines load only one Pod but aggregate all replicas’ resource usage, producing misleading benchmark results.
9 open issues
Use run-specific resources and ownership-scoped cleanup Reject multiple baseline replicas when port-forwarding to one Pod Measure combined memory peak from simultaneous component usage CPU evaluation can miss load-period usage Port-forwarding bypasses baseline replica load balancing Requests without timeouts can block scenarios indefinitely Port-forwarding bypasses Service load balancing Summed component peaks overstate combined peak memory Counter resets cause scenario metrics to be undercounted
Review effort: Balanced
ca2157f to
b5dbb77
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Fixed-name baseline resources and unconditional cleanup can overwrite or delete pre-existing resources, including those belonging to another running comparison.
13 open issues
Stale router metrics can be accepted as post-load results Use run-specific resources and ownership-scoped cleanup Resolve relative oha paths before invoking subprocesses Warmup can hang indefinitely on an unanswered request Reject multiple baseline replicas when port-forwarding to one Pod Measure combined memory peak from simultaneous component usage CPU evaluation can miss load-period usage Port-forwarding bypasses baseline replica load balancing Requests without timeouts can block scenarios indefinitely Port-forwarding bypasses Service load balancing Summed component peaks overstate combined peak memory Counter resets cause scenario metrics to be undercounted Dashboard instructions point to Grafana instead of Perses · New
Review effort: Balanced
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
150910d to
b722631
Compare
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
There was a problem hiding this comment.
🟡 Changes recommended
Fixed baseline resource names and unconditional cleanup can overwrite or delete resources belonging to existing or concurrent runs, even when deployment fails.
14 open issues
Stale router metrics can be accepted as post-load results Use run-specific resources and ownership-scoped cleanup Regression tests are not included in the unit-test recipe · New Resolve relative oha paths before invoking subprocesses Warmup can hang indefinitely on an unanswered request Reject multiple baseline replicas when port-forwarding to one Pod Measure combined memory peak from simultaneous component usage CPU evaluation can miss load-period usage Port-forwarding bypasses baseline replica load balancing Requests without timeouts can block scenarios indefinitely Port-forwarding bypasses Service load balancing Summed component peaks overstate combined peak memory Counter resets cause scenario metrics to be undercounted Dashboard instructions point to Grafana instead of Perses
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Shared baseline resource names and unsafe cleanup can overwrite or delete concurrent runs’ or pre-existing resources, leaving critical isolation and ownership issues unresolved.
15 open issues
Enforce the 30-second minimum run duration · New Stale router metrics can be accepted as post-load results Use run-specific resources and ownership-scoped cleanup Regression tests are not included in the unit-test recipe Resolve relative oha paths before invoking subprocesses Warmup can hang indefinitely on an unanswered request Reject multiple baseline replicas when port-forwarding to one Pod Measure combined memory peak from simultaneous component usage CPU evaluation can miss load-period usage Port-forwarding bypasses baseline replica load balancing Requests without timeouts can block scenarios indefinitely Port-forwarding bypasses Service load balancing Summed component peaks overstate combined peak memory Counter resets cause scenario metrics to be undercounted Dashboard instructions point to Grafana instead of Perses
Review effort: Balanced
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
There was a problem hiding this comment.
🟡 Changes recommended
An output filename ending in .md causes the report to overwrite the JSON results, leaving no machine-readable benchmark output.
3 open issues
15 resolved since last review
Enforce the 30-second minimum run duration Stale router metrics can be accepted as post-load results Use run-specific resources and ownership-scoped cleanup Regression tests are not included in the unit-test recipe Resolve relative oha paths before invoking subprocesses Warmup can hang indefinitely on an unanswered request Reject multiple baseline replicas when port-forwarding to one Pod Measure combined memory peak from simultaneous component usage CPU evaluation can miss load-period usage Port-forwarding bypasses baseline replica load balancing Requests without timeouts can block scenarios indefinitely Port-forwarding bypasses Service load balancing Summed component peaks overstate combined peak memory Counter resets cause scenario metrics to be undercounted Dashboard instructions point to Grafana instead of Perses
Review effort: Balanced
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>
Signed-off-by: James Sturtevant <jsturtevant@gmail.com>



Adds an opt-in same-source Lampyr and Kubernetes comparison with JSON and Markdown results. It provides repeatable measurements for #1 without running scale benchmarks in CI.