Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ exe-suffix := if os() == "windows" { ".exe" } else { "" }
# Kind release installed by `just install-kind` in every workflow (CI, release
# gates, Copilot); providers/kind/templates/kind-config.yaml pins its node image.
kind-version := "v0.33.0"
oha-version := "1.17.0"

default:
@just --list
Expand Down Expand Up @@ -205,6 +206,20 @@ install-mm:
mkdir -p "{{repo-root}}/build/tools"; \
tar --extract --gzip --file "$dir/mm.tar.gz" --directory "{{repo-root}}/build/tools" mm

# Install the pinned oha load generator used by opt-in scale tests.
install-oha:
if [ -x "{{repo-root}}/build/tools/oha" ] && \
[ "$("{{repo-root}}/build/tools/oha" --version)" = "oha {{oha-version}}" ]; then exit 0; fi; \
[ "$(uname -sm)" = "Linux x86_64" ] || { echo "install-oha supports linux-amd64 only" >&2; exit 1; }; \
mkdir -p "{{repo-root}}/build/tools"; \
temporary=$(mktemp "{{repo-root}}/build/tools/oha.XXXXXXXX"); \
trap 'rm -f "$temporary"' EXIT; \
curl --fail --silent --show-error --location --output "$temporary" \
"https://github.com/hatoo/oha/releases/download/v{{oha-version}}/oha-linux-amd64"; \
echo "85c6eb98066b8c8bd513b2dd96fafd8f9863249ea653606bae1ca2e3a75c43d2 $temporary" | sha256sum --check; \
chmod 0755 "$temporary"; \
mv "$temporary" "{{repo-root}}/build/tools/oha"

prepare-workspace: build-ly
if [[ ! -f build/workspace/lampyr.workspace.yaml ]]; then build/ly scaffold workspace build/workspace; fi

Expand Down Expand Up @@ -265,6 +280,7 @@ check-action-versions:

# Run focused unit tests for Lampyr.
test-unit:
python3 "{{repo-root}}/scripts/tests/test_scale_test.py"
go test ./cmd/... ./internal/...
python3 "{{repo-root}}/scripts/tests/test_worker_workspace.py"

Expand Down Expand Up @@ -346,6 +362,23 @@ release-disruption-gate service url:
--service "{{service}}" \
--url "{{url}}"

# Build the normal Kubernetes image from the same Go source as the Lampyr example.
[positional-arguments]
scale-baseline-image image:
docker build --tag "$1" --file scripts/fixtures/scale-workload/Dockerfile .

# Compare Lampyr with a normal Kubernetes Deployment using platform telemetry.
[positional-arguments]
scale-test context service lampyr-url baseline-image duration="30s" concurrency="1,4,16,64" output="build/scale-results.json": install-oha
python3 scripts/scale-test.py \
--context "$1" \
--service "$2" \
--lampyr-url "$3" \
--baseline-image "$4" \
--duration "$5" \
--concurrency "$6" \
--output "$7"

# Run the repository test suites used in CI.
test: test-unit test-cli examples-check test-preview-release cli::test cli::test-intent helper::test azure::test kind::test k3d::test test-examples-local test-kind-integration test-k3d-integration test-existing-kind-integration test-knative-portability test-walkthrough

Expand Down
3 changes: 3 additions & 0 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ Uninstall disables fiber-router trace export before it removes Jaeger.
Remove the add-on before `ly cluster uninstall` when retaining the Kubernetes
cluster; the platform uninstall does not remove optional add-ons.

The [scale framework](scale-testing.md) uses these metrics for repeatable
Lampyr and Kubernetes comparisons.

## Platform metrics

| Metric | Type | Meaning |
Expand Down
73 changes: 73 additions & 0 deletions docs/scale-testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
title: Scale testing
description: Compare Lampyr with the same application in a Kubernetes Deployment.
---

# Scale testing

The opt-in scale framework compares a Lampyr Service with the same Go
application in a normal Kubernetes Deployment. It uses oha for client latency
and throughput, and queries the optional observability add-on for platform
metrics. The live comparison does not run in CI.

Install the add-on and deploy the committed Go standalone example through
Lampyr. Build and publish the normal Kubernetes image from the same source:

```bash
just scale-baseline-image REGISTRY/IMAGE:TAG
docker push REGISTRY/IMAGE:TAG
docker buildx imagetools inspect REGISTRY/IMAGE:TAG
```

Pass the digest-pinned image and the Lampyr URL:

```bash
just scale-test \
CONTEXT \
SERVICE \
https://LAMPYR_URL \
REGISTRY/IMAGE@sha256:DIGEST
```

The default sweep runs each target for 30 seconds at concurrency 1, 4, 16,
and 64. The framework:

- creates a temporary one-replica Deployment and Service with unique names;
- warms both targets;
- runs the same client against Lampyr and Kubernetes;
- queries Prometheus for request, Fiber, CPU, and memory measurements;
- writes `build/scale-results.json` after every scenario; and
- writes a concise Markdown comparison beside it.

The Kubernetes target uses a local Service port-forward, while the Lampyr URL
uses its configured ingress. Record that path difference when interpreting
results. The result metadata records both paths. The framework monitors the
port-forward and stops if it exits, but the tunnel can still limit high
concurrency results. This framework is for repeatable comparisons, not a claim
that the two network paths have identical overhead.

The runner requires one baseline replica and a duration of at least 30 seconds.
Both targets use a 10-second request timeout, including warmup. Cleanup removes
only resources this run created.
Custom JSON output names must not use the `.md` extension.
The context must allow read access to the measured Pods.

The report calculates HTTP success from response status codes and transport
errors. Its latency columns include only successful HTTP responses. Platform
error and shed counts come from Lampyr metrics. The runner waits for a final
healthy Prometheus scrape from every router after each load interval. It also
requires fresh cAdvisor samples from the measured containers and fresh
kube-state-metrics inputs. Missing or changed input series reject the scenario.
Coverage must match the running Pods from the Kubernetes API.
It can wait for up to five more scrape intervals. Counter resets reject the
scenario. The measurement window includes the pre-load router and resource
samples after warmup, and the final collection interval. Peak memory is the
maximum combined usage at one time.

Use the Lampyr Platform Perses dashboard while a run is active. The JSON
result remains the source for comparisons across commits or machines. It
records the image digest, namespace, replica count, oha version, Git state, and
a limited node summary without node addresses or machine identifiers.

Keep the node type, cluster configuration, workload image, Lampyr settings,
duration, and concurrency sweep unchanged when comparing revisions.
2 changes: 1 addition & 1 deletion k8s/observability.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ spec:
image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0@sha256:42cfe3723a5f058171c627537fb57a3ea0f26e4380fa18555a95cb1a1b4cfc5b
args:
- --resources=deployments,namespaces,nodes,pods
- --metric-labels-allowlist=pods=[lampyr.dev/home]
- --metric-labels-allowlist=pods=[lampyr.dev/home,serving.knative.dev/revision]
ports:
- name: metrics
containerPort: 8080
Expand Down
13 changes: 13 additions & 0 deletions scripts/fixtures/scale-workload/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# syntax=docker/dockerfile:1
FROM golang:1.26.7-bookworm@sha256:e8c859f5632dcfde7b32d2012b4351728f6437930887c2f6a91ea242459e5514 AS build
WORKDIR /src
COPY docs/examples/hello-go-standalone/go.mod docs/examples/hello-go-standalone/go.sum ./
RUN go mod download
COPY docs/examples/hello-go-standalone/application.go docs/examples/hello-go-standalone/main.go ./
RUN CGO_ENABLED=0 go build -mod=readonly -trimpath -ldflags="-s -w" -o /service .

FROM scratch
COPY --from=build /service /service
ENV PORT=8080
USER 65532:65532
ENTRYPOINT ["/service"]
Loading