Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 135 additions & 0 deletions .github/workflows/regressdb.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
name: Regression database tests

# ACE on two nodes that hold the regression database of PostgreSQL
# (make installcheck). The nodes differ in server encoding, in the defaults
# of the output settings and, in one case, in the major version. See
# tests/regressdb/README.md.

on:
pull_request:
branches: [ main ]
workflow_dispatch:

permissions:
contents: read

concurrency:
group: regressdb-${{ github.ref }}
cancel-in-progress: true

env:
# The oldest and the newest major version that ACE supports. Before
# PostgreSQL 14, table-diff fails on any difference: it reads the origin
# of each differing row with pg_xact_commit_timestamp_origin().
PG_OLD: "14"
PG_NEW: "19"

jobs:
versions:
runs-on: ubuntu-latest
outputs:
old: ${{ steps.v.outputs.old }}
new: ${{ steps.v.outputs.new }}
steps:
- uses: actions/checkout@v4
- id: v
# A failed lookup in a plain assignment stops the step; inside the
# argument of echo it would publish an empty version.
run: |
old=$(tests/regressdb/pg-version.sh "$PG_OLD")
new=$(tests/regressdb/pg-version.sh "$PG_NEW")
echo "old=$old" >> "$GITHUB_OUTPUT"
echo "new=$new" >> "$GITHUB_OUTPUT"

build:
needs: versions
runs-on: ubuntu-latest
strategy:
matrix:
which: [ old, new ]
env:
VERSION: ${{ matrix.which == 'old' && needs.versions.outputs.old || needs.versions.outputs.new }}
steps:
- uses: actions/checkout@v4

- id: cache
uses: actions/cache@v4
with:
path: ~/pg/${{ env.VERSION }}
key: pg-${{ env.VERSION }}-${{ runner.os }}-${{ hashFiles('tests/regressdb/build-pg.sh') }}

- name: Build PostgreSQL ${{ env.VERSION }}
if: steps.cache.outputs.cache-hit != 'true'
run: |
sudo apt-get update -qq
sudo apt-get install -y -qq bison flex libxml2-dev libicu-dev zlib1g-dev
tests/regressdb/build-pg.sh "$VERSION" ~/pg/"$VERSION"

test:
needs: [ versions, build ]
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
include:
- cell: old
n1: old
n2: old
- cell: new
n1: new
n2: new
- cell: mixed
n1: old
n2: new
name: test (${{ matrix.cell }})
env:
V1: ${{ matrix.n1 == 'old' && needs.versions.outputs.old || needs.versions.outputs.new }}
V2: ${{ matrix.n2 == 'old' && needs.versions.outputs.old || needs.versions.outputs.new }}
steps:
- uses: actions/checkout@v4

- uses: actions/setup-go@v5
with:
go-version-file: go.mod

- uses: actions/cache/restore@v4
with:
path: ~/pg/${{ env.V1 }}
key: pg-${{ env.V1 }}-${{ runner.os }}-${{ hashFiles('tests/regressdb/build-pg.sh') }}
fail-on-cache-miss: true

- uses: actions/cache/restore@v4
if: env.V2 != env.V1
with:
path: ~/pg/${{ env.V2 }}
key: pg-${{ env.V2 }}-${{ runner.os }}-${{ hashFiles('tests/regressdb/build-pg.sh') }}
fail-on-cache-miss: true

- name: Install packages
run: |
sudo apt-get update -qq
# The -dev packages bring the libraries that the cached build uses.
sudo apt-get install -y -qq libxml2-dev libicu-dev locales
# lc_monetary of node n2 (see setup-node.sh).
sudo locale-gen de_DE.UTF-8

- name: Build ACE
run: go build -o "$RUNNER_TEMP/ace" ./cmd/ace

- name: Run ACE on PostgreSQL ${{ env.V1 }} and ${{ env.V2 }}
run: |
tests/regressdb/run-cell.sh "${{ matrix.cell }}" ~/pg/"$V1" ~/pg/"$V2" \
"$RUNNER_TEMP/regressdb" "$RUNNER_TEMP/ace"

- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: regressdb-${{ matrix.cell }}
path: |
${{ runner.temp }}/regressdb/run/results.tsv
${{ runner.temp }}/regressdb/run/logs
${{ runner.temp }}/regressdb/*.log
${{ runner.temp }}/regressdb/*.regress/regression.diffs
if-no-files-found: ignore
38 changes: 38 additions & 0 deletions tests/regressdb/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# ACE on the regression database

This test runs ACE on two nodes that hold the regression database of PostgreSQL, the database that `make installcheck` leaves behind. It has many data types and objects that the integration tests do not have: geometric types, `money`, `tsvector`, `xml`, range and composite types, domains, arrays, partitioned and inherited tables, and many tables without a primary key.

The GitHub workflow `.github/workflows/regressdb.yml` runs it on each pull request to `main`, in three cases:

| Case | Node n1 | Node n2 |
|------|---------|---------|
| `old` | oldest supported major version | same version |
| `new` | newest major version | same version |
| `mixed` | oldest supported major version | newest major version |

In each case node n1 is a UTF8 server with default settings, and node n2 is a LATIN1 server with other defaults for `DateStyle`, `IntervalStyle`, `TimeZone`, `extra_float_digits`, `bytea_output`, `lc_monetary`, `search_path` and `xmloption` (set with `ALTER DATABASE` after the regression tests). Each node runs the regression tests of its own version. The oldest and the newest version are set in the workflow (`PG_OLD`, `PG_NEW`).

## What is checked

`run.py` computes a reference result for each table with plain SQL and checks the result of each ACE command against it:

- `table-diff` for each table, with `--max-connections` 1, the default, and 32, and blocks of 50 rows.
- `mtree build` and `mtree table-diff` for each table with a primary key, with the minimum (3 for `mtree build`, 2 for `mtree table-diff`), the default and 32 connections. Each limit builds its own tree, compares with it and removes it (`mtree teardown-table`). A table with child tables and an empty table must be refused with their own messages.
- `schema-diff --ddl-only` and `schema-diff --compare=structure` for each schema, and `schema-diff` in data mode for `public` with `--max-connections` 1, the default and 32.

The reference result of a table with a primary key comes from both nodes: each row is read as the text of its key and the text of the whole row, with the same output settings that ACE sets on its connections. The number of keys that are on one node only or have a different row text is the number of differences that ACE must report. In a table with inheritance children one key can have several rows; there the rows of a key are compared as a multiset, and only match or diff is checked. A table without columns or a primary key, a table with other key or other columns on the two nodes, or a table on one node only, must give the error that names this problem; when the two nodes have different problems, the error of either node is right. `schema-diff --ddl-only` and `--compare=structure` must find no differences when both nodes run the same major version. `schema-diff` in data mode must report as many matching, differing, missing and failed tables as the reference results give. No command may panic, be ended by a signal, or run longer than the time limit.

Results that differ from this, are known and are not problems of ACE are listed in `known.tsv`, with the reason. A problem of ACE is not listed there: it fails the run until it is fixed. The run also fails on a line of `known.tsv` that no longer matches the result, so that the list stays short and correct.

## Run it locally

On Linux, with the packages that the workflow installs:

```sh
tests/regressdb/build-pg.sh "$(tests/regressdb/pg-version.sh 14)" ~/pg/14
tests/regressdb/build-pg.sh "$(tests/regressdb/pg-version.sh 19)" ~/pg/19
go build -o /tmp/ace ./cmd/ace
tests/regressdb/run-cell.sh mixed ~/pg/14 ~/pg/19 /tmp/regressdb /tmp/ace
```

`RUN_ARGS` passes options to `run.py`, for example `RUN_ARGS='--tables ^public\.gtest --commands table-diff'`. `--tables`, `--commands` and `--conns` are for repeating a part of the run by hand. Such a limited run checks each command it runs in the same way as a full run, but it does not check `known.tsv` for unused lines, and with `--tables` it skips the data mode of `schema-diff`, which always compares the whole schema. The results are in `<workdir>/run/results.tsv`, and the full output of each failed command is in `<workdir>/run/logs`.
45 changes: 45 additions & 0 deletions tests/regressdb/build-pg.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/usr/bin/env bash
#
# Build one PostgreSQL release from the source tarball.
#
# Usage: build-pg.sh <version> <dir>
#
# <dir>/install gets the installed server and client programs.
# <dir>/src keeps the built source tree: the regression tests run from there.
# If <dir>/install/bin/postgres exists, the script does nothing, so a
# restored CI cache is used as it is.

set -euo pipefail

version=${1:?usage: build-pg.sh <version> <dir>}
dir=${2:?usage: build-pg.sh <version> <dir>}

if [ -x "$dir/install/bin/postgres" ]; then
echo "PostgreSQL $version is already built in $dir"
exit 0
fi

mkdir -p "$dir"
dir=$(cd "$dir" && pwd)
rm -rf "$dir/src" "$dir/install"

curl -fsSL "https://ftp.postgresql.org/pub/source/v$version/postgresql-$version.tar.bz2" |
tar -xj -C "$dir"
mv "$dir/postgresql-$version" "$dir/src"

cd "$dir/src"
# libxml and ICU give the regression tests their xml and ICU collation
# data. Readline is not needed: psql runs only scripts here.
./configure --prefix="$dir/install" --with-libxml --with-icu \
--without-readline >"$dir/configure.log" 2>&1 ||
{ tail -n 50 "$dir/configure.log"; exit 1; }
make -j"$(nproc)" -s >"$dir/build.log" 2>&1 ||
{ tail -n 50 "$dir/build.log"; exit 1; }
make -s install >>"$dir/build.log" 2>&1 ||
{ tail -n 50 "$dir/build.log"; exit 1; }

# The regression tests need only pg_regress, the test libraries and the
# test files. Without the object files the CI cache is much smaller.
find "$dir/src" -name '*.o' -delete

"$dir/install/bin/postgres" --version
14 changes: 14 additions & 0 deletions tests/regressdb/known.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Results of run.py that differ from the expected ones and are known.
#
# Columns, separated by tabs: case (old, new, mixed or *), command, object
# (a table or a schema), connections, result, reason. Only the case column
# can be *; the other columns name exactly one run. The result is written
# as in results.tsv, with the number of differences where there is one (for
# example diff:3). A line applies only when the result is exactly the one
# listed. A panic, a timeout or a kill cannot be listed. Two lines with the
# same selector are an error. A line that applies to nothing in a full run
# (no --tables or --commands, all default --conns values) fails the run, so
# remove a line when the cause is fixed.
#
# Only differences that are not problems of ACE belong here. A problem of
# ACE fails the run until it is fixed.
17 changes: 17 additions & 0 deletions tests/regressdb/pg-version.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
#
# Print the latest released minor version of a PostgreSQL major version,
# for example "18.7" for "18". Beta and RC releases are not used.
#
# Usage: pg-version.sh <major>

set -euo pipefail

major=${1:?usage: pg-version.sh <major>}

curl -fsSL https://ftp.postgresql.org/pub/source/ |
grep -o "v${major}\.[0-9]*/" |
tr -d 'v/' |
sort -t. -k2,2n |
tail -n 1 |
grep . || { echo "no release found for PostgreSQL $major" >&2; exit 1; }
53 changes: 53 additions & 0 deletions tests/regressdb/run-cell.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
#!/usr/bin/env bash
#
# Run one case of the regression database test: create two nodes, load the
# regression database on each, and run ACE on them with run.py.
#
# Usage: run-cell.sh <cell> <pgdir1> <pgdir2> <workdir> <ace>
#
# cell name of the case in known.tsv (old, new or mixed)
# pgdir1 directory made by build-pg.sh for node n1
# pgdir2 directory made by build-pg.sh for node n2
# workdir new directory for the data directories, logs and results
# ace ACE binary
#
# Node n1 is a UTF8 server with default settings. Node n2 is a LATIN1
# server with other defaults for the output settings (see setup-node.sh).
# Environment: PORT1 and PORT2 (default 5501 and 5502), and RUN_ARGS with
# extra options for run.py.

set -euo pipefail

cell=${1:?usage: run-cell.sh <cell> <pgdir1> <pgdir2> <workdir> <ace>}
pg1=$(cd "${2:?}" && pwd)
pg2=$(cd "${3:?}" && pwd)
workdir=${4:?}
ace=$(cd "$(dirname "${5:?}")" && pwd)/$(basename "$5")
port1=${PORT1:-5501}
port2=${PORT2:-5502}
here=$(cd "$(dirname "$0")" && pwd)

mkdir -p "$workdir"
workdir=$(cd "$workdir" && pwd)

stop() {
"$pg1/install/bin/pg_ctl" -D "$workdir/n1" -m fast stop >/dev/null 2>&1 || true
"$pg2/install/bin/pg_ctl" -D "$workdir/n2" -m fast stop >/dev/null 2>&1 || true
}
trap stop EXIT

echo "== node n1: $("$pg1/install/bin/postgres" --version), UTF8, default settings"
"$here/setup-node.sh" "$pg1" "$workdir/n1" "$port1" UTF8 default
echo "== node n2: $("$pg2/install/bin/postgres" --version), LATIN1, changed settings"
"$here/setup-node.sh" "$pg2" "$workdir/n2" "$port2" LATIN1 changed

# The newer psql of the two reads the reference rows from both servers.
v1=$("$pg1/install/bin/pg_config" --version | grep -o '[0-9]*' | head -n 1)
v2=$("$pg2/install/bin/pg_config" --version | grep -o '[0-9]*' | head -n 1)
psql="$pg1/install/bin/psql"
[ "$v2" -gt "$v1" ] && psql="$pg2/install/bin/psql"

echo "== ACE"
# shellcheck disable=SC2086
python3 "$here/run.py" --ace "$ace" --psql "$psql" --port1 "$port1" \
--port2 "$port2" --workdir "$workdir/run" --cell "$cell" ${RUN_ARGS:-}
Loading
Loading