Skip to content

D267: statistics aggregates, ADD_MONTHS, MONTHS_BETWEEN and NVL2 from receipts - #127

Merged
kvlonge merged 14 commits into
mainfrom
fix/d267-functions
Nov 6, 2026
Merged

kvlonge merged 14 commits into
mainfrom
fix/d267-functions

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Decision D267 (function rules from the website lane's corpus round 4): the statistics aggregates, ADD_MONTHS/MONTHS_BETWEEN and NVL2, each converted only as the recorded receipts show. Evidence comes from A's e510d1a (23 edge cases) and a586f83 (variance ties, scales, large-mean FLOAT).

Rules

Rule What it converts How the receipts pin it
rule.nvl2 NVL2(a, b, c), exactly three arguments, a not semi-structured CASE WHEN a IS NOT NULL; branches unify without reading text, as recorded for NUMBER, FLOAT, text and untyped NULL. Text beside another type refuses.
rule.variance VARIANCE, VAR_SAMP, VAR_POP, VARIANCE_POP, STDDEV, STDDEV_SAMP, STDDEV_POP of a NUMBER (scale ≤ 12), aggregate or window The exact (n·Σx² − (Σx)²) / (n(n−1)), or / n², rounded half up to NUMBER(38, min(6+2s, 12)). The t009 tie 1/128 gives 0.007813, and the scales 1, 3, 5 and 12 are recorded. STDDEV is the FLOAT sqrt of that rounded variance, bit-exact on every receipt.
rule.variance_float The same of FLOAT or text, without DISTINCT The same formula in doubles; 6.583333333333333 is matched exactly, while the two-pass formula misses by 1 ulp. Sums go through the float_sum_order guard.
rule.covariance COVAR_SAMP, COVAR_POP of NUMBER/FLOAT pairs, aggregate or window (Σxy − Σx·Σy/n) / d over complete pairs, the only one of four candidate orderings that matches all 12 receipts exactly. Sums go through float_sum_order.
rule.add_months ADD_MONTHS of a DATE, TIMESTAMP_NTZ, or TIMESTAMP_TZ whose offset is known DATEADD by month, the count rounded half away from zero (2.50 → 3, −1.50 → −2); a month's last day stays a last day (2023-02-28 + 1 → 03-31), otherwise the day is clamped; the time of day is kept. The DATEADD core is shared, not copied, and DATEADD/TIMEADD verdicts are unchanged (98/8 before and after).
rule.months_between MONTHS_BETWEEN of DATE/TIMESTAMP_NTZ Whole months where the days match or both are month ends, with the clock ignored; otherwise plus (Δdays + Δclock) / 31 days, divided exactly (nanoseconds) and rounded at six decimals into NUMBER(27,6). Six-decimal ties are impossible for DATE inputs (31 is odd); no recording shows a timestamp tie.

What still refuses, on purpose

  • CORR (all 12 cases): no formula reproduces the NUMBER(10,2) receipt 0.9974958437759694 exactly, so it is not approximated. Its obligations cite those refused cases, so CORR stays refused and fails closed.
  • VARIANCE(DISTINCT) and any statistic DISTINCT over an ordered window: Snowflake rejects both (0A000 / 42601). DISTINCT is accepted elsewhere, as recorded.
  • ADD_MONTHS of a TIMESTAMP_TZ column whose offset isn't known (t003).
  • The SYSTEM$TYPEOF decimal cases (t007/t008): refused by the existing TYPEOF guard, not by these rules.

Declared divergences

  • The large-mean FLOAT cases (variance t011, covar_samp/covar_pop t006) now declare float_sum_order. Their sums of products near 1e18 cannot be proven order-exact, so the converted SQL raises loudly instead of guessing Snowflake's cancellation.
  • Policy: the statistics join numeric_range (a NUMBER total beyond 38 digits raises) and float_sum_order; ADD_MONTHS joins timestamp_range, as DATEADD already does.

Refactors (no behaviour change)

  • conversion/constants.rs was at 2,002 lines after main merged in. The TO_CHAR format-element table moved into a text_formats domain together with strftime_pattern/clock_format, its readers. The statistics constants sit beside their only reader.

Measured

  • Full verify on this head (current main merged in): 11,826 cases, 0 failed; 226/226 rules verified. Before this change the D267 cases refused; now +150 passed, +3 declared divergences.
  • Forms, against current main's baseline: 0 regressions (89 → 98 matching without the D272 loader); the baseline is re-recorded. 9 forms go refused → matching: NVL2, MONTHS_BETWEEN, VAR_SAMP (aggregate and over), VARIANCE_POP, STDDEV_SAMP, STDDEV_POP, COVAR_SAMP over, COVAR_POP over. The rest of the family is held back only by the refusals listed above.

Draft. GitHub Actions is blocked by billing; the local D94 gate counts.

23 cases appended to existing files, for receipts before converting:
- STDDEV, STDDEV_SAMP, STDDEV_POP, VARIANCE, VAR_SAMP, VAR_POP,
  VARIANCE_POP, CORR, COVAR_SAMP, COVAR_POP: values near 1e9 with a
  small spread, and NUMBER(10,2) inputs with SYSTEM$TYPEOF.
- ADD_MONTHS: negative counts from month ends, day 31 into shorter
  months, across a year.
- MONTHS_BETWEEN: month ends and equal days with times of day, and a
  long negative span.
Recorded from wip/d267-edges 148695c (23 cases, utc-v1, no failures).
NVL2(a, b, c) of exactly three arguments, a not semi-structured,
converts to CASE WHEN a IS NOT NULL THEN b ELSE c END, its branches
unified without reading text, as the recorded NUMBER, FLOAT, text and
untyped NULL cases show. Text beside another type, a VARIANT first
argument and any other arity refuse.
The receipts show VARIANCE of a NUMBER is a rounded NUMBER and STDDEV
the square root of that rounded value; these cases pin the rounding at
an exact tie, the scale rule beyond scales 0 and 2, and how a FLOAT
variance near 1e9 cancels.
Recorded from fix/d267-functions b204059 (3 cases, utc-v1, no failures).
# Conflicts:
#	crates/sqlquack/src/analysis/tests/test_coverage_batch.rs
conversion/constants.rs reached 2,002 lines with main merged in. The
TO_CHAR format elements, month-name adjacency, bare FF and separators
now live in text_formats with strftime_pattern and clock_format, which
read them. No behaviour change.
VARIANCE, VAR_SAMP, VAR_POP and VARIANCE_POP of a NUMBER of scale s are
the exact (n*sum(x^2) - sum(x)^2) / (n(n-1)), or / n^2, rounded half up
to NUMBER(38, min(6 + 2s, 12)); STDDEV, STDDEV_SAMP and STDDEV_POP are
the FLOAT square root of that rounded variance, as recorded bit for bit.
FLOAT and text use the same formula in doubles; COVAR_SAMP and
COVAR_POP use (sum(xy) - sum(x)sum(y)/n) / d over complete pairs. Both
rely on float_sum_order, whose guard raises for the large-mean FLOAT
cases, now declared. VARIANCE(DISTINCT) and DISTINCT over an ordered
window refuse, as Snowflake rejects them. CORR still refuses: no
formula reproduces its NUMBER(10,2) receipt.
ADD_MONTHS moves by DATEADD of the months rounded half away from zero,
and a month's last day on to the last day of the month it lands in,
keeping the time of day; a TIMESTAMP_TZ moves on its wall clock where
its offset is known. MONTHS_BETWEEN of DATEs and TIMESTAMP_NTZs is the
whole months where the days match or both are month ends, else the
months plus the day and clock difference over 31 days, exact and
rounded at six decimals into NUMBER(27,6). The statistics constants
move beside their only reader.
Nine forms move from refused to matching: NVL2, MONTHS_BETWEEN,
VAR_SAMP and its window form, VARIANCE_POP, STDDEV_SAMP, STDDEV_POP,
and COVAR_SAMP and COVAR_POP over windows. Nothing regresses.

kvlonge commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Landing gate (D301) on c70e255 (fd1c3b9 + main de14f81; the only conflict, results-baseline.json, taken from main):

  • make check-ci: exit 0
  • make test-rust: every suite passed
  • make verify-recorded: 12033 cases, 7997 passed, 1035 diverged as expected, 2964 refused, 37 typed, 0 pending, 0 unproven, 0 failed; 226 of 226 rules.
  • Translation results against main: 0 regressions, 299 other changes (this PR's rules moving statements from refused to matching, and its new statements). 78910c3 re-records the baseline from that run; against main it still shows 0 regressions, and the baseline holds the measure. Forms: 68 of 1183 match.
  • Audit: approved at fd1c3b9; the merge adds only main.

@kvlonge
kvlonge marked this pull request as ready for review November 6, 2026 21:37
@kvlonge
kvlonge merged commit 78910c3 into main Nov 6, 2026
1 check failed
@kvlonge
kvlonge deleted the fix/d267-functions branch November 7, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant