Skip to content

Commit 6794064

Browse files
authored
feat(native): add core types, exact JSON emitter and config parsing (#992)
* feat(native): add core identities, diagnostics, text positions and typed SQL values * feat(native): add an exact Python JSON emitter with oracle tests The emitter refuses integers beyond Python's 4300-digit string limit and values nested deeper than 128 levels instead of exhausting the stack. orjson is required at 3.11.7 or later, whose float layout the emitter reproduces, and a test pins that layout. * feat(native): add a sqlbuild-config crate with tomllib-compatible TOML and project config reading Adds TOML loading that matches tomllib, rejecting TOML 1.1-only syntax, and typed project and local config readers for the fields discovery needs. Any native error defers to the Python loader. Internal oracle hooks compare both against Python on seeded and repository documents. Tables keep tomllib's order of first creation even when an implicit table is declared later, CRLF is normalized and a leading byte order mark is rejected as tomllib does. Container depth and dotted key length are bounded before parsing so deep documents are refused instead of exhausting the stack. tomllib's namespace rules are replayed over the parser events, so dotted keys into arrays of tables or declared tables, headers over dotted or inline tables, and extended inline tables are rejected as tomllib rejects them. * feat(native): add a PyYAML-compatible YAML 1.1 loader with oracle tests Composes saphyr-parser events into PyYAML's node graph and emulates the YAML 1.1 resolver and SafeConstructor, including merge keys, base-60 numbers, timestamps and duplicate-key handling. Constructs PyYAML or LibYAML handle differently defer to Python. Oracle tests compare against SafeLoader and CSafeLoader on seeded and repository documents. Literal and folded scalars are read by a port of PyYAML's block scalar scanner, cross-checked against the parser, so chomping matches PyYAML with or without a final line break. A leading byte order mark is skipped. Constructs the two PyYAML loaders read differently, such as tabs, '?' inside flow scalars and directives, are deferred, and the oracle tests require deferral wherever SafeLoader and CSafeLoader disagree. Shared nodes are built once, and construction depth plus a size budget proportional to the input bound alias and merge expansion. Property lookups use code positions computed once per document and mapping keys are hashed by Python equality, so large single-line flow collections load in linear time.
1 parent 7f7a4ff commit 6794064

98 files changed

Lines changed: 7747 additions & 8 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 74 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
members = [
33
"crates/sqlbuild-core",
44
"crates/sqlbuild-sqltext",
5+
"crates/sqlbuild-config",
56
"crates/sqlbuild-analysis",
67
"crates/sqlbuild-rules",
78
"crates/sqlbuild-python",
@@ -15,6 +16,7 @@ resolver = "2"
1516
order = [
1617
"sqlbuild-core",
1718
"sqlbuild-sqltext",
19+
"sqlbuild-config",
1820
"sqlbuild-analysis",
1921
"sqlbuild-rules",
2022
"sqlbuild-python",
@@ -53,12 +55,14 @@ polyglot-sql-function-catalogs = { git = "https://github.com/chio-labs/polyglot.
5355
] }
5456
quote = "1"
5557
rayon = "1"
58+
saphyr-parser = "0.1"
5659
regex = "1"
5760
serde = { version = "1", features = ["derive"] }
5861
# Every crate relies on insertion-ordered JSON objects, which the extension has always built with.
5962
serde_json = { version = "1", features = ["preserve_order"] }
6063
sha2 = "0.10"
6164
sqlbuild-analysis = { path = "crates/sqlbuild-analysis" }
65+
sqlbuild-config = { path = "crates/sqlbuild-config" }
6266
sqlbuild-core = { path = "crates/sqlbuild-core" }
6367
sqlbuild-rules = { path = "crates/sqlbuild-rules" }
6468
sqlbuild-sqltext = { path = "crates/sqlbuild-sqltext" }
@@ -67,6 +71,8 @@ sqlparser = { version = "0.58", features = ["visitor"] }
6771
syn = { version = "2", features = ["full", "visit", "extra-traits"] }
6872
tempfile = "3"
6973
toml = "0.9"
74+
toml_edit = { version = "0.25", default-features = false, features = ["parse"] }
75+
toml_parser = "1"
7076
walkdir = "2"
7177

7278
[workspace.lints.rust]

‎crates/README.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,9 @@ that can be tested and benchmarked with `cargo test` alone.
66

77
| Crate | Responsibility |
88
|---|---|
9-
| `sqlbuild-core` | Constants shared by several crates and the panic boundary that turns an unwinding native failure into a named compiler error. |
9+
| `sqlbuild-core` | Constants shared by several crates, the panic boundary that turns an unwinding native failure into a named compiler error, passive identity, diagnostic and SQL value types, text positions, and a JSON emitter whose output matches Python's `json.dumps` and `orjson` byte for byte. |
1010
| `sqlbuild-sqltext` | Lexical SQL text without polyglot: comment, quote and parenthesis scanning, the rules quote policy, model header tokenization and matching, static variable substitution and static reference extraction. |
11+
| `sqlbuild-config` | Configuration file reading without Python: `tomllib`-compatible TOML, PyYAML `safe_load`-compatible YAML 1.1, and typed project and local config readers for the fields discovery needs. Any error means "defer to Python", which re-parses for exact messages. |
1112
| `sqlbuild-analysis` | SQL analysis over polyglot: SQL tokens, query analysis, the binding catalog, semantic validation and usage, column references, and SQL-test extraction, planning and rendering. |
1213
| `sqlbuild-rules` | Built-in rules and the rules engine, the custom-rule host, SQL lint, quality checks and formatting, rules configuration and the request models. It also owns the build identity script. |
1314
| `sqlbuild-python` | The only PyO3 crate: the `_native` module, its Python classes and functions, conversions from Python objects, and the process allocator. |
@@ -18,9 +19,13 @@ Dependencies point one way, from the top of this graph to the bottom:
1819

1920
```text
2021
sqlbuild-python
21-
-> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-sqltext -> sqlbuild-core
22+
-> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-config -> sqlbuild-sqltext -> sqlbuild-core
2223
```
2324

25+
`sqlbuild-config` does not depend on the crates below it today; its place in the order only
26+
fixes which crates may use it. The JSON emitter lives in `sqlbuild-core` so that any layer can
27+
produce text that must equal Python's `json.dumps` output.
28+
2429
Each crate may depend only on crates below it, and may also skip layers. The order is declared
2530
in `[workspace.metadata.native-layers]` in the workspace `Cargo.toml`, and
2631
`make check-native-layering` (part of `make check-ci`) fails when:

‎crates/sqlbuild-config/Cargo.toml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
[package]
2+
name = "sqlbuild-config"
3+
version.workspace = true
4+
description = "Project configuration reading for SQLBuild: TOML and PyYAML-compatible YAML."
5+
edition.workspace = true
6+
license.workspace = true
7+
publish.workspace = true
8+
rust-version.workspace = true
9+
10+
[dependencies]
11+
saphyr-parser.workspace = true
12+
toml_edit.workspace = true
13+
toml_parser.workspace = true
14+
15+
[dev-dependencies]
16+
tempfile.workspace = true
17+
18+
[lints]
19+
workspace = true
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
//! Calendar limits of Python's `datetime` types and characters shared by the readers.
2+
3+
pub(crate) const MIN_YEAR: i64 = 1;
4+
pub(crate) const MAX_YEAR: i64 = 9999;
5+
pub(crate) const MONTHS_PER_YEAR: i64 = 12;
6+
pub(crate) const FEBRUARY: i64 = 2;
7+
pub(crate) const MAX_HOUR: i64 = 23;
8+
pub(crate) const MAX_MINUTE: i64 = 59;
9+
pub(crate) const MAX_SECOND: i64 = 59;
10+
pub(crate) const MAX_MICROSECOND: i64 = 999_999;
11+
pub(crate) const BYTE_ORDER_MARK: char = '\u{feff}';
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
//! Structured failures of reading configuration files.
2+
3+
/// Why a configuration document could not be loaded natively.
4+
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
5+
pub enum ConfigErrorKind {
6+
/// The text is not a valid document; Python's parser rejects it too.
7+
Syntax,
8+
/// The document parses but a value cannot be built, as PyYAML's constructor or `tomllib` reports.
9+
Construct,
10+
/// The document uses a feature the native reader leaves to Python.
11+
Unsupported,
12+
/// A required file does not exist.
13+
Missing,
14+
/// A field SQLBuild reads has the wrong type or an invalid value.
15+
InvalidField,
16+
}
17+
18+
/// A configuration failure; the Python reader re-parses the file to report its exact message.
19+
#[derive(Clone, Debug, PartialEq, Eq)]
20+
pub struct ConfigError {
21+
pub kind: ConfigErrorKind,
22+
pub message: String,
23+
pub line: Option<usize>,
24+
pub column: Option<usize>,
25+
}
26+
27+
impl ConfigError {
28+
/// An error without a position.
29+
pub fn new(kind: ConfigErrorKind, message: impl Into<String>) -> Self {
30+
Self {
31+
kind,
32+
message: message.into(),
33+
line: None,
34+
column: None,
35+
}
36+
}
37+
}

‎crates/sqlbuild-config/src/lib.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
//! Project configuration reading: `tomllib`-compatible TOML and PyYAML-compatible YAML.
2+
3+
#![forbid(unsafe_code)]
4+
5+
pub(crate) mod constants;
6+
pub mod errors;
7+
pub mod models;
8+
pub mod project;
9+
pub mod toml;
10+
pub mod yaml;
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
//! Configuration values with the value domain of Python's `tomllib` and PyYAML `safe_load`.
2+
3+
use crate::constants::{
4+
FEBRUARY, MAX_HOUR, MAX_MICROSECOND, MAX_MINUTE, MAX_SECOND, MAX_YEAR, MIN_YEAR,
5+
MONTHS_PER_YEAR,
6+
};
7+
8+
/// A calendar date, Python's `datetime.date`.
9+
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
10+
pub struct ConfigDate {
11+
pub year: u16,
12+
pub month: u8,
13+
pub day: u8,
14+
}
15+
16+
/// A time of day with microseconds, Python's naive `datetime.time`.
17+
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
18+
pub struct ConfigTime {
19+
pub hour: u8,
20+
pub minute: u8,
21+
pub second: u8,
22+
pub microsecond: u32,
23+
}
24+
25+
/// A date and time, Python's `datetime.datetime`, with its UTC offset in seconds when aware.
26+
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
27+
pub struct ConfigDateTime {
28+
pub date: ConfigDate,
29+
pub time: ConfigTime,
30+
pub utc_offset_seconds: Option<i32>,
31+
}
32+
33+
impl ConfigDate {
34+
/// The date, or `None` where Python's `datetime.date` constructor raises.
35+
pub fn checked(year: i64, month: i64, day: i64) -> Option<Self> {
36+
let leap = year % 4 == 0 && (year % 100 != 0 || year % 400 == 0);
37+
let month_days = match month {
38+
FEBRUARY => 28 + i64::from(leap),
39+
4 | 6 | 9 | 11 => 30,
40+
_ => 31,
41+
};
42+
let valid = (MIN_YEAR..=MAX_YEAR).contains(&year)
43+
&& (1..=MONTHS_PER_YEAR).contains(&month)
44+
&& (1..=month_days).contains(&day);
45+
valid.then_some(Self {
46+
year: year as u16,
47+
month: month as u8,
48+
day: day as u8,
49+
})
50+
}
51+
}
52+
53+
impl ConfigTime {
54+
/// The time, or `None` where Python's `datetime.time` constructor raises.
55+
pub fn checked(hour: i64, minute: i64, second: i64, microsecond: i64) -> Option<Self> {
56+
let valid = (0..=MAX_HOUR).contains(&hour)
57+
&& (0..=MAX_MINUTE).contains(&minute)
58+
&& (0..=MAX_SECOND).contains(&second)
59+
&& (0..=MAX_MICROSECOND).contains(&microsecond);
60+
valid.then_some(Self {
61+
hour: hour as u8,
62+
minute: minute as u8,
63+
second: second as u8,
64+
microsecond: microsecond as u32,
65+
})
66+
}
67+
}
68+
69+
/// One loaded configuration value; mappings keep document order and Python key equality.
70+
#[derive(Clone, Debug, PartialEq)]
71+
pub enum ConfigValue {
72+
Null,
73+
Bool(bool),
74+
Integer(i64),
75+
BigInteger(String),
76+
Float(f64),
77+
String(String),
78+
Date(ConfigDate),
79+
DateTime(ConfigDateTime),
80+
Time(ConfigTime),
81+
List(Vec<ConfigValue>),
82+
Map(Vec<(ConfigValue, ConfigValue)>),
83+
}
84+
85+
impl ConfigValue {
86+
/// The entries of a mapping, or `None` for any other value.
87+
pub fn as_map(&self) -> Option<&[(ConfigValue, ConfigValue)]> {
88+
match self {
89+
Self::Map(entries) => Some(entries),
90+
_ => None,
91+
}
92+
}
93+
94+
/// The value of a string key in a mapping, as Python's `mapping.get(key)` finds it.
95+
pub fn get(&self, key: &str) -> Option<&ConfigValue> {
96+
self.as_map()?
97+
.iter()
98+
.find(|(candidate, _)| matches!(candidate, Self::String(text) if text == key))
99+
.map(|(_, value)| value)
100+
}
101+
}

0 commit comments

Comments
 (0)