Design proposal: a dialect-generic parser where dialects override and reuse grammar rules (v2) #2910

This issue proposes a design for how dialects work in sqlparser v2. It follows on from the v2 proposal in #2893 and covers one part of it: how the parser is generic over a dialect, and how dialects override and reuse grammar rules.

This is not just a sketch. There is a working implementation of the whole design:

• Parser: andygrove/datafusion-sqlparser-rs@6f8e464 (branch v2-proposal). It implements all 14 built-in dialects other than ANSI and Generic. It accepts 99.1% of the statements that the current crate's tests write for a specific dialect, in that dialect.
• Tested with DataFusion: feat: port the SQL frontend to the proposed sqlparser v2 (draft, do not merge) datafusion#27304 ports DataFusion's whole SQL frontend to it: DFParser, the SQL planner, the unparser and DataFusion's own SQL dialect. All of DataFusion's tests pass, including all 546 sqllogictest files. The PR is a draft and is not meant to be merged. It shows that the design works for the crate's largest user.

Feedback on the design is welcome, especially from people who maintain a dialect or a custom dialect downstream.

The problem

Today, dialect logic lives in the core parser. A dialect can only step in at a few entry points (parse_statement, parse_prefix, parse_infix, ...). So any difference in the middle of a statement becomes a dialect_of! check or a supports_* flag inside a parser of more than 23,000 lines. Every dialect shares one grammar, so syntax leaks between dialects, and reserved words from one dialect affect all the others.

1. The parser is generic over the dialect

Parser<'a, D: Dialect> is generic over a dialect type rather than holding a &dyn Dialect:

let statements = parse_sql::<Postgres>("SELECT x::int FROM t")?;

Every grammar rule is a method on the Dialect trait. Each method's default implementation is the core rule, which is a free function in sqlparser::grammar:

pub trait Dialect: 'static + Sized {
    type Ext: AstExt;

    fn parse_statement<'a>(p: &mut Parser<'a, Self>) -> PResult<Statement<'a, Self::Ext>> {
        grammar::parse_statement(p)
    }

    fn parse_join<'a>(p: &mut Parser<'a, Self>) -> PResult<Option<Join<'a, Self::Ext>>> {
        grammar::parse_join(p)
    }

    fn parse_limit<'a>(p: &mut Parser<'a, Self>) -> PResult<Limit<'a, Self::Ext>> {
        grammar::parse_limit(p)
    }

    // ... about 150 hooks in all, from lexing (`lex_token`, `is_reserved`)
    // through expressions (`parse_prefix`, `infix_rule`) to statements.
}

The core grammar has no knowledge of any specific dialect. Core is simply impl Dialect for Core { type Ext = Builtin; }, with every rule left as the default.

Because the dialect is a type parameter, every call to a hook is resolved at compile time and can be inlined. The core parser has no dialect_of! checks and no runtime flag lookups. For code that only learns the dialect at runtime, DialectKind and builtin::parse_sql_as(kind, sql) dispatch once at the top to the right monomorphized parser.

2. Rules call each other through D::, so an override applies everywhere

This is the key rule of the design. A core rule never calls another core rule directly. It always goes through the dialect, either as D::rule(p) or as a Parser method that forwards to D:::

pub fn parse_statement<'a, D: Dialect>(p: &mut Parser<'a, D>) -> PResult<Statement<'a, D::Ext>> {
    if p.at_keyword("INSERT") {
        return D::parse_insert_statement(p);
    }
    if p.at_keyword("CREATE") {
        return D::parse_create(p);
    }
    // ...
}

impl<'a, D: Dialect> Parser<'a, D> {
    pub fn parse_table_ref(&mut self) -> PResult<TableRef<'a, D::Ext>> {
        spanned!(self, D::parse_table_ref(self))
    }
}

So when MySQL overrides parse_limit to handle LIMIT 5, 10, the override applies in every place a LIMIT can appear: top-level queries, subqueries, CTEs and set operations. None of those call sites need to change. A dialect overrides exactly the rules that differ and inherits the rest:

impl Dialect for MySql {
    type Ext = Builtin;

    fn parse_limit<'a>(p: &mut Parser<'a, Self>) -> PResult<Limit<'a, Builtin>> {
        // Parses `LIMIT count` and MySQL's `LIMIT offset, count`.
    }
}

An override usually handles its own syntax first and then falls back to the core rule by calling grammar::rule(p), much like calling super in an object-oriented language.

3. Dialects build on each other by delegating to generic rules

Rust has no implementation inheritance between types, so a dialect cannot "extend" another dialect type. Instead, each dialect exports its rules as public functions that are generic over any dialect, and other dialects delegate to them hook by hook:

// In spark.rs: Spark's join rule, usable by any dialect.
pub fn parse_join<'a, D: Dialect>(p: &mut Parser<'a, D>) -> PResult<Option<Join<'a, D::Ext>>> {
    let Some(kind) = eat_semi_or_anti(p)? else {
        return grammar::parse_join(p); // Fall back to the core
    };
    let table = p.parse_table_ref()?; // Dispatches to D::parse_table_ref
    let constraint = grammar::parse_join_constraint(p)?;
    Ok(Some(Join { kind, spelled_out: false, table, constraint }))
}

// In databricks.rs
impl Dialect for Databricks {
    type Ext = Builtin;

    fn parse_join<'a>(p: &mut Parser<'a, Self>) -> PResult<Option<Join<'a, Builtin>>> {
        spark::parse_join(p)
    }

    fn parse_qualify<'a>(p: &mut Parser<'a, Self>) -> PResult<Option<Expr<'a, Builtin>>> {
        snowflake::parse_qualify(p)
    }

    fn is_reserved(word: &str) -> bool {
        !word.eq_ignore_ascii_case("ALL") && grammar::is_reserved(word)
    }
    // ...
}

How the "inheritance" works

When Databricks calls spark::parse_join, the call is monomorphized as spark::parse_join::<Databricks>. Inside Spark's rule, every nested call (p.parse_table_ref(), D::parse_expr, ...) dispatches to Databricks' hooks, not Spark's. This is the same "late binding" that virtual methods on self give in an object-oriented language. A borrowed rule is not a frozen copy of the other dialect's behavior. It runs in the borrowing dialect's grammar. For example, Spark's join rule parses the joined table using Databricks' table factor rule, along with any Databricks-only syntax that rule accepts.

So the model is:

There are two differences from class inheritance, and they are deliberate:

• Inheritance is per rule, not per dialect. A dialect is not tied to one parent. Databricks takes most of its rules from Spark, QUALIFY from Snowflake, and others from BigQuery and Oracle. Redshift is built on PostgreSQL. A dialect takes exactly the rules it wants from whichever dialects have them, as mixins.
• Inheritance is explicit. A dialect writes one short forwarding method for each rule it borrows. It never picks up another dialect's syntax by accident, so a change to Spark can only affect Databricks through rules that Databricks chose to share. This is what keeps dialects strict: PostgreSQL rejects backtick-quoted names, and only PostgreSQL has CREATE OPERATOR.

Rules are written so that they can be shared. For example, spark::parse_join is generic over any D: Dialect. A rule that builds nodes of the shared built-in AST is bounded by D: Dialect<Ext = Builtin> instead.

4. Operators: Pratt parsing with rules from the dialect

Expressions use a Pratt parser. For each token after an operand, the parser asks the dialect's infix_rule hook how to handle it. The answer is an InfixRule that holds the binding power and the parse action together, so the two cannot drift apart:

fn infix_rule<'a>(p: &Parser<'a, Self>) -> Option<InfixRule<'a, Self>> {
    if p.at_op("??") {
        return Some(InfixRule::binary(prec::ADD, BinaryOp::Named("??".into())));
    }
    grammar::infix_rule(p) // The core operators
}

Binding powers are constants in grammar::prec, and a dialect places its own operators relative to them. Keywords and operators are matched by their source text rather than through a central Keyword or Token enum. So a dialect can add a token (lex_token), an operator (infix_rule) or a keyword without any change to the core.

5. Extending the AST: AstExt

A dialect chooses its AST through an associated type, type Ext: AstExt. AstExt has slots for the places a dialect can add nodes: expressions, statements, table factors, join conditions, order-by items, the end of a query, and the data after SELECT and INSERT:

impl AstExt for Acme {
    type Statement<'a> = Ping<'a>; // Your own node
    type Expr<'a> = NoExt;         // Slots you do not use
    // ...
}

All built-in dialects share one extension type, Builtin, so downstream tools handle a single tree whichever built-in dialect parsed it. AstExt is separate from Dialect for exactly that reason: many dialects can share one AST type. A third-party dialect can reuse Builtin, and borrow any built-in rule, or define its own extension type to add nodes. With the visitor feature, the new nodes join in through #[derive(Visit, VisitMut)].

6. Third-party dialects are equal to built-in ones

Built-in dialects use only the public API, so a dialect in another crate can do everything a built-in one can: add tokens, operators, statements and AST nodes, and borrow rules from any built-in dialect. examples/custom_dialect.rs is a complete example.

DataFusion's port is the real-world test of this. v2 has no GenericDialect, so DataFusion defines its own dialect in its own crate, using only the public API. The dialect is DuckDB's rules plus the syntax DataFusion takes from other dialects:

• LEFT SEMI/ANTI joins, DISTRIBUTE BY and INSERT OVERWRITE, from Spark
• ASOF JOIN and value:key paths, from Snowflake
• LIMIT offset, count, <=>, XOR and backtick-quoted names, from MySQL
• DISTINCT ON and RETURNING, from PostgreSQL
• INSERT OR REPLACE, from SQLite
• a few additions of DataFusion's own, such as RIGHT SEMI/ANTI joins and its bitwise operator precedence

For example, its join rule adds RIGHT SEMI and RIGHT ANTI joins, hands ASOF to snowflake::parse_join, and leaves the rest to the rules underneath. Every hook either delegates to a built-in dialect's rule or is one of these additions. A side benefit is that DataFusion's SQL syntax becomes a defined list of rules rather than "whatever GenericDialect accepts". DataFusion's own statements (CREATE EXTERNAL TABLE, COPY TO, ...) are built on the same Parser API.

Trade-offs

• More code is generated. Each dialect gets its own monomorphized copy of the rules it uses. This is the cost of static dispatch and inlining.
• Borrowing a rule takes some boilerplate. A dialect writes one forwarding method for each rule it borrows, instead of inheriting all of another dialect's rules at once. This is intended (see above), but it is visible: databricks.rs is mostly forwarding methods.
• Choosing where the hooks go. A dialect can only change the grammar where there is a hook. Too few hooks bring back dialect checks inside rules, and too many make the trait hard to navigate. The current set grew from porting all 14 dialects and DataFusion. Where to draw that line is the main open question.
• No GenericDialect. Strict dialects mean you have to know which database the SQL is for. A downstream crate that wants a permissive dialect can build one by borrowing rules from several dialects, as DataFusion does.

Questions for discussion

1. Is per-rule delegation the right model for sharing rules between dialects, or would people prefer a way to inherit all of another dialect's rules (for example, a macro that forwards every hook to a parent dialect)?
2. Is the current set of hooks in the right places, and is about 150 hooks the right size for the Dialect trait?
3. Are the AstExt slots enough for the custom dialects people maintain today?

The README and AGENTS.md in the branch go into more detail on the rest of the design, including the zero-copy AST and the strict dialects.