feat(dupscore): ignore forced adapter overrides and compare call targets

docs/sql-test-coverage-rules.md

@@ -26,14 +26,16 @@ When a model's rows come only from its inputs, this test passes for almost any m
 adds no coverage. `SQBRTEST203` flags a model-mode test when all of the following hold:
 
 - Every mock (`__ref__`, `__source__`, `__seed__`, `__dbt_ref__`, `__table_fn__`) is a single
-  `SELECT` without a set operator that is filtered by `WHERE FALSE` or `WHERE 1 = 0`, limited by
-  `LIMIT 0`, or is `SELECT * FROM __EMPTY_FIXTURE()`. A `UNION` whose last branch is filtered is
-  not empty.
+  `SELECT` without a set operator that is limited by `LIMIT 0`, is `SELECT * FROM __EMPTY_FIXTURE()`,
+  or is filtered by `WHERE FALSE` or `WHERE 1 = 0` without calling any function (an aggregate such
+  as `ORDER BY COUNT(*)` can still return a row). A `UNION` whose last branch is filtered is not
+  empty.
 - Every `__expected__<model>` CTE is empty in the same way, or there is none.
-- Every `__assert__` CTE is a bare row-existence check, `SELECT <anything> FROM __ref("<model>")`
-  on a tested model, with no `WHERE`, `JOIN`, `GROUP BY`, `HAVING`, `QUALIFY`, set operator,
-  `EXISTS`, or subquery.
-- The test has no `__macro__` mocks.
+- Every `__assert__` CTE is a bare row-existence check, `SELECT <columns or constants> FROM
+  __ref("<model>")` on a tested model, with no `WHERE`, `JOIN`, `GROUP BY`, `HAVING`, `QUALIFY`,
+  set operator, `EXISTS`, subquery, `LIMIT`, `OFFSET` or `FETCH`, and no function call in its
+  select list or `ORDER BY`.
+- The test has no `__macro__` mocks and no model query overrides.
 
 Macro, UDF, and table-function tests are never flagged, and SQL that the rules parser cannot read
 is never flagged.

scripts/dupscore/README.md

@@ -13,17 +13,23 @@ Reports concrete function-level clones:
   chars versus lifetimes). Test code is skipped by default: Python `tests/`, Rust `tests/` and
   `benches/` directories, `#[cfg(test)]` items and modules, and `#[test]` functions.
   `--include-tests` analyzes them too. The dbt integration is included, unlike in `report`.
-- Token streams are normalised: identifiers become one placeholder, literals one placeholder per
-  kind, keywords and operators stay. Comments, Python docstrings, annotations, and `->` are
-  dropped.
+- Token streams are normalised: local names, parameters, receivers, and plain attribute reads
+  become one placeholder, literals one placeholder per kind, and keywords and operators stay.
+  Call targets keep their names, so two functions with the same layout that call different
+  helpers do not look identical: Python `f(...)` and `obj.f(...)` keep `f`; Rust keeps any
+  identifier directly followed by `(` or `!` (functions, methods, tuple constructors, macros)
+  except the name after `fn`. Comments, Python docstrings, annotations, and `->` are dropped.
 - Candidates come from MOSS-style winnowed fingerprints: 12-token k-grams, minimum of every
   window of 6 k-grams, so any shared run of 17 or more normalised tokens yields a shared
   fingerprint. Fingerprints present in more than 25 units are boilerplate and ignored. Pairs
   sharing fingerprints with a Jaccard index of at least 0.2 are scored with the `difflib`
   ratio of their normalised streams.
 - Categories: `exact` (identical tokens after dropping comments, docstrings, and annotations),
-  `renamed` (identical normalised stream), `near-miss` (ratio at or above `--min-similarity`,
-  default 0.8). Units under `--min-tokens` normalised tokens (default 60) are ignored.
+  `renamed` (identical normalised stream, so only abstracted names and literals differ),
+  `near-miss` (ratio at or above `--min-similarity`, default 0.8). Units under `--min-tokens`
+  normalised tokens (default 60) are ignored. A near-miss pair whose smaller unit has fewer
+  than 80 tokens must reach a ratio of at least 0.9: short wrappers share most of their
+  skeleton by idiom, so this removes a noisy low end without affecting exact or renamed pairs.
 - Similar pairs are grouped into transitive clusters and ranked by estimated duplicated tokens.
 
 Useful options:
@@ -47,6 +53,31 @@ reason = "Backends intentionally mirror one protocol."
 
 A pair is suppressed when both sides match the entry's globs (`*` also matches `/`).
 
+Contract methods that every implementation must define itself are declared once:
+
+```toml
+[[contract_exemption]]
+contract = "src/sqlbuild/adapter/contract/classes/strict_adapter.py:StrictAdapter"
+paths = ["src/sqlbuild/adapter/contract/classes/*", "src/sqlbuild/adapters/*/classes/*"]
+forbidden_owners = ["src/sqlbuild/adapter/contract/classes/base_adapter.py:BaseAdapter"]
+reason = "The strict adapter contract test requires each adapter to define every method."
+```
+
+The contract method names are the class's abstract methods (`@abstractmethod` or
+`@abc.abstractmethod`, including those inherited from resolvable bases and not overridden),
+parsed with `ast` from the analysed worktree; nothing is imported. A method is a forced override
+when its top-level class lies under `paths`, derives from the contract class, and would
+otherwise inherit that method from the contract, a `forbidden_owners` class, or nowhere. Links
+between two forced overrides are hidden and clusters left without links disappear; links from
+a forced override to any other unit (a private helper copy, or an override that could have been
+inherited instead) are still reported. The summary counts forced overrides hidden this way.
+Forced overrides that remain in a reported cluster are marked `[forced]` in text output
+(`"forced_override": true` in JSON) and listed after the other members, so a cluster's first
+lines show what to fix. They also add nothing to the duplicated-token estimate that ranks
+clusters: a forced override already serves as the copy that must stay.
+An entry whose contract file is absent from the analysed tree is inactive; a missing class in an
+existing file is a configuration error.
+
 ## `dupscore report` and `dupscore pair A B`
 
 The package-pair ranking fused from call-graph, state fan-in, dataclass overlap, same-name, and

scripts/dupscore/_helpers/clones/clustering.py

@@ -19,11 +19,12 @@ def cluster_clone_pairs(
     units: list[CloneUnit],
     pairs: list[ClonePair],
     change_of: Callable[[CloneUnit], str | None],
+    forced: frozenset[int],
 ) -> list[CloneCluster]:
     """Group pairs into connected components ordered by estimated duplicated tokens."""
 
     clusters: list[CloneCluster] = [
-        _build_cluster(units=units, pairs=component, change_of=change_of)
+        _build_cluster(units=units, pairs=component, change_of=change_of, forced=forced)
         for component in _connected_pair_groups(pairs)
     ]
     return sorted(clusters, key=_cluster_sort_key)
@@ -63,10 +64,16 @@ def _build_cluster(
     units: list[CloneUnit],
     pairs: list[ClonePair],
     change_of: Callable[[CloneUnit], str | None],
+    forced: frozenset[int],
 ) -> CloneCluster:
     unit_indexes: list[int] = sorted(
         _unit_indexes(pairs),
-        key=lambda index: (units[index].path, units[index].start_line, units[index].name),
+        key=lambda index: (
+            index in forced,
+            units[index].path,
+            units[index].start_line,
+            units[index].name,
+        ),
     )
     position: dict[int, int] = {unit_index: slot for slot, unit_index in enumerate(unit_indexes)}
     best_similarity: dict[int, float] = {}
@@ -76,6 +83,7 @@ def _build_cluster(
     weights: list[float] = [
         len(units[unit_index].normalized) * best_similarity[unit_index]
         for unit_index in unit_indexes
+        if unit_index not in forced
     ]
     links: list[ClonePairLink] = sorted(
         (
@@ -102,19 +110,30 @@ def _build_cluster(
             end_line=units[unit_index].end_line,
             tokens=len(units[unit_index].normalized),
             change=change_of(units[unit_index]),
+            forced_override=unit_index in forced,
         )
         for unit_index in unit_indexes
     )
     return CloneCluster(
         category=category,
         similarity_min=min(similarities),
         similarity_max=max(similarities),
-        duplicated_tokens=round(sum(weights) - max(weights)),
+        duplicated_tokens=_duplicated_tokens(
+            weights=weights, has_forced=len(weights) < len(members)
+        ),
         members=members,
         links=tuple(links),
     )
 
 
+def _duplicated_tokens(*, weights: list[float], has_forced: bool) -> int:
+    """Estimate removable tokens: keep one copy, or a required forced override when present."""
+
+    if has_forced:
+        return round(sum(weights))
+    return round(sum(weights) - max(weights))
+
+
 def _cluster_sort_key(cluster: CloneCluster) -> tuple[int, float, str, int]:
     first: CloneMember = cluster.members[0]
     return (-cluster.duplicated_tokens, -cluster.similarity_max, first.path, first.start_line)

scripts/dupscore/_helpers/clones/matching.py

@@ -13,6 +13,8 @@
     CLONE_CANDIDATE_MIN_JACCARD,
     CLONE_MAX_FINGERPRINT_UNITS,
     CLONE_MIN_FINGERPRINTS,
+    CLONE_SMALL_NEAR_MISS_MIN_SIMILARITY,
+    CLONE_SMALL_NEAR_MISS_TOKENS,
 )
 from scripts.dupscore.models import ClonePair, CloneUnit
 
@@ -63,12 +65,16 @@ def find_clone_pairs(
             continue
         if shared / (left_size + right_size - shared) < CLONE_CANDIDATE_MIN_JACCARD:
             continue
+        required: float = _required_similarity(
+            smaller=min(len(units[left].normalized), len(units[right].normalized)),
+            min_similarity=min_similarity,
+        )
         similarity: float = _stream_similarity(
             left=units[left].normalized,
             right=units[right].normalized,
-            min_similarity=min_similarity,
+            min_similarity=required,
         )
-        if similarity >= min_similarity:
+        if similarity >= required:
             pairs[(left, right)] = ClonePair(
                 left=left,
                 right=right,
@@ -78,6 +84,14 @@ def find_clone_pairs(
     return [pairs[key] for key in sorted(pairs)]
 
 
+def _required_similarity(*, smaller: int, min_similarity: float) -> float:
+    """Demand closer matches from short units, whose shared skeletons are mostly idioms."""
+
+    if smaller < CLONE_SMALL_NEAR_MISS_TOKENS:
+        return max(min_similarity, CLONE_SMALL_NEAR_MISS_MIN_SIMILARITY)
+    return min_similarity
+
+
 def _stream_similarity(
     *,
     left: tuple[str, ...],

scripts/dupscore/_helpers/clones/python_units.py

@@ -74,7 +74,12 @@ def extract_python_units(*, relative_path: str, source: str) -> tuple[CloneUnit,
     except (SyntaxError, tokenize.TokenError):
         return ()
     starts: list[_Position] = [item.start for item in tokens]
-    normalized: list[str | None] = [_normalize(item) for item in tokens]
+    call_targets: frozenset[_Position] = _call_target_positions(
+        tree=tree, lines=io.StringIO(source).readlines()
+    )
+    normalized: list[str | None] = [
+        _normalize(item=item, call_target=item.start in call_targets) for item in tokens
+    ]
     for span in _ignored_spans(tree.body):
         end: int = bisect_left(starts, span.end)
         if span.statement and end < len(tokens) and tokens[end].type == tokenize.NEWLINE:
@@ -107,9 +112,11 @@ def extract_python_units(*, relative_path: str, source: str) -> tuple[CloneUnit,
     return tuple(units)
 
 
-def _normalize(item: tokenize.TokenInfo) -> str | None:
+def _normalize(*, item: tokenize.TokenInfo, call_target: bool) -> str | None:
     if item.type == tokenize.NAME:
-        return item.string if item.string in _KEYWORDS else PLACEHOLDER_IDENTIFIER
+        if call_target or item.string in _KEYWORDS:
+            return item.string
+        return PLACEHOLDER_IDENTIFIER
     if item.type == tokenize.OP:
         return None if item.string == _RETURN_ARROW else item.string
     if item.type in _FSTRING_BODY_TYPES:
@@ -122,6 +129,39 @@ def _normalize(item: tokenize.TokenInfo) -> str | None:
     return _LITERAL_PLACEHOLDERS.get(item.type, item.string)
 
 
+def _call_target_positions(*, tree: ast.Module, lines: list[str]) -> frozenset[_Position]:
+    """Locate the name tokens of called functions (``f(...)``) and methods (``x.f(...)``)."""
+
+    positions: set[_Position] = set()
+    for node in ast.walk(tree):
+        if not isinstance(node, ast.Call):
+            continue
+        function: ast.expr = node.func
+        if isinstance(function, ast.Name):
+            positions.add(
+                _token_position(lines=lines, line=function.lineno, byte_column=function.col_offset)
+            )
+        elif (
+            isinstance(function, ast.Attribute)
+            and function.end_lineno is not None
+            and function.end_col_offset is not None
+        ):
+            end: _Position = _token_position(
+                lines=lines, line=function.end_lineno, byte_column=function.end_col_offset
+            )
+            positions.add((end[0], end[1] - len(function.attr)))
+    return frozenset(positions)
+
+
+def _token_position(*, lines: list[str], line: int, byte_column: int) -> _Position:
+    """Convert an ``ast`` UTF-8 byte column into the character column ``tokenize`` reports."""
+
+    text: str = lines[line - 1] if 0 < line <= len(lines) else ""
+    if text.isascii():
+        return (line, byte_column)
+    return (line, len(text.encode("utf-8")[:byte_column].decode("utf-8", errors="replace")))
+
+
 def _collect_unit_nodes(*, body: list[ast.stmt], prefix: str) -> list[_UnitNode]:
     collected: list[_UnitNode] = []
     for statement in body:

scripts/dupscore/_helpers/clones/rendering.py

@@ -9,6 +9,7 @@
 from scripts.dupscore.models import CloneCluster, CloneMember, CloneReport
 
 _INDENT: str = "     "
+_FORCED_MARKER: str = " [forced]"
 
 
 def render_clone_text(*, report: CloneReport, top: int) -> str:
@@ -20,7 +21,8 @@ def render_clone_text(*, report: CloneReport, top: int) -> str:
     scope: str = f" changed since {report.since}" if report.since is not None else ""
     lines: list[str] = [
         f"dupscore clones{scope}: {len(report.clusters)} clusters "
-        f"({counts}; {report.allowlisted_pairs} allowlisted pairs hidden)"
+        f"({counts}; {report.allowlisted_pairs} allowlisted pairs hidden; "
+        f"{report.contract_exempt_members} contract-exempt members hidden)"
     ]
     for position, cluster in enumerate(report.clusters[:top], start=1):
         lines.append(f"{position:3d}. {_cluster_heading(cluster)}")
@@ -43,6 +45,7 @@ def render_clone_json(*, report: CloneReport, top: int) -> str:
         "since": report.since,
         "unit_counts": report.unit_counts,
         "allowlisted_pairs": report.allowlisted_pairs,
+        "contract_exempt_members": report.contract_exempt_members,
         "total_clusters": len(report.clusters),
         "clusters": [
             {"rank": position, **asdict(cluster)}
@@ -64,7 +67,8 @@ def _cluster_heading(cluster: CloneCluster) -> str:
 
 def _member_line(member: CloneMember) -> str:
     change: str = f" [{member.change}]" if member.change is not None else ""
+    forced: str = _FORCED_MARKER if member.forced_override else ""
     return (
         f"{member.path}:{member.start_line}-{member.end_line} {member.name} "
-        f"({member.tokens} tokens){change}"
+        f"({member.tokens} tokens){forced}{change}"
     )

scripts/dupscore/_helpers/clones/rust_units.py

@@ -100,6 +100,7 @@
 _RUST_SUFFIX: str = ".rs"
 _DIRECTORY_SUFFIX: str = "/"
 _OWNER_SEPARATOR: str = "::"
+_CALL_SUFFIXES: frozenset[str] = frozenset({"(", "!"})
 
 
 def extract_rust_units(
@@ -123,10 +124,27 @@ def extract_rust_units(
     )
 
 
-def _normalize(item: RustToken) -> str:
-    if item.kind == RUST_KIND_IDENTIFIER:
-        return item.text if item.text in _RUST_KEYWORDS else PLACEHOLDER_IDENTIFIER
-    return _LITERAL_PLACEHOLDERS.get(item.kind, item.text)
+def _normalize_stream(tokens: list[RustToken]) -> list[str]:
+    normalized: list[str] = []
+    for index, item in enumerate(tokens):
+        if item.kind == RUST_KIND_IDENTIFIER:
+            keep: bool = item.text in _RUST_KEYWORDS or _is_call_target(tokens=tokens, index=index)
+            normalized.append(item.text if keep else PLACEHOLDER_IDENTIFIER)
+        else:
+            normalized.append(_LITERAL_PLACEHOLDERS.get(item.kind, item.text))
+    return normalized
+
+
+def _is_call_target(*, tokens: list[RustToken], index: int) -> bool:
+    """Name a called function, method, or macro: an identifier directly before ``(`` or ``!``."""
+
+    following: int = index + 1
+    if following >= len(tokens):
+        return False
+    after: RustToken = tokens[following]
+    if after.kind != RUST_KIND_PUNCTUATION or after.text not in _CALL_SUFFIXES:
+        return False
+    return index == 0 or tokens[index - 1].text != _FUNCTION_KEYWORD
 
 
 def _match_brackets(tokens: list[RustToken]) -> list[int]:
@@ -341,7 +359,7 @@ def _emit_unit(self, *, start: int, close: int, owner: str | None, item_start: i
                 name=f"{owner}{_OWNER_SEPARATOR}{function_name}" if owner else function_name,
                 start_line=self._tokens[item_start].line,
                 end_line=self._tokens[close].line,
-                normalized=[_normalize(item) for item in body],
+                normalized=_normalize_stream(body),
                 concrete=[item.text for item in body],
             )
         )

scripts/dupscore/_helpers/contracts/__init__.py

@@ -0,0 +1 @@
+"""Dupscore clones helpers."""