docs(content-drive): spec for field-filter chunk multiplier (#37184)

[ihoffmann-dot]

Spec-Kit PR 1 of 2. Carries the spec alone. Needs a developer approval (not a merge) before /speckit-plan runs.

Resolves the spec phase of #37184.

Proposed Changes

• spec.md — 2 prioritized user stories, 9 functional requirements (2 resolved decisions recorded inline, no open clarifications remaining), 4 success criteria, edge cases, dependencies/coordination, and the dotCMS Legacy Considerations section.

Summary

When a Content Drive field filter routes entirely to the search index (correct per ADR-0018 — searchable-flagged fields are index-resolved by design, this is not a mis-routing bug), the DB-first hybrid chunk loop still re-scans the database candidate set in fixed-size chunks until enough index-narrowed matches survive. On a ~21,000-item folder this took 4 repeats and made the response ~6x slower than an equivalent search. buildPureESQuery already builds nearly everything needed for a single-pass, index-only resolution for this specific case — it's missing only a folder-scoping clause. Routing to that path when zero DB-required criteria (Tag, Relationship), no workflow filter, and no free-text term are present should collapse the repeated scans to one, with no change to any other filter combination's behavior or freshness guarantee.

Decisions recorded (no product sign-off needed, resolved as technical calls)

• No dedicated feature-flag kill switch — the freshness trade-off this narrowly extends is the same one ADR-0018 already accepts by default for free-text search, so no flag is needed for that case either. The existing general search-strategy config remains the fallback.
• The narrow read-your-writes trade-off for this one case ships as default behavior — not escalated to product, since it extends an already-accepted architectural trade-off rather than introducing a new one.

Coordination note

Shares the underlying candidate-scan query with #37183 (spike on that query's plan instability). Whichever direction #37183 lands on must preserve this fix's single-scan-per-request assumption.

Checklist

• Tests — land in PR 2, written and confirmed failing first (Constitution Principle V)
• Security Implications Contemplated — permission filtering is untouched; this only changes which code path resolves an already-permitted query

Additional Info

Parent epic #36814. Originally investigated as item 2 of #37148 (umbrella investigation issue).

/speckit-adr-context consulted ADR-0018 (database-first search for Content Drive) — this fix operates strictly inside that contract: it changes how many times the existing hybrid strategy's DB step runs for one already-correctly-routed case, not which source of truth is authoritative for any criterion. No new ADR proposed.

🤖 Generated with Claude Code

This PR fixes: #37184

[claude]

Claude finished @ihoffmann-dot's task in 42s —— View job

---

Spec Review — Content Drive Field-Filter Chunk Multiplier (#37184)

This PR adds a single spec document (no code). Reviewed for internal consistency, completeness, and alignment with the stated architectural contract (ADR-0018). No executable code, so there are no bugs/security/convention findings to make.

New Issues

• 🟡 Medium: specs/37184-content-drive-field-filter-chunk-multiplier/spec.md:3 — Feature Branch is recorded as 37148-field-filter-chunk-multiplier, but the actual PR branch is issue-37184-content-drive-field-filter-chunk-multiplier. The stated branch also references the umbrella issue number (37148) rather than this spec's issue (37184). Worth correcting so /speckit-plan//speckit-tasks operate against the right branch reference.

Observations (non-blocking, for author consideration)

• FR-005 vs. FR-002 enumeration (:99 / :108): FR-002 gates the single-pass path on "no workflow filter and no free-text or file-name term," while FR-005 lists the fallback triggers as "Tag/Relationship, OR workflow, OR free-text/file-name." These are consistent, but FR-002 phrases the index-resolvability condition (Tag/Relationship) separately from the workflow/free-text/file-name conditions. Consider making FR-002 state the full negative condition explicitly (no DB-required criterion and no workflow and no free-text/file-name term) so the two requirements read as exact complements — this is the crux of the routing decision and will drive the test matrix in PR 2.
• SC-002 measurability (:146): "within 20% of the closest equivalent content-search operation" is a good target, but the spec doesn't define what "equivalent content-search operation" means concretely (same folder scope? same field predicate translated to a search query?). Pinning that comparison down will make the success criterion objectively testable rather than requiring judgment at test time.
• Relationship-filter test coverage (:150, :71): US2/SC-003 correctly call out combining a field filter with a Relationship filter, but the Independent Test for US1 (:31) and FR list emphasize Tag much more than Relationship. Since Relationship is the other DB-required criterion, PR 2's tests should give it equal weight — flagging so it isn't dropped when tasks are derived.
• Zero-match edge case (:81): Good that this is called out. The acceptance scenarios (:38) only assert the positive-match case explicitly at the scenario level; a zero-match acceptance scenario under US1 would tighten the "single pass even when nothing matches" guarantee that the edge case describes.

Alignment check

• The spec stays strictly inside ADR-0018's contract (DB-first remains default; index-only never becomes default) — consistent throughout (:112, :163-170). ✅
• Permission filtering is explicitly preserved on every path (FR-004, :105; edge case :86). ✅
• The two recorded decisions (no kill switch FR-009; read-your-writes trade-off in Assumptions) are documented with reasoning and scoped narrowly to the zero-DB-criteria case. ✅
• Coordination with Spike: Content Drive folder-listing query has planner-driven latency on very large folders #37183 (shared candidate-scan query) is captured with the single-scan-per-request invariant the other work must preserve (:172-180). ✅

Overall the spec is thorough and self-consistent. The only blocking-ish item is the branch-name field; the rest are refinements to make PR 2's TDD task derivation crisper.

_{· branch issue-37184-content-drive-field-filter-chunk-multiplier}

[fabrizzio-dotCMS]

Reviewed against main at 88af0bad55, and against ADR-0018 in platform-adrs. (Paraphrasing the ADR rather than quoting it — this repo is public and that one isn't. Section names given so an internal reader can check me.)

Structure is complete, User Story 2 and FR-005/FR-008 protect the DB-routed combinations properly, and the Dependencies & Coordination section correctly identifies the shared query with item 1. The Assumption about which field types route where is accurate — I verified ContentDriveFieldFilterResolver#routingBucketFor (:216-219) sends Tag and Relationship to DB and Text/Select/Boolean/Date/Category to INDEX.

My concern is that the spec never names which implementation it means, and the two candidates have opposite compliance profiles against ADR-0018. Its own success criterion pushes toward the non-compliant one.

---

The two implementations FR-002 allows

FR-002 requires the DB scan to run "at most once per request, regardless of folder size or how sparse the matches are." There are two ways to get there:

(i) Stay hybrid; fetch all candidates in one DB query instead of chunking. ADR-0018-compliant — the DB still drives the candidate set, ordering, pagination and permissions. But for a 20k folder you materialize 20k inodes and then have to ES-filter them, and the ES boolean-clause ceiling is 1024 (calculateMaxInodesPerESQuery, BrowserAPIImpl.java:797-823), so processMultipleESQueries subdivides into roughly two dozen ES round trips. You trade 4 DB scans for 1 large scan plus ~24 index queries. BROWSER_DB_MAX_SCAN_ROWS (default 50,000, :735) caps the scan.

(ii) Route to PURE_ES. This is what #37148 item 2 actually proposes — "the existing PURE_ES heuristic could serve the query directly instead." See below.

Everything in Legacy Considerations reads as if (i) is meant. Everything in the parent issue and in SC-002 points at (ii). The spec never picks, and almost every other question depends on which it is.

On the PURE_ES path, FR-003 / FR-004 / FR-007 are new work, not preservation

The spec states these three as behaviour to keep. On that path they don't exist today:

Spec requirement	PURE_ES as implemented	ADR-0018 routing table
FR-003 "MUST still be scoped to the folder … never return content from outside the requested folder"	buildPureESQuery (:606-692) emits conhost only — the site. There is no conFolder clause anywhere in it. The inline comment says "Host/folder filter"; the code filters host. The method doc says "without inode filtering".	assigns parent folder / path to DB, as structural and authoritative
FR-004 permission filter "sourced from the database, after candidates are retrieved, for every request"	permissions come from contentletAPI.search(esQuery, …, user, respectFrontEndRoles) (:571-578) — index-side, not a DB filterCollection after candidate selection	assigns the READ-permission filter to DB, applied after candidate selection
FR-007 "same items, same order, same pagination behavior"	ordering via browserQuery.sortBy, pagination via startRow + indexCount (:581-585) — not the DB cursor (contentCursor / generateNextContentCursor) the hybrid path uses	assigns sorting and pagination to DB, on the grounds that only the DB sees every row

The ADR is also explicit that PURE_ES is available behind configuration but must not become the default, because it surrenders read-your-writes for all criteria — and that the DB-resolved criteria must never be silently re-routed to the index for speed, the remedy for slowness being a DB index or query change instead. Making PURE_ES the automatic path for a whole request class, shipped default-on with no flag (FR-009), is close to the scenario that clause exists to prevent. Whether it counts as "becoming the default" is a question for the ADR's decision-makers, not one the spec can settle by assertion.

Separately, PURE_ES throws today when field criteria are present:

// :493-501
// PURE_ES bypasses the DB select entirely and doesn't build per-field clauses,
// so it can neither apply DB-routed (Tag) predicates nor index-routed field clauses.
if (!browserQuery.getFieldCriteria().isEmpty()) {
    throw new DotRuntimeException("Content Drive field filters (userSearchable) "
            + "are not supported under the PURE_ES heuristic; use HYBRID_SINGLE_CHUNKED_QUERY_ES.");
}

So the closing Assumption — "the existing per-field index query logic … is assumed to be reusable as-is for the single-pass case; no new field-to-index translation logic is expected to be needed" — is half right. buildFieldCriteriaESClauses is reusable. The path that would host it refuses field criteria by design and has no folder scoping. That is not a no-op.

The freshness trade-off being justified already exists

Assumptions spends its longest resolution defending "accepting a brief, index-lag-bounded delay before a just-written item appears in a field-filter-only search" as a deliberate, scoped, new trade-off.

But that is already today's behaviour. In the hybrid loop, getChunkFiltered(browserQuery, true, …) narrows each DB chunk with an ES query that already carries the index-routed field clauses (buildBaseESQuery → buildFieldCriteriaESClauses, :1238, :1262-1275). A just-written, not-yet-indexed item is already excluded from a field-filtered result. ADR-0018's "defer to the index" section mandates exactly that for searchable fields.

So there is no new concession on that axis and no product decision needed for it — good news for the change. The cost is that the spec spends its whole risk budget here and consequently never examines the three things that would change under (ii): folder scoping, permission sourcing, and ordering/pagination.

SC-002 is unreachable under (i) until item 1 lands

SC-002 asks for response time within 20% of the closest equivalent content-search operation. Per #37148 that baseline is 117 ms against Drive's 705 ms, so the target is ~140 ms.

Under (i), a single candidate scan of a 20k folder costs 450–461 ms today. SC-002 is arithmetically unreachable. It becomes reachable only once item 1's folder-first CTE brings that scan to 101–108 ms.

Dependencies & Coordination does flag item 1, but frames it as "lowers this case's exposure to item 1's problem" rather than "SC-002 is unachievable without it". As written, an implementer chasing SC-002 with item 1 unlanded is pushed straight to PURE_ES. Item 1's spec (#37230) is still open.

---

Smaller items

FR-009's escape hatch doesn't work for the case it guards. FR-009 declines a kill switch because BROWSE_API_HEURISTIC_TYPE "remains the escape hatch of last resort if the single-pass path needs to be disabled entirely." The only two legal values are HYBRID_SINGLE_CHUNKED_QUERY_ES and PURE_ES (:460-465), defaulting to hybrid (:697). If the single-pass lands inside the hybrid branch, switching to PURE_ES doesn't disable it — it throws on any request carrying field criteria (:496-501). There is no value that yields "hybrid, previous chunked behaviour."

"0 open clarifications" overstates where the decisions stand. Two were resolved inline on 2026-08-24, both classified as technical calls needing no product sign-off. Even granting that the freshness relaxation is narrower than claimed (above), the routing question — does a whole request class stop being DB-first — is squarely an ADR question, and ADR-0018 has named decision-makers. That belongs as an open clarification addressed to them, not a resolved one.

No test type is named. SC-001 counts candidate-scan iterations and SC-003/SC-004 assert "100% of tested combinations", but nothing says what runs them or whether any of it is reachable without the 20k-folder dataset. Worth noting ContentDriveFieldFilterTest already exists in dotcms-integration and its own header says it asserts the ADR-0018 routing contract (:66) — that is the natural home, and Principle V wants the tests written and failing first.

Stale javadoc worth fixing while nearby. FieldSearchCriteria.RoutingBucket.DB is documented as "Resolved against the database (Tag)" (:42-43), but the resolver routes Tag and Relationship to DB. The spec's Assumption gets this right; the enum comment doesn't.

---

Verified, no action needed

• Tag / Relationship → DB, Text / Select / Boolean / Date / Category → INDEX (ContentDriveFieldFilterResolver.java:210-229) ✅
• buildFieldCriteriaESClauses skips non-INDEX criteria (:1268); appendFieldCriteriaDBPredicates skips non-DB (:2210) ✅
• Chunk size 900 (BROWSER_CONTENT_CHUNK_SIZE, :547); ~3,337 candidates over 900 ≈ 4 chunks — consistent with SC-001's "four to at most one" ✅
• BROWSE_API_HEURISTIC_TYPE exists, defaults to HYBRID_SINGLE_CHUNKED_QUERY_ES (:697) ✅
• ADR-0018 correctly identified as binding, and the spec is right that whether structural filtering should move to the index is already settled by it and out of scope here ✅
• All four mandatory sections of the dotCMS override template present, plus Assumptions ✅

Adjacent, not in scope while PURE_ES stays off: buildPureESQuery's text clause still uses the broad leading-wildcard catchall:*<kw>* form (:658-663) that the hybrid path deliberately replaced under #36688 (see the comment at :1191-1198). It's evidence that PURE_ES is unmaintained, which matters if the fix intends to route real traffic through it.

---

What I'd ask for

1. Name the implementation — (i) a single large DB scan inside the hybrid strategy, or (ii) PURE_ES / a new index-only path. The spec is currently readable both ways and most of the above depends on it.
2. If (ii): reopen as an ADR-0018 question for its decision-makers, and rewrite FR-003/FR-004/FR-007 as requirements to build rather than behaviour to preserve.
3. If (i): state that SC-002 depends on item 1 landing first, and add an acceptance criterion on ES round-trip count — that's what (i) trades the DB scans for.
4. Drop or rescope the freshness trade-off in Assumptions; it's already shipped behaviour and already mandated by the ADR.
5. Fix FR-009 — either identify a real disable path or say plainly that there isn't one.

Sequencing note, same as the sibling specs: #37148 is closed and its guidance was to land item 1 first and then re-evaluate whether items 2–4 are still needed. Here the dependency isn't only sequencing — per SC-002 above it determines whether this item's own success criterion is achievable without stepping outside the ADR.