docs(content-drive): spec for listing long-text projection trim (#37185)

[ihoffmann-dot]

Spec-Kit PR 1 of 2. Carries the spec alone. Needs a developer approval (not a merge) before /speckit-plan runs.

Resolves the spec phase of #37185.

Proposed Changes

• spec.md — 2 prioritized user stories, 5 functional requirements, 3 success criteria, 8 acceptance criteria, regression-risk/blast-radius analysis, and the dotCMS Legacy Considerations section. No open clarifications remaining.

Summary

Every row POST /api/v1/drive/search returns for a generic Content item carries the full value of that contentlet's long-text fields (WYSIWYG, TextArea, Story Block) even though the listing is a grid. On the benchmarked dataset a 40-row page measured 159 KB, ~73% of it a single long-text field, paid on serialization, on the wire, and in browser memory.

⚠️ Correction to the original issue framing

The issue's premise — "the grid doesn't render these fields" — is not quite right. The grid renders any field flagged "Show In List" on the content type, including long-text ones, with no field-type restriction. An unconditional removal would silently blank that column for any content type configured that way.

Decision: truncate to 150 characters rather than remove, applied uniformly regardless of listed status. This bounds payload in every case and preserves the Show In List column as a preview instead of blanking it. The frontend already visually clips these cells with CSS ellipsis — that's display-only clipping of the full string already in the response, so server-side truncation is what actually removes the bytes; no frontend change required.

The recommended implementation mirrors an existing precedent (DefaultTransformStrategy#addBinaries, same trigger point, same per-field-type iteration) doing a bounded-length map.put instead of map.remove.

Checklist

• Tests — land in PR 2, written and confirmed failing first (Constitution Principle V): integration test pinning the key set (and truncation length) of a generic-Content listing row from both the Content Drive and Site Browser paths, since none exists today
• The dead item.body read in the Postman collection's "Text Search - Alpha Filter" test needs removing or replacing with a real assertion
• Rollback-safety label — this spec's analysis reads the change as rollback-safe (removing/shrinking a field value is a forward-compatibility concern, not a rollback one), contrary to the original issue's stated assumption; dotCMS/core auto-applies rollback labels, so this shouldn't need hand-setting either way

Additional Info

Parent epic #36814. Originally investigated as item 3 of #37148 (umbrella investigation issue). dotContentMap is shared with Site Browser (POST /api/v1/browser) and reachable from the legacy DWR Site Browser and from OSGi via APILocator.getBrowserAPI() — broader blast radius than Content Drive alone, covered in the spec's Regression Risk section.

/speckit-adr-context consulted — no ADR identified as binding for this item; it changes which fields survive into the response, not where data is read from. No new ADR proposed.

🤖 Generated with Claude Code

This PR fixes: #37185

[claude]

Claude finished @ihoffmann-dot's task in 1m 51s —— View job

---

Reviewing the spec…

• Read spec.md
• Check Story Block interaction with the recommended approach (found a design gap)
• Verify the load-bearing file:line references (agent running)
• Post review

[fabrizzio-dotCMS]

Reviewed the spec against main (I read at 88af0bad55; the spec pins 87745ff315, nothing I checked moved in between).

Containing this to the transform layer is the right call — I agree with the scope, and with rejecting option (b). Duplicating content-type field introspection into a 2,700-line class that owns no other field responsibility would buy no containment, since both approaches modify the same dotContentMap and therefore hit Site Browser identically.

Most of what follows is inside that scope. The spec is unusually well-researched — I re-verified every file:line claim in it and they all reproduce, including both corrections it makes to the issue. My concerns are about the decision that was made last (OQ-1) not having been re-validated against the code afterwards.

---

One caveat on the framing

The spec is scoped to the transform layer on the premise that this is where the cost sits. That is true for the payload, but not for the load.

By the time DefaultTransformStrategy runs, findContentlets(inodes) (BrowserAPIImpl.java:1033,1072) has already materialized every contentlet in full from cache or DB, and copyContentlet.getMap() (DotContentletTransformerImpl.java:101) has already copied every field, bodies included, into the map. So the fix recovers Jackson serialization, wire bytes and browser retention, and leaves the DB/cache read and the map build untouched.

That is a defensible place to stop — a partially loaded contentlet would break the inode-keyed contentlet cache and the ADR-0018 read-your-writes contract, so a real column-level projection is a redesign rather than a fix. Note that OQ-3 (client-declared projection) does not change this either: it still filters after the load, it only moves the decision to the caller.

But the spec should state it, because it bounds what this item can deliver. "Reducing the other 27% of the payload" is listed as a non-goal; "the contentlet is still loaded and mapped in full" is not, and it is the first thing a reviewer of PR 2 will ask.

---

Blocking: the recommended implementation is silently undone for Story Block and JSON fields

StrategyResolverImpl#resolveStrategies (StrategyResolverImpl.java:126-143) adds DefaultTransformStrategy first, then the option-triggered strategies. StoryBlockViewStrategy#transform (StoryBlockViewStrategy.java:48-71) then re-reads the value from the source Contentlet — not from the map — and does map.put(field.variable(), jsonAsMap). JSONViewStrategy.java:44-58 does the same for JSON fields.

Both STORY_BLOCK_VIEW and JSON_VIEW are in defaultOptions, which is exactly the option set dotContentMap uses. So a truncation applied inside DefaultTransformStrategy is overwritten for every Story Block field.

FILTER_BINARIES is not a valid precedent for this, and for a specific reason: it works because BINARIES_VIEW is not in defaultOptions, so nothing runs after it to undo the map.remove. The spec's "same strategy, same trigger point, the mechanism is otherwise identical" does not hold for the two field types that are in defaultOptions.

Fixable — a strategy ordered last, or the cleanup(map) hook DotContentletTransformerImpl.java:104 already calls on the last strategy — but it is a different change from the one described, and the "roughly four files, ~15 lines" estimate no longer holds.

Related: there is no string to truncate for Story Block. After STORY_BLOCK_VIEW the map value is a LinkedHashMap, not a String. "WysiwygField.class, TextAreaField.class and StoryBlockField.class fields get their string value cut to 150 characters" is only true for the first two. Story Block needs its own decision (drop the key? keep the first N nodes? serialize-then-cut, which produces invalid JSON?) and it is not made anywhere in the document.

OQ-1's rationale does not hold in the grid it is protecting

Truncation was chosen over removal because removal would blank a "Show In List" column. But extra columns fall through #columnTypeForField (dot-content-drive-shell.component.ts:433-458) to TEXT, and TEXT renders as plain interpolation — @default { {{ value }} } (dot-folder-list-view.component.html:271). So today:

• a listed WYSIWYG cell shows escaped raw HTML. The first 150 characters of a real body are frequently markup (<p><span style="...">…) with no visible prose, so the truncation would preserve a column showing nothing readable;
• a listed Story Block cell renders [object Object].

The column the decision protects is already broken for two of the three field types in scope. That does not necessarily make truncation the wrong choice, but the argument that justifies it needs redoing with this in hand — option (i), rejected for "blanking a configured column silently", is blanking a column that currently shows markup soup or [object Object]. If the truncation is meant to yield a readable preview it has to extract text before cutting, which is a third distinct change.

The remove→truncate decision did not reach the spec body

It reached issue #37185 and this PR description, but not spec.md:

Location	Still says
Verification method → Integration	"long-text keys absent (AC-001)" — AC-001 says truncated
Verification method → Unit	"the option's removal logic … asserting exactly which keys survive"
AC-008	"removing a long-text key could plausibly blank a rendered column"
Backward compatibility / rollback	"Here N removes a field, so rolling back to N-1 restores it"

The rollback paragraph is the one that changes conclusion. Truncation is worse than removal for forward compatibility: a VTL template, an OSGi plugin, or a Site Browser consumer reading body gets a silently wrong, plausible-looking value instead of a missing one — a failure no UtilMethods.isSet() guard catches. That deserves saying explicitly, and OQ-6 should be re-answered on truncation semantics rather than inheriting the reasoning from the rejected option.

The PR description is ahead of the spec here — its checklist correctly says the test pins "the key set (and truncation length)", and hedges "removing/shrinking". Worth syncing the document to it.

Nothing gates latency

AC-001 only requires payload to drop by at least half, against a baseline where the body is 73% of it. Truncating to 150 characters should land near the full 73%, so as written an implementation that misses Story Block entirely (per the blocker above) still passes. p50/p95 appears only under "Manual / measurement" as something to record, not to meet. Given the ~65 ms the issue attributes to this item is partly the map build — which this fix does not touch — a latency AC, or an explicit statement that latency is not gated, would help.

Smaller items

• Test coverage is understated. "There is no ContentDriveResourceTest and no ContentDriveHelperTest" is true by name, but six Drive test classes exist: ContentDriveFieldFilterTest, ContentDriveHelperContentletAPIComparisonTest, ContentDriveKeywordSearchTest, ContentDriveWorkflowFilterTest, ContentDriveWorkflowArchiveStepTest, ContentDriveFieldFilterResolverTest. I grepped all six and the substantive claim survives — none asserts on a row's key set or on body — but naming them would turn "will break no existing test" from an assertion into a verified one.
• ADR-0018 is dismissed on the wrong grounds. "Cited for item 2, not this item" — but ContentDriveFieldFilterTest.java:66 asserts the ADR-0018 routing contract in the Drive path, and BrowserAPIImpl.java:1995,2192,2531 implement it there. The dismissal is probably still right, since truncation is post-hydration and cannot change where data is read from; it should be stated on that basis.
• The new TransformOptions constant has no proposed name — every AC refers to "the new option".
• 150 is a bare magic number. No derivation (grid column width? p50 title length?). Given the blast radius reaches Site Browser, VTL and OSGi, either tie it to OQ-7's Config property or justify it.
• No truncation marker. Nothing tells a consumer the value was cut. Trailing …, a sibling <field>Truncated: true, or nothing — but decided in the spec, not in PR 2.
• Character vs code point. substring(0, 150) on a boundary that splits a surrogate pair emits a lone surrogate into the JSON response. Needs an AC or an explicit non-concern.

PR description

Proposed Changes describes a different template than the one used. spec.md follows spec-issue-template.md and has 0 user stories, 0 FR- requirements, 0 SC- success criteria, and no "dotCMS Legacy Considerations" section — the 8 acceptance criteria and the regression-risk analysis are the accurate parts. More importantly it says "No open clarifications remaining" while the spec carries 7 open questions, 6 unresolved, under a heading that reads "must be answered before /speckit-plan finalizes". Someone approving on the description alone would be approving past the gate.

---

Suggested sequencing

Answer OQ-2 (does Site Browser change too) and OQ-4 (which field types count as long text) first — they bound the change — then re-run OQ-1 with the rendering evidence above and add a Story Block sub-decision. If the WYSIWYG column really is unreadable today, option (iv) / OQ-3 becomes more attractive than the spec currently rates it, since the frontend already knows its showInListFields and can ask for exactly what it will render.

Also worth confirming before building: #37148 is closed, and its guidance was to land item 1 first and then re-evaluate whether items 2–4 are still needed. Item 1's spec (#37230) is still open, so that re-evaluation has not happened yet.

Credit where it is due — the Postman correction is right (allItemsMatch appears exactly once in the collection, line 995, its own declaration; the item.body read is dead), the "webAssetOptions() removes nothing, it merely decorates less" correction is right, and "no item.body read anywhere in core-web on a drive result" is right (the only hit is dot-template-builder.component.html:23, an unrelated template body). Every other code reference checks out too.