Skip to content

Stream a spreadsheet's content.xml instead of holding all of it: parser options #782

Description

@andiwand

Split out of #762, which #780 otherwise finishes, and the sibling of #781.

After #780 a 1.4 MB ods with a 297 MB content.xml peaks at 626 MB. 400 MB of
that is a floor nothing above the parser can move
: 284 MB of inflated source
plus 116 MB of pugixml nodes. xml::parse(const abstract::File &) mallocs
file.size() and hands it to load_buffer_inplace_own, so the buffer is the
dom's string storage — every name and value is a pointer into it — and it lives
as long as the document. It is not a copy that can be dropped.

pugixml offers no way out: it has no sax or pull api at all (load_string,
load, load_file, load_buffer{,_inplace,_inplace_own}, and nothing else),
and load(std::istream &) is worse rather than better — load_stream_impl reads
the whole stream into its own buffer first, a chunk list it then concatenates
when the stream cannot seek, and parses only after. What we do now is already the
minimum for a dom.

The stream side is not the blocker. Zip entries inflate through an istream
already and the archive's read callback is re-entrant since #775.

The prize is not only the 400 MB

A streaming decode is what lets HtmlConfig::spreadsheet_limit reach the
parser. A 100-row budget on a million-row sheet would stop inflating after a
fraction of a percent of the file — bounded memory and bounded time, instead
of building everything and then rendering 100 rows (#781).

What we could use

libexpat in conan-center, expat/2.6.x Chunk-fed push api (XML_Parse(p, buf, len, isFinal)), mature, small. Gets entities, attribute normalisation, cdata and encodings right. Push, not pull, so it wants either a state machine on top or rendering driven straight from the callbacks.
libxml2 xmlTextReader in conan-center A real pull api, but a large dependency and a large attack surface for a library that hand-rolls pdf, cfb, rtf and snappy.
yxml no recipe, ~1 kloc, single file, MIT Byte-at-a-time state machine, would be vendored. No dtd, which for odf is a feature.
rapidxml, tinyxml2 Dom only, same trap.

Three tiers, and they all keep the dom model

xml_node::set_name/set_value copy into the document's own pool
(strcpy_insitu), so a pugixml tree built programmatically owns its strings and
needs no source buffer. That means a windowed decode can still hand the registry,
odf_style and the adapters a real pugi::xml_node — only its lifetime
changes.

  1. A splitter, not a parser. Find the byte span of each table:table-row in
    the inflating stream and hand it to load_buffer(..., parse_fragment).
    pugixml still does the hard parts; the scanner only has to know where markup
    ends — quotes, comments, cdata, pis — and depth-track nested tables. ~200
    lines, and directly testable: the spans must match what a whole-file parse
    says they are. Prefixes never resolve (we match "table:table-cell"
    literally), so a fragment without its root is fine.
  2. expat or yxml building pugixml subtrees. A real parser feeding
    append_child/set_name/append_attribute. Everything downstream is
    untouched. More code than the splitter, none of its subtleties. The safer
    choice if this has to be right for all of odf and not just spreadsheets.
  3. Own parser, own model, no dom. Drops the buffer, the nodes and the
    element registry — tens of MB for a spreadsheet — for the most work, and
    read-only.

Two wrinkles to settle before writing any of it

  • Odf gives no sheet dimensions up front. The extent is known only after
    walking every row, and feat(html): report the sheet a view cut, and budget its cells #757's sheet_rendered_extent/cut reporting needs it.
    So either two streaming passes — re-inflating 297 MB is well under a second,
    cheap against being killed — or emit first and report after. This shapes the
    api, so decide it first.
  • Edit and save need the dom. A splice into a window that has been discarded
    means nothing, so the streaming path is read-only and chosen at open, with the
    dom path kept for everything else. Spreadsheets are already
    is_editable() == false, so nothing is lost today.

Related: #764 wants the same windowing for a different reason — render the
visible part and fetch the rest as the reader scrolls. If we build windows, build
them once for both.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions