You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Split out of #762, which #780 otherwise finishes, and the sibling of #781.
After #780 a 1.4 MB ods with a 297 MB content.xml peaks at 626 MB. 400 MB of
that is a floor nothing above the parser can move: 284 MB of inflated source
plus 116 MB of pugixml nodes. xml::parse(const abstract::File &) mallocs file.size() and hands it to load_buffer_inplace_own, so the buffer is the
dom's string storage — every name and value is a pointer into it — and it lives
as long as the document. It is not a copy that can be dropped.
pugixml offers no way out: it has no sax or pull api at all (load_string, load, load_file, load_buffer{,_inplace,_inplace_own}, and nothing else),
and load(std::istream &) is worse rather than better — load_stream_impl reads
the whole stream into its own buffer first, a chunk list it then concatenates
when the stream cannot seek, and parses only after. What we do now is already the
minimum for a dom.
The stream side is not the blocker. Zip entries inflate through an istream
already and the archive's read callback is re-entrant since #775.
The prize is not only the 400 MB
A streaming decode is what lets HtmlConfig::spreadsheet_limit reach the parser. A 100-row budget on a million-row sheet would stop inflating after a
fraction of a percent of the file — bounded memory and bounded time, instead
of building everything and then rendering 100 rows (#781).
What we could use
libexpat
in conan-center, expat/2.6.x
Chunk-fed push api (XML_Parse(p, buf, len, isFinal)), mature, small. Gets entities, attribute normalisation, cdata and encodings right. Push, not pull, so it wants either a state machine on top or rendering driven straight from the callbacks.
libxml2 xmlTextReader
in conan-center
A real pull api, but a large dependency and a large attack surface for a library that hand-rolls pdf, cfb, rtf and snappy.
yxml
no recipe, ~1 kloc, single file, MIT
Byte-at-a-time state machine, would be vendored. No dtd, which for odf is a feature.
rapidxml, tinyxml2
Dom only, same trap.
Three tiers, and they all keep the dom model
xml_node::set_name/set_value copy into the document's own pool
(strcpy_insitu), so a pugixml tree built programmatically owns its strings and
needs no source buffer. That means a windowed decode can still hand the registry, odf_style and the adapters a real pugi::xml_node — only its lifetime
changes.
A splitter, not a parser. Find the byte span of each table:table-row in
the inflating stream and hand it to load_buffer(..., parse_fragment).
pugixml still does the hard parts; the scanner only has to know where markup
ends — quotes, comments, cdata, pis — and depth-track nested tables. ~200
lines, and directly testable: the spans must match what a whole-file parse
says they are. Prefixes never resolve (we match "table:table-cell"
literally), so a fragment without its root is fine.
expat or yxml building pugixml subtrees. A real parser feeding append_child/set_name/append_attribute. Everything downstream is
untouched. More code than the splitter, none of its subtleties. The safer
choice if this has to be right for all of odf and not just spreadsheets.
Own parser, own model, no dom. Drops the buffer, the nodes and the
element registry — tens of MB for a spreadsheet — for the most work, and
read-only.
Two wrinkles to settle before writing any of it
Odf gives no sheet dimensions up front. The extent is known only after
walking every row, and feat(html): report the sheet a view cut, and budget its cells #757's sheet_rendered_extent/cut reporting needs it.
So either two streaming passes — re-inflating 297 MB is well under a second,
cheap against being killed — or emit first and report after. This shapes the
api, so decide it first.
Edit and save need the dom. A splice into a window that has been discarded
means nothing, so the streaming path is read-only and chosen at open, with the
dom path kept for everything else. Spreadsheets are already is_editable() == false, so nothing is lost today.
Related: #764 wants the same windowing for a different reason — render the
visible part and fetch the rest as the reader scrolls. If we build windows, build
them once for both.
Split out of #762, which #780 otherwise finishes, and the sibling of #781.
After #780 a 1.4 MB ods with a 297 MB
content.xmlpeaks at 626 MB. 400 MB ofthat is a floor nothing above the parser can move: 284 MB of inflated source
plus 116 MB of pugixml nodes.
xml::parse(const abstract::File &)mallocsfile.size()and hands it toload_buffer_inplace_own, so the buffer is thedom's string storage — every name and value is a pointer into it — and it lives
as long as the document. It is not a copy that can be dropped.
pugixml offers no way out: it has no sax or pull api at all (
load_string,load,load_file,load_buffer{,_inplace,_inplace_own}, and nothing else),and
load(std::istream &)is worse rather than better —load_stream_implreadsthe whole stream into its own buffer first, a chunk list it then concatenates
when the stream cannot seek, and parses only after. What we do now is already the
minimum for a dom.
The stream side is not the blocker. Zip entries inflate through an
istreamalready and the archive's read callback is re-entrant since #775.
The prize is not only the 400 MB
A streaming decode is what lets
HtmlConfig::spreadsheet_limitreach theparser. A 100-row budget on a million-row sheet would stop inflating after a
fraction of a percent of the file — bounded memory and bounded time, instead
of building everything and then rendering 100 rows (#781).
What we could use
expat/2.6.xXML_Parse(p, buf, len, isFinal)), mature, small. Gets entities, attribute normalisation, cdata and encodings right. Push, not pull, so it wants either a state machine on top or rendering driven straight from the callbacks.xmlTextReaderThree tiers, and they all keep the dom model
xml_node::set_name/set_valuecopy into the document's own pool(
strcpy_insitu), so a pugixml tree built programmatically owns its strings andneeds no source buffer. That means a windowed decode can still hand the registry,
odf_styleand the adapters a realpugi::xml_node— only its lifetimechanges.
table:table-rowinthe inflating stream and hand it to
load_buffer(..., parse_fragment).pugixml still does the hard parts; the scanner only has to know where markup
ends — quotes, comments, cdata, pis — and depth-track nested tables. ~200
lines, and directly testable: the spans must match what a whole-file parse
says they are. Prefixes never resolve (we match
"table:table-cell"literally), so a fragment without its root is fine.
append_child/set_name/append_attribute. Everything downstream isuntouched. More code than the splitter, none of its subtleties. The safer
choice if this has to be right for all of odf and not just spreadsheets.
element registry — tens of MB for a spreadsheet — for the most work, and
read-only.
Two wrinkles to settle before writing any of it
walking every row, and feat(html): report the sheet a view cut, and budget its cells #757's
sheet_rendered_extent/cut reporting needs it.So either two streaming passes — re-inflating 297 MB is well under a second,
cheap against being killed — or emit first and report after. This shapes the
api, so decide it first.
means nothing, so the streaming path is read-only and chosen at open, with the
dom path kept for everything else. Spreadsheets are already
is_editable() == false, so nothing is lost today.Related: #764 wants the same windowing for a different reason — render the
visible part and fetch the rest as the reader scrolls. If we build windows, build
them once for both.