gh-83895: Accept input larger than 2 GiB in the C implementation of ElementTree - #156746
Conversation
…n of ElementTree XMLParser.feed() and ElementTree.parse() raised OverflowError, because Expat takes the length as an int. The data is now fed to Expat in chunks of 1 MiB, as xml.parsers.expat does since bpo-17089, so the pure Python implementation already accepted such input.
# Conflicts: # Lib/test/test_xml_etree.py
|
Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13. |
|
Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14. |
|
Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15. |
|
Sorry, @serhiy-storchaka, I could not cleanly backport this to |
|
Sorry, @serhiy-storchaka, I could not cleanly backport this to |
|
Sorry, @serhiy-storchaka, I could not cleanly backport this to |
|
GH-156771 is a backport of this pull request to the 3.15 branch. |
|
GH-156772 is a backport of this pull request to the 3.14 branch. |
|
GH-156773 is a backport of this pull request to the 3.13 branch. |
|
…on of ElementTree (GH-156746) (GH-156772) XMLParser.feed() and ElementTree.parse() raised OverflowError, because Expat takes the length as an int. The data is now fed to Expat in chunks of 1 MiB, as xml.parsers.expat does since bpo-17089, so the pure Python implementation already accepted such input. (cherry picked from commit 9259e8b)
|
|
|
|
…on of ElementTree (GH-156746) (GH-156773) XMLParser.feed() and ElementTree.parse() raised OverflowError, because Expat takes the length as an int. The data is now fed to Expat in chunks of 1 MiB, as xml.parsers.expat does since bpo-17089, so the pure Python implementation already accepted such input. (cherry picked from commit 9259e8b)
XMLParser.feed()andElementTree.parse()raisedOverflowError: size does not fit in an intfor input larger than 2 GiB, because Expat takes the length as anint. The data is now fed to Expat in chunks of 1 MiB, exactly asxml.parsers.expathas done since bpo-17089, so the pure Python implementation already accepted such input:Chunking is not slower for ordinary documents. Parsing 1 GiB with a million elements takes 2.37 s with chunks and 2.60 s with a single
XML_Parse()call, presumably because 1 MiB chunks stay in cache.test_length_overflow, which asserted the old error, is replaced by a bigmem test of the new behaviour, plus two cheap tests which run always, because the chunking loop is entered for any input larger than 1 MiB.