Repository navigation
Fix integer overflows in the heuristic univariate gcd - #112
Merged
Merged
Conversation
bound_valuation kept coefficient bit counts in an int, and the evaluation shifted coefficients by degree * n bits through an unsigned. Coefficients of 2^31 or more bits, or a degree * n of 2^32 or more, did not fit: the evaluation point or the shift came out wrong, so the gcd was wrong or reconstruct_polynomial wrote past its coefficient array. Count bits in size_t, and skip the heuristic (falling back to the subresultant gcd) when the shift does not fit in an unsigned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ahmed-irfan
self-requested a review
October 21, 2026 14:39
ahmed-irfan
approved these changes
Oct 22, 2026
Member
There was a problem hiding this comment.
Approve. The guard stops the 32-bit shift overflow, and the new test covers the case.
Suggestions, not blocking:
- Remove
bound_valuation@Base 1.3fromdebian/libpoly0.symbols. The function is nowstatic, sodpkg-gensymbolscan fail. - In
reconstruct_polynomial, replace the assert with a check that returns 0. Release builds remove the assert.
bound_valuation is now static, so libpoly.so.0 no longer exports it, and dpkg-gensymbols would report the listed symbol as missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
reconstruct_polynomial asserted that the value has at most max_size digits, and release builds compile the assert out, so a violation would write past the dense polynomial. It now frees its temporaries and returns 0, and the heuristic treats that as a failed attempt (the next power, or the subresultant fallback). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nlewycky
force-pushed
the
fix-gcd-heuristic-bound
branch
from
October 22, 2026 04:00
3dd5673 to
d390ef2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bound_valuation kept coefficient bit counts in an int, and the evaluation shifted coefficients by degree * n bits through an unsigned. Coefficients of 2^31 or more bits, or a degree * n of 2^32 or more, did not fit: the evaluation point or the shift came out wrong, so the gcd was wrong or reconstruct_polynomial wrote past its coefficient array. Count bits in size_t, and skip the heuristic (falling back to the subresultant gcd) when the shift does not fit in an unsigned.