Skip to content

Fix Newton inverse subtraction precision - #547

Open
OskarEichler wants to merge 1 commit into
ruby:masterfrom
OskarEichler:codex/fix-newton-inverse-precision
Open

Fix Newton inverse subtraction precision#547
OskarEichler wants to merge 1 commit into
ruby:masterfrom
OskarEichler:codex/fix-newton-inverse-precision

Conversation

@OskarEichler

Copy link
Copy Markdown

Summary:

  • remove an accidental nested SIZET2NUM conversion in Newton reciprocal iteration
  • pass the intended n / 2 precision directly to BigDecimal_sub2

Why:
The current expression converts n / 2 to a Ruby Integer, then feeds that VALUE back to SIZET2NUM as though it were a native size_t. On tagged-integer builds this inflates the requested precision substantially and performs avoidable work.

Verification:

  • full current suite: 266 tests, 13,616 assertions, 0 failures/errors/omissions
  • RBS suite: 146 tests, 1,805 assertions, 0 failures/errors
  • five controlled 50,000-digit reciprocal runs, 50 iterations each: baseline average 0.25346s; fixed average 0.20403s (about 19.5% faster)
  • numerical accuracy invariant passed in every run

Compatibility:
The correction restores the precision value already expressed by the surrounding n / 2 calculation; results remain within the existing accuracy invariant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant