Mercurial > octave
changeset 21747:61f3575250e4
rats: Fix round-off corner case for 32-bit systems (bug #47964)
* pr-output.cc (rational_approx): Invert convergence test to prevent
aborting early due to round-off error in 32-bit environments. Use std::abs
instead of fabs.
| author | Mike Miller <mtmiller@octave.org> |
|---|---|
| date | Thu, 19 May 2016 15:47:24 -0700 |
| parents | ac8b8bdae98f |
| children | 176536b15d68 |
| files | libinterp/corefcn/pr-output.cc |
| diffstat | 1 files changed, 3 insertions(+), 3 deletions(-) [+] |
line wrap: on
line diff
--- a/libinterp/corefcn/pr-output.cc Thu May 19 14:40:35 2016 -0400 +++ b/libinterp/corefcn/pr-output.cc Thu May 19 15:47:24 2016 -0700 @@ -389,7 +389,7 @@ double nextd = d; // Have we converged to 1/intmax ? - if (fabs (frac) < 1 / static_cast<double> (std::numeric_limits<int>::max ())) + if (std::abs (flip) > static_cast<double> (std::numeric_limits<int>::max ())) { lastn = n; lastd = d; @@ -417,8 +417,8 @@ else if (buf.str ().length () > static_cast<unsigned int>(len)) break; - if (fabs (n) > std::numeric_limits<int>::max () - || fabs (d) > std::numeric_limits<int>::max ()) + if (std::abs (n) > std::numeric_limits<int>::max () + || std::abs (d) > std::numeric_limits<int>::max ()) break; s = buf.str ();