Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

core: improve float parsing #6

Open
cdragan opened this issue Sep 16, 2016 · 0 comments
Open

core: improve float parsing #6

cdragan opened this issue Sep 16, 2016 · 0 comments
Assignees
@cdragan
Labels
bug

Comments

@cdragan
Copy link
Member

cdragan commented Sep 16, 2016

The current routine used for extracting doubles from strings has a minor roundoff error when the number of digits reaches the limits of double precision.
@cdragan cdragan added the bug label Sep 16, 2016
@cdragan cdragan self-assigned this Sep 16, 2016
cdragan added a commit that referenced this issue Nov 25, 2017
@cdragan
core: improve float parsing
5c8b66d 
issue #6

* Fix bugs in parsing some floats
* Process dot and decimal exponent in the same manner
* Add option in parsing test to use strtod as a reference
* Add option in parsing test to generate random numbers

About 4% of randomly generated floats still have an error on the last
bit.  This is due to loss of precision.  In order to improve precision
further, multi-precision integers would have to be used.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cdragan cdragan

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cdragan