Truncation and rounding of numbers: Difference between revisions
(Importing text file) |
mNo edit summary |
||
(2 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
The [[Routine Operations#trunc|trunc routine operation]] uses the C floor(x) function, this returns the highest integer that is less than or equal to x. So trunc(1.5) is 1, trunc(0.5) is 0 and trunc(-0.5) is -1. | |||
Very small -ve values will trunc() to the value below it, just as 1.999999999999 would trunc to 1 but display as 2. The internal precision is about 15 digits. | |||
It all comes down to the application you have for the trunc-ed number, normally [[Routine Operations#round|round]] is much safer and should be used to convert a number which *should* be integer to an actual integer (with the precision limits of a C "double" that is). | |||
For internal double to int conversion, I actually use a delta of 1.0e-6. Values are display-rouded to 4 decimal places. And values are compared for equality using a dynamic precision based on the highest value, of 10 digits, | |||
---- | |||
<font size="2">idkbase note 68</font> | |||
[[Category:Routine]] | |||
[[Category:Routine | |||
[[Category:Attribute]] | [[Category:Attribute]] | ||
[[Category:Runtime Engine]] | [[Category:Runtime Engine]] | ||
Latest revision as of 05:00, 12 January 2008
The trunc routine operation uses the C floor(x) function, this returns the highest integer that is less than or equal to x. So trunc(1.5) is 1, trunc(0.5) is 0 and trunc(-0.5) is -1.
Very small -ve values will trunc() to the value below it, just as 1.999999999999 would trunc to 1 but display as 2. The internal precision is about 15 digits.
It all comes down to the application you have for the trunc-ed number, normally round is much safer and should be used to convert a number which *should* be integer to an actual integer (with the precision limits of a C "double" that is).
For internal double to int conversion, I actually use a delta of 1.0e-6. Values are display-rouded to 4 decimal places. And values are compared for equality using a dynamic precision based on the highest value, of 10 digits,
idkbase note 68