Truncation and rounding of numbers: Difference between revisions
mNo edit summary |
mNo edit summary |
||
Line 1: | Line 1: | ||
The [[Routine | The [[Routine Operations#trunc|trunc routine operation]] uses the C floor(x) function, this returns the highest integer that is less than or equal to x. So trunc(1.5) is 1, trunc(0.5) is 0 and trunc(-0.5) is -1. | ||
Very small -ve values will trunc() to the value below it, just as 1.999999999999 would trunc to 1 but display as 2. The internal precision is about 15 digits. | Very small -ve values will trunc() to the value below it, just as 1.999999999999 would trunc to 1 but display as 2. The internal precision is about 15 digits. |
Latest revision as of 04:00, 12 January 2008
The trunc routine operation uses the C floor(x) function, this returns the highest integer that is less than or equal to x. So trunc(1.5) is 1, trunc(0.5) is 0 and trunc(-0.5) is -1.
Very small -ve values will trunc() to the value below it, just as 1.999999999999 would trunc to 1 but display as 2. The internal precision is about 15 digits.
It all comes down to the application you have for the trunc-ed number, normally round is much safer and should be used to convert a number which *should* be integer to an actual integer (with the precision limits of a C "double" that is).
For internal double to int conversion, I actually use a delta of 1.0e-6. Values are display-rouded to 4 decimal places. And values are compared for equality using a dynamic precision based on the highest value, of 10 digits,
idkbase note 68