Truncation and rounding of numbers

From Planimate Knowledge Base
Revision as of 04:00, 12 January 2008 by Rick (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

The trunc routine operation uses the C floor(x) function, this returns the highest integer that is less than or equal to x. So trunc(1.5) is 1, trunc(0.5) is 0 and trunc(-0.5) is -1.

Very small -ve values will trunc() to the value below it, just as 1.999999999999 would trunc to 1 but display as 2. The internal precision is about 15 digits.

It all comes down to the application you have for the trunc-ed number, normally round is much safer and should be used to convert a number which *should* be integer to an actual integer (with the precision limits of a C "double" that is).

For internal double to int conversion, I actually use a delta of 1.0e-6. Values are display-rouded to 4 decimal places. And values are compared for equality using a dynamic precision based on the highest value, of 10 digits,

idkbase note 68