Truncation and rounding of numbers: Difference between revisions

From Planimate Knowledge Base
Jump to navigation Jump to search
(Importing text file)
 
mNo edit summary
 
(2 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<font face="MS Sans Serif">The Truncating operation uses the floor(x) function, this returns the highest integer that is less than or equal to x. So trunc(1.5) is 1, trunc(0.5) is 0 and trunc(-0.5) is -1.
The [[Routine Operations#trunc|trunc routine operation]] uses the C floor(x) function, this returns the highest integer that is less than or equal to x. So trunc(1.5) is 1, trunc(0.5) is 0 and trunc(-0.5) is -1.


Very small -ve values will trunc() to the value below it, just as 1.999999999999 would trunc to 1 but display as 2. The internal precision is about 15 digits.


Very small -ve values will trunc() to the value below it, just as 1.999999999999 would trunc to 1 but display as 2. The internal precision is about 15 digits.
It all comes down to the application you have for the trunc-ed number, normally [[Routine Operations#round|round]] is much safer and should be used to convert a number which *should* be integer to an actual integer (with the precision limits of a C "double" that is).
 
For internal double to int conversion, I actually use a delta of 1.0e-6. Values are display-rouded to 4 decimal places. And values are compared for equality using a dynamic precision based on the highest value, of 10 digits,
 
----




It all comes down to the application you have for the trunc-ed number, normally "round" is much safer and should be used to convert a number which *should* be integer to an actual integer (with the precision limits of a C "double" that is).




For internal double to int conversion, I actually use a delta of 1.0e-6. Values are display-rouded to 4 decimal places. And values are compared for equality using a dynamic precision based on the highest value, of 10 digits,


<font size="2">idkbase note 68</font>


</font>
[[Category:Routine]]
----
[[Category:Routine Line]]
[[Category:Attribute]]
[[Category:Attribute]]
[[Category:Decision Making]]
[[Category:Runtime Engine]]
[[Category:Runtime Engine]]
[[Category:View/Attribute]]
<font size="2">idkbase note 68</font>

Latest revision as of 05:00, 12 January 2008

The trunc routine operation uses the C floor(x) function, this returns the highest integer that is less than or equal to x. So trunc(1.5) is 1, trunc(0.5) is 0 and trunc(-0.5) is -1.

Very small -ve values will trunc() to the value below it, just as 1.999999999999 would trunc to 1 but display as 2. The internal precision is about 15 digits.

It all comes down to the application you have for the trunc-ed number, normally round is much safer and should be used to convert a number which *should* be integer to an actual integer (with the precision limits of a C "double" that is).

For internal double to int conversion, I actually use a delta of 1.0e-6. Values are display-rouded to 4 decimal places. And values are compared for equality using a dynamic precision based on the highest value, of 10 digits,




idkbase note 68