Freddie Akeroyd: New page: ==Why does A/B give the wrong answer?== If A and B are both integers, then integer division is performed; if either is a floating point number then the integer value is first converted int...

Freddie Akeroyd — Wed, 27 May 2009 17:29:01 GMT

New page: ==Why does A/B give the wrong answer?== If A and B are both integers, then integer division is performed; if either is a floating point number then the integer value is first converted int...

New page

==Why does A/B give the wrong answer?==
If A and B are both integers, then integer division is performed; if either is a floating point number then the integer value is first converted into a floating point number before division is performed. Thus:
<pre>
1/2 == 0 # 1 and 2 are of Integer type
</pre>
but
<pre>
1/2.0 == 0.5 # 2.0 is a floating point number, so integer 1 is converted to 1.0 before division is done
</pre>
If you are unsure what data type two variables may be, then you can force real division using either of these statements
<pre>
RES = AS_REAL(A) / B
RES = 1.0 * A / B # equivalent to (1.0*A)/B as "*" and "/" are equal priority so the expression is evaluated left to right
</pre>

Common problems - Revision history

Freddie Akeroyd: New page: ==Why does A/B give the wrong answer?== If A and B are both integers, then integer division is performed; if either is a floating point number then the integer value is first converted int...