Common problems
From OpenGENIE
Revision as of 17:29, 27 May 2009 by Freddie Akeroyd (talk | contribs) (New page: ==Why does A/B give the wrong answer?== If A and B are both integers, then integer division is performed; if either is a floating point number then the integer value is first converted int...)
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
Why does A/B give the wrong answer?
If A and B are both integers, then integer division is performed; if either is a floating point number then the integer value is first converted into a floating point number before division is performed. Thus:
1/2 == 0 # 1 and 2 are of Integer type
but
1/2.0 == 0.5 # 2.0 is a floating point number, so integer 1 is converted to 1.0 before division is done
If you are unsure what data type two variables may be, then you can force real division using either of these statements
RES = AS_REAL(A) / B RES = 1.0 * A / B # equivalent to (1.0*A)/B as "*" and "/" are equal priority so the expression is evaluated left to right
