Common problems
From OpenGENIE
Jump to navigationJump to searchThe printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
Why does A/B give the wrong answer?
If A and B are both integers, then integer division is performed; if either is a floating point number then the integer value is first converted into a floating point number before division is performed. Thus:
1/2 == 0 # 1 and 2 are of Integer type
but
1/2.0 == 0.5 # 2.0 is a floating point number, so integer 1 is converted to 1.0 before division is done
If you are unsure what data type two variables may be, then you can force real division using either of these statements
RES = AS_REAL(A) / B RES = 1.0 * A / B # equivalent to (1.0*A)/B as "*" and "/" are equal priority so the expression is evaluated left to right
