Common problems

From OpenGENIE
Jump to: navigation, search

Why does A/B give the wrong answer?

If A and B are both integers, then integer division is performed; if either is a floating point number then the integer value is first converted into a floating point number before division is performed. Thus:

1/2 == 0       # 1 and 2 are of Integer type


1/2.0 == 0.5   # 2.0 is a floating point number, so integer 1 is converted to 1.0 before division is done

If you are unsure what data type two variables may be, then you can force real division using either of these statements

RES = 1.0 * A / B   # equivalent to  (1.0*A)/B  as "*" and "/" are equal priority so the expression is evaluated left to right