Posted by Weston Ganger
I was noticing a weird issue in ruby where I had a division statement that was returning an integer when it should have returned a decimal value. Check this out.
12 / 5 #=> 2 <-- This is the wrong value, it should be 2.4
Turns out this is because its doing integer division so it returns an integer. To make it return a decimal value you should add a decimal point to one of the number.
12.0 / 5 #=> 2.4 12 / 5.0 #=> 2.4 12.to_f / 5 #=> 2.4
This still may not product the correct value sometimes. There are odd cases where floats will return the wrong values.
0.1 + 0.2 #=> 0.30000000000000004 0.05 + 0.93 #=> 0.9800000000000001
Wait a second, what? Yes this is extremely basic math and its failing. This is why they always say use Decimals not Floats when doing accurate calculations. So here is how to get the correct value.
number = 0.1 + BigDecimal('0.2') # Note the value passed to BigDecimal is a string, this is important to maintain accuracy number.to_s('F') #=> 0.3 '# F' specifies using floating point notation instead of engineering
Related External Links: