Format longeng - rounding error

1 view (last 30 days)
Dave
Dave on 6 Sep 2013
Hello All,
When using "format longeng", I am getting strange results to basic maths but when using "format short" the correct / expected answer is returned:
>> format short
>> 0.088 - 0.08
ans =
0.0080
>> format longeng
>> 0.088 - 0.08
ans =
7.99999999999999e-003
Is this a bug? Am I missing something? would this make a difference to my simulations?
For reference, I am running Microsoft Windows 7 Version 6.1 (Build 7601: Service Pack 1) on an Intel Xeon 64bit W3520 2.67GHz
MATLAB Version: 8.1.0.604 (R2013a)

Accepted Answer

Roger Stafford
Roger Stafford on 6 Sep 2013
Whatever your matlab version and whatever your operating system, you are attempting to do decimal arithmetic on a machine that uses binary numbers. For that reason it cannot represent 0.008 exactly. For that matter it can represent neither 0.088 nor 0.08 exactly. If you display the number the machine actually uses to sufficiently many digits, you will always be able to see the difference. However (for double precision numbers) the error tends to be out in the fifteenth or sixteenth decimal place. It is something all computer users must become accustomed to.
  1 Comment
Dave
Dave on 6 Sep 2013
Roger,
I have also been looking at http://support.microsoft.com/kb/42980 - this helped me to understand.
Many thanks, Dave

Sign in to comment.

More Answers (0)

Categories

Find more on Function Creation in Help Center and File Exchange

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!