Path: news.mathworks.com!not-for-mail
From: <HIDDEN>
Newsgroups: comp.soft-sys.matlab
Subject: Re: robust truncation at n decimal places
Date: Sun, 7 Dec 2008 05:53:02 +0000 (UTC)
Organization: The MathWorks, Inc.
Lines: 23
Message-ID: <ghfobu$blk$1@fred.mathworks.com>
References: <545015a5-3f51-41e3-a40b-1cf7db531b11@35g2000pry.googlegroups.com>
Reply-To: <HIDDEN>
NNTP-Posting-Host: webapp-02-blr.mathworks.com
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
X-Trace: fred.mathworks.com 1228629182 11956 172.30.248.37 (7 Dec 2008 05:53:02 GMT)
X-Complaints-To: news@mathworks.com
NNTP-Posting-Date: Sun, 7 Dec 2008 05:53:02 +0000 (UTC)
X-Newsreader: MATLAB Central Newsreader 1187260
Xref: news.mathworks.com comp.soft-sys.matlab:505433


"Felipe G. Nievinski" <fgnievinski@gmail.com> wrote in message <545015a5-3f51-41e3-a40b-1cf7db531b11@35g2000pry.googlegroups.com>...
> Hi. Given
>     x = 0.99899998
>     y = 0.99900000
>     za = sscanf(sprintf('%.7f\n',x), '%f')
>     zb = fix(x./10^-7).*10^-7
>     za == y
>     zb == y
> then za == y is true, but zb == y is false.
> As you see, it's possible to perform a robust truncation at n decimal
> places through conversion to/from character string. I was trying to
> achive the same using computationally cheaper double precision
> arithmetic, without success. Is that it or am I missing something?
> 
> Thanks,
> Felipe.

  Your 'fix' is doing truncation - that is to say, rounding toward zero - whereas sprintf('%.7f\n',x) is doing a round to nearest.  It's not surprising you get different answers.  Also for better accuracy you should be multiplying by 10^7 instead of dividing by 10^-7 and then dividing by 10^7 instead of multiplying by 10^-7, because the number 10^-7 cannot be represented precisely with binary floating point numbers.

  Nevertheless even with these corrections, you probably cannot count on always getting precisely the same answers clear out to the 53rd binary place unless you know exactly how sprintf's and sscanf's algorithms work.  You should be making comparison like abs(za-zb) < tol where tol is something like 10^-15*abs(x)

Roger Stafford