Approximation problem???

Tales — Fri, 20 Nov 2009 16:42:18 -0500

Hi,

I am having a really weird problem that I cant find a reason. After a lot of calculation I was getting a wrong value for a variable, so I debbuged to find where was the problem, and I found a simple division mistake. A double h=0.0030 and another double hh=1/h .
Of course hh is suposed to be 333.3333 but I am getting hh=333.8898
Here is the piece of code:

h=str2double(cell2mat(raw(nameLinNum,densColumn)));
hh=1/h;

Does anyone hava an idea on how to fix it???

Thanks

Re: Approximation problem???

dpb — Fri, 20 Nov 2009 17:01:47 -0500

Tales wrote:
...
> I am having a really weird problem that I cant find a reason. After a
> lot of calculation I was getting a wrong value for a variable, so I
> debbuged to find where was the problem, and I found a simple division
> mistake. A double h=0.0030 and another double hh=1/h .
> Of course hh is suposed to be 333.3333 but I am getting hh=333.8898
> Here is the piece of code:
>
> h=str2double(cell2mat(raw(nameLinNum,densColumn)));
> hh=1/h;
>
> Does anyone hava an idea on how to fix it???
...

Undoutedly h _isn't_ 0.00300000... but closer to 0.002995...

Try

disp(h-0.0030)

during a debugging session and see what is returned.

Why that is isn't determinable from the data provided but delving into
where the underlying data come from and how they're processed will
surely uncover same.

--

MATLAB Central Newsreader - Approximation problem???

Approximation problem???

Re: Approximation problem???