how to make fscanf faster

Hi, all

I have problem with getting data from a sensor using serial communication rs-232. The problem is fscanf takes much time. There is profiler result below.

time  calls      for i=1:1:1501 
2.61   1501          Out(i,1) = eval(fscanf(s));        
0.01   1501      end

The sensor provides data(13X1 char) when it is called. I used a for-loop to acquire whole data (1501X13)

1501 calls increase its time. If I reduce the number of calls, the accuracy will be reduced. I want to have high accurate data that means I need to call 1501 times. Is there any way to make fscanf faster? OR any better code to get data faster in serial communication?

Thanks in advance.

Answer 1

[Cedric]

15 years ago, I would use fread and "format" the data myself. I don't know how the technology evolved, but my guess is that fscanf, which is quite flexible and powerful, generates a significant overhead.

The list of functions dedicated to managing serial com. can be found here: http://www.mathworks.com/help/matlab/serial-port-devices.html

One additional point, do you really need to format the data "real time" as you receive it? Couldn't you just buffer it for a moment and then format the whole in one shot if really needed? On this matter, you could read the doc of fread, in particular the section about InputBufferSize.

EDIT after seeing your screenshot.

If you read directly 1501 floats with fscanf() or 1501*13 char or uint8 with fread(), you don't need the loop at all for reading the whole stream.

One thing that could happen in your profiling (but I am unsure now, because I've not been playing with serial coms for a long time) is that both fscanf() and fread() are blocking methods that won't return before they get the exact amount of data that you ask them to receive. So in all three cases, the 2.6x seconds that you are measuring are the time that it takes internally to receive 1501*13 bytes (for your bit-rate setting), plus a little overhead due to MATLAB processing. And the overhead is almost not visible because MALTAB format processing goes much faster than usual serial communications.

[Jong-Hwan Kim]

Thank you for your advice. I did as you mentioned. I set up InputBufferSize as 13*1501 = 19513 and used fread to get whole data at once, but that took as much time as fscanf does. It does not show any faster speed. Do you have another idea? Thanks.

[Cedric]

I didn't see at first that you were using eval(). As Walter is pointing out, you might be measuring eval() and neither fscanf() nor fread(), in the sense that it requires much more work to interpret on the fly an expression given as a string, than to execute even a complicated format processing with fscanf(). So unless you have a strong reason for using eval(), this is the very first thing to eliminate if you want to increase efficiency.

Then, what is the format/structure of the data stream that you are receiving? Does it really need to be converted into text? If you were for example receiving ascii codes, you could just read them as char with fread(), and then reshape its output into a 1501x13 array.

[Jong-Hwan Kim]

The data acquired from the sensor is like that,

>> fscanf(S)

ans =

 49431.000000

That is 13 char including a space at the end. Using fscanf(s,'%f') does not provide quite diffirence from eval(fscanf(s)). Using char(fread(s,19513)) produced 10.03 sec even it does not have for-loop. Is there another approach? Thanks.

[Jong-Hwan Kim]

Additional Question After reading your recent comment.

So, How can I use fread() to acquire all the data without a loop. Honestly, when I commanded "char(fread(s,19513))", the data comes 19513 X 1 size of array. For example, ...

4

9

4

3

1

.

0

0

0

0

0

0

O

K

It took about 10 seconds and How can I get only 49431? Thank you.

[Cedric]

To summarize, if it makes no difference when you profile with or without feval, or when you profile reading by 13 characters packets or all at once, then it means that the bottleneck is the communication (and that feval,format processing, etc introduce almost no overhead in comparison to the communication).

This doesn't leave you with a lot of options other than to use another type of communication (USB,Firewire,GPIB?), or to change the format of your data. For example, if you have any control on this, send 32bits floats instead of 13*8bits chars. You can compute a rough estimate of the absolute limit of your RS232 communication actually, and see that if your setup is working at 115200 bit/s, sending 19513 characters with 10bits per char (8+2 for the frame) will take 19513*10/115200 = 1.7s approximately. So whatever you do with the current data format, your profiler will never go below 1.7s.

[Jong-Hwan Kim]

Thank you so much. I really appreciate that. I will try and see.

Answer 2

[Walter Roberson]

You do not know you are testing the speed of fscanf(): you might be testing the speed of eval(). Are you certain you need eval?

[Jong-Hwan Kim]

I did not test about how different between eval(fscanf(s)) and just fscanf(s). The profiler shows the line spent 2.6 sec. Let me check it first. However, I used eval to convert from char to number. Is there any option for it that might be faster than using eval? Thanks.

[Walter Roberson]

Do you mean characters such as 'Pi/7' ? Or do you mean characters such as '0.9258e+04' ? If you mean the later, then use fscanf(s, '%f'). If the number of numbers per line is consistent then it is more efficient to repeat the '%f', as in fscanf(s, '%f%f%f%f%f') for 5 space-separated numbers.

[Cedric]

EDIT: please, discard this comment and see Jan's comment below!

Following up on Walter comment, you could generate a format string for e.g. 1501 floats with:

 fmt = char(repmat('%f', 1, 1501)) ;
 values = fscanf(fid, fmt) ;

[Jan]

@Cedric: You can omit the "char()", because repmat('%f', 1, 1501) replies a char already. Is this faster than fscanf(fid, '%f', [1, 1501])?

[Cedric]

Jan, to be honest, I had forgotten that fscanf can take a size arg!

[Jan]

I do not have access to a Matlab computer currently, so I'm not sure which method is faster. A naive C-mex approach to interpret ASCII strings from a file as a decimal number can be much faster than Matlab's FSCANF, see e.g. FEX: str2doubleq. But unfortunately the results are not reliably, e.g. for exceptions like NaN, Inf, malformed numbers, and for numbers which cannot be represented as binary values exactly. Then e.g. the parser of Googles V8 engine is much smarter: http://code.google.com/p/double-conversion/ . But another problem remains, that I do not know an efficient method to access a file from inside a Mex-function, which has been open from Matlab. Therefore I cannot simply plug in the V8 code to create a faster FREAD.

[Jong-Hwan Kim]

Additional result

eval(fgets(s)) and eval(fgetl(s)) produced the same response time. -,.-

Thank you all for advising for my problem. I did what you guys mentioned, but the results showed that the response time was the same as 2.62 sec. Here are detail results from the Profiler.

It was difficult to upload results as pictures here, let me write down below. (1) eval(fscanf(s)) => 2.62 sec (2) fscanf(s, '%f') => 2.62 sec (3) fmt = remat('%f',1,1501); => 0.03 sec Out(i,1) = fscanf(s,fmt); => 2.59 sec => total 2.62 sec

All three approach has the same response time, but all of them acquired data correctly. Is there another approach? I am thinking of that the problem comes from not algorithm but the sensor. I don't know. :( If you guys have another opinion I would appreciate your advice. Thank you.

[Cedric]

See the EDIT in my answer..

[Jong-Hwan Kim]

I tried to use the parallel processing to make the serial communication faster, like

parfor i=1:1:1501                  
   Out(i,1) = str2double(fscanf(s));
end

But the error message said

Unsuccessful read: OBJ must be connected to the hardware with FOPEN.

Error stack: fscanf.m at 154

Error in Radar_System_04_301 (line 38) parfor i=1:1:301

Could you tell me how to fix the code? Thank you.

Answer 3

[Mo]

Has anybody a new answer to the latest question here? Why this error might be produces "Unsuccessful read: OBJ must be connected to the hardware with FOPEN." while the port is already open and ready to use?

[Jan]

Please open a new thread for a new question.