Thread Subject: Avoiding FOR loop for a large data stream

Hi everyone!

I need to read a large data stream (up to 4 million rows). The data stream consists of "frames" with variable length. There's no "end of frame" character in between.

The first byte gives you the number of messageID in the first frame, then there are 2 bytes for that ID, one byte for the length of the data field and then the data itself. This is repeated until all IDs of a frame are done. Then it all starts again with the number of messageIDs in the next frame.

Here's an example:

2 there are 2 messages in this frame (frame 1)
1 HEX ID of message, part one
1 HEX ID of message, part 2
4 length of data field
255 data
255 data
255 data
255 data
2 HEX ID of message, part one
2 HEX ID of message, part 2
3 length of data field
127 data
127 data
127 data
1 there is 1 message in this frame (frame 2)
3 HEX ID of message, part one
3 HEX ID of message, part two
6 length of data field
64 data
64 data
64 data
64 data
64 data
64 data


So I would like to sort it in a way that every message is put in a row and the frame nr is added:

ID1 | data field length | data | FrameNr 1
ID2 | data field length | data | FrameNr 1
ID1 | data field length | data | FrameNr 2

So far I can do this with nested loops...but as u know this takes forever. Unfortunately I don't know any solution to this due to the variable length of the messages/frames.

Do I have to implement this with mex?

Thanks a lot!
Volker

"Volker Klink" <klinkv.NOSPAM@yahoo.de> wrote in message <h9adaj$3nq$1@fred.mathworks.com>...
> Hi everyone!
>
*SNIP description of data structure
> So far I can do this with nested loops...but as u know this takes forever.

No, when properly programmed it does not take forever!

> Unfortunately I don't know any solution to this due to the variable length of the messages/frames.

Use a while loop instead of a for-loop.

Jos

On 22 Sep, 13:44, "Volker Klink" <klinkv.NOS...@yahoo.de> wrote:
> Hi everyone!
>
> I need to read a large data stream (up to 4 million rows). The data stream consists of "frames" with variable length. There's no "end of frame" character in between.

> Do I have to implement this with mex?

No. You have to learn how to program. Parsing files is
a standard task, and decoding the data format should not
need to take very long. As others already have mentioned,
for-loops are not the tool for the job.

Mind you, converting numbers from text format to internal
binary format *does* take a long time - expect 30-60 seconds
per 100 MBytes of text-formatted numbers (not necessarily
file size) to be converted.

Rune