How to read Binary data little endain from a . jsf file and display an image?

I am stuck on trying to extract the data of a .jsf file to produce an image of the binary data. When viewing the format of the data I have the following:

From what I was told by the creator: Each message starts with a SonarMessageHeaderType structure (from SonarMessages.h) which is 16 bytes long. The key field, byteCount indicates the bytes to follow. So to read a JSF file, read the 16-byte header, and then read the number of bytes to follow to get to the next message.To create a time series for this format, extract the array of 16-bit unsigned integers, scale by 2 ** -n where N is the weighting factor in the header.

For this extraction I am trying to use Matlab to read the specific file and then input the data into a function to output an image. From a previous post I have posted,Previous post, my code so far looks as followed:

clear
clc

fid= fopen('20160503.143239.0000000.jsf', 'r', 'l') % removed b from 'rb'

if fid == -1
    error('File is not opened');
end
%data=[];
record=0;

%X= fseek( fid, 256, 'cof')

while ~feof(fid)
    
   data = fread(fid,1,'uint16');
   %image(data)
   record=record+1;
end

fclose(fid);

I am opening the data file to read binary data (rb) and then using ' l ' to use little endain as the machinefmt states machinefmt . I set an if statement to determine if the file is opened correctly, which when I run the above code it does not produce an error. I then run into two situations that I am stuck on. If I am supposed to read the binary data which follows a 16 byte header and a 240 byte sub header , should I use the

X = fseek(fid, 256, 'cof')

to skip over the header and sub header and start at the binary data before reading the data?

Second problem is when I am trying to read the data. I created a while statement to have the fread function read the file until the end. The file contains over 22,000 lines of data. From what the creator said I need to extract a 16-bit unsigned integer array, so i used the 'uint16' in the fread. When I run the code however, I am receiving an empty 0x0 array for data. The attached image shows part of the file in a jsf viewer for more detail about the file. Could anyone put me in the right direction onto how exactly I should be extracting the file to get the array and plug it into a function like image() to see the acoustic images? Would I have to get some kind of count on the file being read to be able to set data = zeros(x,y) before, then reopen the file and extract? Sorry in advance for the lack of coding skill, this is my first time trying to extract a file using Matlab.

[Guillaume]

It would probably be helpful if you shared the specification of your file format (that SonarMessages.h file and probably others). I don't really understand what your screenshot is showing.

The format of your binary file sounds well designed, so it's probably easy to decode.

Note that 'b' is not listed as a valid permission type for fopen. Matlab probably accepts (and ignore) it for compatibility with C. Files are open as binary by default.

Also note that by default, files are opened in little endian format as that's the native format of all platforms currently supported by matlab. It doesn't hurt to specify 'l' even though it's the default.

[James McGinley]

Sorry about not providing the .h file I have attached it here. The image that I attached in the original post is what the messages look like for each line, the right side is the details of the selected message. I am not sure how much I could show of the viewer as it may be classified (I am a Master's student working on a Navy project). The .jsf file was created by edgetechEdgetech pdf

I also forgot to mention it is side-scan sonar so there is 2 pings sorry for the confusion

[Guillaume]

Right, now I understand what your screenshot is showing. It's a list of the message ids and their offsets, with the details of a few messages on the right. It's not particularly useful in helping to understand the file format.

The pdf you linked on the other hand is all that is needed.

In order to read the messages you need to read the message header. Some fields you can ignore and skip but personally, I'd keep everything around in a structure. Then you can read the message data. Now, the structure of the message data varies by message type. If you don't care about a type of message you can just skip it (or store it as a stream of bytes) but for these messages you are interested in, you need to decode the message data according to its format.

So, which message type are you interested in? If it's type 80, you can't just ignore the 240 byte header, some information is required to decode the samples properly.

[James McGinley]

The messages that I am interested in is type 80. (the sidescan sonar data).

Answer 1

[Guillaume]

Untested code obviously since I don't have a file to test with. Bugs/typos possible.

See attached.

edit: fixed a few bugs

[James McGinley]

Wow, so I would have to actually read each and every header / subheader in the file to be able extract the information. I will test the code out in the morning and report back how it goes! I really appreciate you taking the time to help me!

[Guillaume]

No you don't have to read all of the header. You could indeed skip most of it and reading just the essential fields (I've marked those in the code). However, since the information is there, why not read it. Experience has taught me that sooner or later you're going to want that information.

However, I've realised that I should have structured the program better by reading each header in just one fread and then splitting it in afterward into fields as defined by by a table or cell array definition. That would be more easier to debug, less error-prone and easier to extend.

I may rewrite it tonight if I've got the time and am sufficiently motivated.

[James McGinley]

I am receiving the following error:

Error using decodejsf>decodesonardata (line 119)
Should have read 240 bytes. Check code above for mistakes

Error in decodejsf>decodenextmessage (line 43)
        message.content = decodesonardata(fid, datasize);

Error in decodejsf (line 25)
        messages = [messages, decodenextmessage(fid)]; %#ok<AGROW> We don't know how many messages there will be
 

I will check the code for any missmatches with the pdf details

[Guillaume]

Yes, it was missing the depth field. I've fixed the original code (and a few other bugs).

I've also written a completely new design as explained above. I'd recommend you use that. It's easier to extend to support other types of messages. It also automatically check that the record description actually matches what is expected so mistakes like the above are easier to catch.

With that new code, if you don't want to keep just the essential fields of the sonar message, you could replace the sonarheader description with:

        sonarheader = cell2table({
            ''                    16   'int8'
            'msb'                 1    '*uint16'   %probably critical
            ''                    16   'int8'
            'dataformat'          1    'int16'     %critical field
            ''                    78   'int8'
            'samplecount'         1    'uint16'    %critical field
            ''                    52   'int8'
            'weightingfactor'     1    'int16'     %critical field
            ''                    70   'int8'
            }, 'VariableNames', {'Name', 'Count', 'Type'}); 

and it'd still work.

Answer 2

[Ivan V. Dmitriev]

Try to use https://www.mathworks.com/matlabcentral/fileexchange/73214-gjsf

[Guillaume]

I would recommend adding some documentation on how to use your code directly to the FileExchange submission rather than forcing potential users to dig through your other website.

Comparing your code for reading message 80 to what I wrote in my answer (see decodejsf2.m), you have a mismatch between your structure declaration and the actual reading. You're following the MSB with just 3 reserved field (I've got 5), however when you read the data, after the MSB you're filling two non-declared LSB fields before reading 3 resereved fields.

%your declaration
Head=struct(..
    .., 'MSB',zeros(1,LenHead),'Reserved2',zeros(3,LenHead), ..
    
%your reading:
Head.MSB(m)=fread(fId,1,'uint16'); 
%these two fields not declared!
Head.LSB(m)=fread(fId,1,'uint16');
Head.LSB2(m)=fread(fId,1,'uint16');
%now the reserved fields
Head.Reserved2(:,m)=fread(fId,3,'int16');

See my code for a maintainable way of specifying the data structure which guarantees that the reading actually matches the structure.

[Ivan V. Dmitriev]

>> I would recommend adding some documentation on how to use your code directly to the FileExchange submission rather than forcing potential users to dig through your other website.

Please explain what do you mean? The same pdf-document is in the archive both on the FileExchange and on the "other website".

>> See my code for a maintainable way of specifying the data structure which guarantees that the reading actually matches the structure.

Looked at your code. Thanks for the programming style, I took a couple of things for myself. But, for use "in production":

>> tic;messages=decodejsf2('d:\2\088_EVT_site_sentral.jsf');toc

Elapsed time is 4300.944202 seconds.

>> tic;JsfHead=gJsfHeaderRead('d:\2\088_EVT_site_sentral.jsf',1);toc

Mess: 0040=System Status Message (private), Num: 251 [ Subs: 0, Num: 251; Chan: 0, Num: 251 ]

Mess: 0080=Sonar Data Message, Num: 66231 [ Subs: 20, Num: 33116; Chan: 0 1, Num: 16558 16558 ][ Subs: 21, Num: 33115; Chan: 0 1, Num: 16557 16558 ]

Mess: 0181=N......

Mess: 2060=Pressure Sensor Data, Num: 3931 [ Subs: 101, Num: 3931; Chan: 0, Num: 3931 ]

Elapsed time is 10.023261 seconds.

>> tic;[Head,Data]=gJsf0080Read(JsfHead,0,20);toc

Elapsed time is 103.890087 seconds.