http://www.mathworks.com/matlabcentral/newsreader/view_thread/265483
MATLAB Central Newsreader  Guess the data encoding!
Feed for thread: Guess the data encoding!
enus
©19942015 by MathWorks, Inc.
webmaster@mathworks.com
MATLAB Central Newsreader
http://blogs.law.harvard.edu/tech/rss
60
MathWorks
http://www.mathworks.com/images/membrane_icon.gif

Tue, 10 Nov 2009 16:59:19 +0000
Guess the data encoding!
http://www.mathworks.com/matlabcentral/newsreader/view_thread/265483#693587
Geoffrey Akien
Hi everybody, I'm trying to reverse engineering an instrument at work (University). Using a serial spy cable I've figured out what its sending. At regular intervals the device broadcasts some information (shown in hexadecimal here), with the corresponding values the software spits out:<br>
<br>
00 76 64 04 00 DE == 3.590E004<br>
00 8C 61 0C 00 F9 == 1.012E001<br>
<br>
(I can give out more values if anyone is interested.)<br>
<br>
It seems like either the first set of 00s are signify the start if the data, or perhaps the last two bytes are a checksum of some sort so that the number is single precision. I've been messing around with typecast for a while now and can't seem to get anywhere.<br>
<br>
In the mean time I'm trying to get hold of the author of the software, but right now I can't for the life of me figure out what encoding the software is using.<br>
<br>
Any suggestions?

Tue, 10 Nov 2009 20:48:01 +0000
Re: Guess the data encoding!
http://www.mathworks.com/matlabcentral/newsreader/view_thread/265483#693627
Jan Simon
Dear Geoffrey!<br>
<br>
> Hi everybody, I'm trying to reverse engineering an instrument at work (University). Using a serial spy cable I've figured out what its sending. At regular intervals the device broadcasts some information (shown in hexadecimal here), with the corresponding values the software spits out:<br>
> <br>
> 00 76 64 04 00 DE == 3.590E004<br>
> 00 8C 61 0C 00 F9 == 1.012E001<br>
> <br>
> (I can give out more values if anyone is interested.)<br>
<br>
Please give us some more numbers.<br>
Jan

Wed, 11 Nov 2009 10:56:04 +0000
Re: Guess the data encoding!
http://www.mathworks.com/matlabcentral/newsreader/view_thread/265483#693797
Geoffrey Akien
"Jan Simon" <matlab.THIS_YEAR@nMINUSsimon.de> wrote in message <hdcji0$j04$1@fred.mathworks.com>...<br>
> Dear Geoffrey!<br>
> <br>
> > Hi everybody, I'm trying to reverse engineering an instrument at work (University). Using a serial spy cable I've figured out what its sending. At regular intervals the device broadcasts some information (shown in hexadecimal here), with the corresponding values the software spits out:<br>
> > <br>
> > 00 76 64 04 00 DE == 3.590E004<br>
> > 00 8C 61 0C 00 F9 == 1.012E001<br>
> > <br>
> > (I can give out more values if anyone is interested.)<br>
> <br>
> Please give us some more numbers.<br>
> Jan<br>
<br>
Firstly, those numbers I gave you were wrong, since I had some extra processing turned on in the software. Secondly, I've done some digging and I've figured out how the software encodes integers, but not floating points.<br>
<br>
When it sends commands to change the settings an example is:<br>
<br>
(all numbers in hexadecimal)<br>
<br>
10000: 82 00 10 27 B9<br>
100: 82 00 64 00 E6<br>
10: 82 00 0A 00 8C<br>
<br>
The first byte is the command to write a new parameter, the second byte says which parameter to change, and the final byte is a checksum. So 10000 in hex is 2710, which gets sent in the "wrong" order. But that's integers.<br>
<br>
When the software is collecting data, the device broadcasts 6byte lumps:<br>
<br>
There's an option to change the data to linear or not and I have no idea which is the native data format.<br>
<br>
00 51 38 04 00 8D == 3.178528E0004 (roughly 34.5 dB  not clear if it arrives as logarithmic or not)<br>
00 BC 47 04 00 07 == 3.316628E0004<br>
00 0C 64 0C 00 7C == 1.019e1<br>
<br>
The 1st and 5th bytes are always 00. I have a suspicion that the first byte signifies the start of transmission, and that perhaps the last byte is a checksum? I've tried playing around with typecast(uint8(data(2:5)), 'single') with and without byte swapping but I can't seem to arrive at a sensible answer.<br>
<br>
Thanks

Thu, 12 Nov 2009 10:07:01 +0000
Re: Guess the data encoding!
http://www.mathworks.com/matlabcentral/newsreader/view_thread/265483#694137
Jan Simon
Dear Geoffrey!<br>
<br>
> 00 51 38 04 00 8D == 3.178528E0004 (roughly 34.5 dB  not clear if it arrives as logarithmic or not)<br>
> 00 BC 47 04 00 07 == 3.316628E0004<br>
> 00 0C 64 0C 00 7C == 1.019e1<br>
<br>
Dechiffering a code with 32 bits with just 3 examples is a kind of random guessing. Please give me more data.<br>
Jan

Thu, 12 Nov 2009 15:30:20 +0000
Re: Guess the data encoding!
http://www.mathworks.com/matlabcentral/newsreader/view_thread/265483#694233
Geoffrey Akien
I hope this lot helps, I've given you two sets at different levels so you might be able to get something out of it. I'm not sure if the program receives it in logarithmic or linear so there's a set for each (sorry can't get common data sets for that).<br>
<br>
Thanks<br>
<br>
<br>
Set 1 (LED on)<br>
<br>
Intensity/dB Data<br>
34.96496 00 62 39 04 00 9F<br>
34.91826 00 48 3D 04 00 89<br>
34.92692 00 8F 3C 04 00 CF<br>
34.99225 00 1B 37 04 00 56<br>
34.95064 00 94 3A 04 00 D2<br>
34.87488 00 E7 40 04 00 2B<br>
35.02823 00 1A 34 04 00 52<br>
34.97919 00 32 38 04 00 6E<br>
35.02529 00 59 34 04 00 91<br>
35.01391 00 4C 35 04 00 85 <br>
34.95083 00 90 3A 04 00 CE<br>
34.87460 00 ED 40 04 00 31<br>
34.91756 00 57 3D 04 00 98<br>
34.88213 00 4C 40 04 00 90<br>
<br>
Intensity Data<br>
3.205632E0004 00 65 3B 04 00 A4<br>
3.221248E0004 00 28 3D 04 00 69<br>
3.288833E0004 00 AF 44 04 00 F7<br>
3.178254E0004 00 49 38 04 00 85<br>
3.234709E0004 00 AB 3E 04 00 ED<br>
3.214348E0004 00 61 3C 04 00 A1<br>
3.242072E0004 00 7E 3F 04 00 C1<br>
3.149922E0004 00 0A 35 04 00 43<br>
3.231469E0004 00 4E 3E 04 00 90<br>
3.222498E0004 00 4C 3D 04 00 8D<br>
3.165778E0004 00 DC 36 04 00 16<br>
<br>
<br>
Set 2 (LED off)<br>
<br>
Intensity/dB Data<br>
16.81825 00 23 24 0A 00 51<br>
16.80014 00 A6 25 0A 00 D5<br>
16.80355 00 5D 25 0A 00 8C<br>
16.80056 00 9D 25 0A 00 CC<br>
16.80098 00 94 25 0A 00 C3<br>
16.79494 00 15 26 0A 00 45<br>
16.79335 00 37 26 0A 00 67<br>
16.79434 00 22 26 0A 00 52<br>
16.78652 00 C9 26 0A 00 F9<br>
16.78858 00 9D 26 0A 00 CD<br>
16.78165 00 31 27 0A 00 62<br>
16.78774 00 AF 26 0A 00 DF<br>
16.78839 00 A1 26 0A 00 D1<br>
16.78357 00 08 27 0A 00 39<br>
16.77936 00 62 27 0A 00 93<br>
16.78039 00 4C 27 0A 00 7D<br>
16.78338 00 0C 27 0A 00 3D<br>
16.78582 00 D8 26 0A 00 08<br>
16.77623 00 A5 27 0A 00 D6<br>
<br>
Intensity Data<br>
2.113343E0002 00 CF 29 0A 00 02<br>
2.115189E0002 00 20 2A 0A 00 54<br>
2.115439E0002 00 2B 2A 0A 00 5F<br>
2.120963E0002 00 1D 2B 0A 00 52<br>
2.122014E0002 00 4B 2B 0A 00 80<br>
2.124989E0002 00 CD 2B 0A 00 02<br>
2.129459E0002 00 90 2C 0A 00 C6<br>
2.133501E0002 00 40 2D 0A 00 77<br>
2.137182E0002 00 E0 2D 0A 00 17<br>
2.143617E0002 00 F7 2E 0A 00 2F

Thu, 12 Nov 2009 21:21:27 +0000
Re: Guess the data encoding!
http://www.mathworks.com/matlabcentral/newsreader/view_thread/265483#694355
tristram.scott@ntlworld.com (Tristram Scott)
Geoffrey Akien <geoff.akien@gmail.com> wrote:<br>
> I hope this lot helps, I've given you two sets at different levels so you<br>
might be able to get something out of it. I'm not sure if the program<br>
receives it in logarithmic or linear so there's a set for each (sorry can't<br>
get common data sets for that).<br>
> <br>
<br>
The extra data helps a lot. It looks to me as though the data<br>
corresponding to the intensities is contained in the first four bytes, but<br>
they are arranged from least to most significant byte, so turn them around.<br>
<br>
h = [<br>
'04396200' <br>
'043D4800' <br>
'043C8F00' <br>
'04371B00' <br>
'043A9400' <br>
'0440E700' <br>
'04341A00' <br>
'04383200' <br>
'04345900' <br>
'04354C00' <br>
'043A9000' <br>
'0440ED00' <br>
'043D5700' <br>
'04404C00' <br>
]<br>
<br>
y = [<br>
34.96496<br>
34.91826<br>
34.92692<br>
34.99225<br>
34.95064<br>
34.87488<br>
35.02823<br>
34.97919<br>
35.02529<br>
35.01391<br>
34.95083<br>
34.87460<br>
34.91756<br>
34.88213<br>
]<br>
<br>
x = hex2dec(h)<br>
plot(x,y,'+')<br>
<br>
<br>
 <br>
Dr Tristram J. Scott <br>
Energy Consultant

Fri, 13 Nov 2009 00:01:05 +0000
Re: Guess the data encoding!
http://www.mathworks.com/matlabcentral/newsreader/view_thread/265483#694390
Jan Simon
Dear Tristram, dear Geoffrey!<br>
<br>
> The extra data helps a lot. It looks to me as though the data<br>
> corresponding to the intensities is contained in the first four bytes, but<br>
> they are arranged from least to most significant byte, so turn them around.<br>
> <br>
> h = [<br>
> '04396200' <br>
> '043D4800' <br>
> '043C8F00' <br>
> '04371B00' <br>
> '043A9400' <br>
> '0440E700' <br>
> '04341A00' <br>
> '04383200' <br>
> '04345900' <br>
> '04354C00' <br>
> '043A9000' <br>
> '0440ED00' <br>
> '043D5700' <br>
> '04404C00' <br>
> ]<br>
> <br>
> y = [<br>
> 34.96496<br>
> 34.91826<br>
> 34.92692<br>
> 34.99225<br>
> 34.95064<br>
> 34.87488<br>
> 35.02823<br>
> 34.97919<br>
> 35.02529<br>
> 35.01391<br>
> 34.95083<br>
> 34.87460<br>
> 34.91756<br>
> 34.88213<br>
> ]<br>
> <br>
> x = hex2dec(h)<br>
> plot(x,y,'+')<br>
<br>
Well done, Tristram!<br>
y = hex2dec(x) * 1.82800636878895e007  47.9199538652254<br>
This is not exactly what I expected.<br>
Jan

Fri, 13 Nov 2009 06:55:05 +0000
Re: Guess the data encoding!
http://www.mathworks.com/matlabcentral/newsreader/view_thread/265483#694443
Geoffrey Akien
That's fantastic! I didn't think of doing it that way, I'll remember that technique for next time. Thanks Tristram!

Fri, 13 Nov 2009 08:52:39 +0000
Re: Guess the data encoding!
http://www.mathworks.com/matlabcentral/newsreader/view_thread/265483#694465
tristram.scott@ntlworld.com (Tristram Scott)
Geoffrey Akien <geoff.akien@gmail.com> wrote:<br>
> That's fantastic! I didn't think of doing it that way, I'll remember<br>
> that technique for next time. Thanks Tristram!<br>
<br>
The pattern was there in the larger data set, with the fourth column<br>
looking pretty much the same within a group of values.<br>
<br>
The last two digits look to be a simple checksum. They are the last eight<br>
bits of the total of all the rest of the line.<br>
<br>
00 D8 26 0A 00 08<br>
<br>
D8 + 26 + 0A = 108.<br>
<br>
<br>
<br>
 <br>
Dr Tristram J. Scott <br>
Energy Consultant