File Exchange

image thumbnail

DataHash

version 1.5 (6.66 KB) by

MD5 or SHA hash for array, struct, cell or file

98 Downloads

Updated

View License

DATAHASH - Hash for Matlab array, struct, cell or file
Hash = DataHash(Data, Opt)
  Data: Array of built-in types (U)INT8/16/32/64, SINGLE, DOUBLE (real or complex)
        CHAR, LOGICAL, CELL, STRUCT (scalar or array, nested), function_handle.
      
  Opt: Options struct:
        Opt.Method: 'SHA-1', 'SHA-256', 'SHA-384', 'SHA-512', 'MD2', 'MD5'.
        Opt.Format: Output format: 'hex', 'HEX', 'double', 'uint8', 'base64'
        Opt.Input:
             'array': The contents, type and size of the input [Data] are
                    considered for the creation of the hash. Nested CELLs
                    and STRUCT arrays are parsed recursively. Empty arrays of
                    different type reply different hashs.
             'file': [Data] is treated as file name and the hash is calculated
                    for the files contents.
             'bin': [Data] is a numerical, LOGICAL or CHAR array. Only the
                    binary contents of the array is considered, such that
                    e.g. empty arrays of different type reply the same hash.
             'ascii': Same as 'bin', but only the 8-bit ASCII part of the 16-bit
                    Matlab CHARs is considered.
  Hash: String or numeric vector.
EXAMPLES:
Default: MD5, hex:
  DataHash([]) % 7de5637fd217d0e44e0082f4d79b3e73
SHA-1, Base64:
  S.a = uint8([]);
  S.b = {{1:10}, struct('q', uint64(415))};
  Opt.Format = 'base64';
  Opt.Method = 'SHA-1';
  DataHash(S, Opt) % ZMe4eUAp0G9TDrvSW0/Qc0gQ9/A
Comparison with standard hash programs:
    Opt.Method = 'SHA-256';
    Opt.Input = 'ascii'; % Important! Matlab CHARs have 16 bits
    DataHash('abc', Opt)
Feel free to ask me about a version for Matlab 6.5.

Michael Kleder's "Compute Hash" works similar, but does not accept structs, cells or files:
  http://www.mathworks.com/matlabcentral/fileexchange/8944
"GetMD5" is 2 to 100 times faster, but it replies only MD5 hashes and a C-compiler is required:
  http://www.mathworks.com/matlabcentral/fileexchange/25921

Tested: Matlab 7.7, 7.8, 7.13, 8.6, WinXP, Win7/64, Java: 1.3, 1.6, 1.7

Bugreports and enhancement requests are welcome.

Comments and Ratings (46)

Kai Zhang

Chris

Chris (view profile)

Thank you very much.

Keith Ma

Thanks for this submission.

It just works, and the author is also quick and competent in user support. Thanks!

Jan Simon

Jan Simon (view profile)

@Christian: Thanks you to finding this bug. It is fixed now.

Christian

Hey this is a great function! I've run into a problem though. I found if the file is empty (size of 0 bytes), it throws an error:

Error using java.security.MessageDigest$Delegate/update
A Non-scalar value was passed for a scalar argument

Error in DataHash (line 231)
   Engine.update(Data);

Perhaps it should return whatever the hash is for a empty stream or an empty string?

Awesome function! It would be great if other encodings (such as base32 and base36) were supported as well. I implemented base32 as an extension of yours base64 (though not 100% sure if correctly) but have no clue how to get base36.

Jan Simon

Jan Simon (view profile)

@Uilke: Wow, this is a bug in the documentation. 240f... is the expected output. The results have been changed in the version submitted on 30-Mar-2015, but I forgot to update the help text accordingly.

Tried datahash([]) on Win7 64-bits with ML 7.13 (R2011b) and 8.6 (R2015b) and with both get 240f5f01f052bd89f38da2165dcf25c7 instead of the in the help mentioned 7de5637fd217d0e44e0082f4d79b3e73
Any ideas why?

Matt Raum

@Matt Raum: I found a workaround using the undocumented serialization function getByteStreamFromArray

DataHash(getByteStreamFromArray(MyClassObj))

gets me what I need.

Matt Raum

Jan -- Great work on this, very handy! I'd like to be able to use this to hash an instance of a custom class but I'm getting the error below:

Warning: Type of variable not considered: MyClass
> In DataHash>CoreHash_ at 336
  In DataHash at 253

Can you help?

Matt Raum

Jan Simon

Jan Simon (view profile)

@Ivan Cordon: The effect is explained in NOTES in the help section and was discussed at 26 May 2012:
To get the same result as in the online generator, only the binary contents of the data must be considered, not the Matlab class, and only the 8-bit ASCII part of the 16-bit Matlab CHAR:
Opt.Method='SHA-256'; Opt.Input='bin';
DataHash(uint8('ivan'), Opt)
Then you get cd0b9452... also.

Or a little bit shorter with the newest version:
Opt.Method='SHA-256'; Opt.Input='ascii';
DataHash('ivan', Opt)

Ivan Cordon

Dear Jan,

The program works for me but I seem to be getting a wrong result. I inputting the Opt struct with these parameters:

Method: 'SHA-256'

and calling the function as follows:

Hash = DataHash('ivan', Opt)

the result i am getting is:

53607f405e61c5b4bd7796d5ef1ce7cc123559094123c314726c68aac1e21dab

whereas checking on the different online generators (http://www.xorbin.com/tools/sha256-hash-calculator) i get this:

cd0b9452fc376fc4c35a60087b366f70d883fc901524daf1f122fbd319384f6a

Do you know why I am getting this difference?

Thanks!!

Jan Simon

Jan Simon (view profile)

@Haitam: Do you want to obtain the clear text from the hash value? This is possible only if the input has not more bits than the hash value. For longer messages the result is not unique. And even for short messages the hashing algorithms are not designed for a reverse analysis. It is possible e.g. by a brute force attack, but this problem is not covered by my submission.

Haitham

Dear simon , the code is working great , i would like to know how i can recompute the hash value?
The data hash generated 558f68181d2b0c9d57d41ce7aa36b71d9
i would like to get the original value before hashing it.
thanks a lot...

Jan Simon

Jan Simon (view profile)

@Haitham: Look at the examples for a valid method to call this function. The error message tells you that you did not provide input arguments, or too many of them. Check the command or contact me under the address given in the help section (THISYEAR is "2010", sorry, a typo).

Haitham

Dear jan
thanks for the file , but i am getting error while running the code it self.

Error using DataHash>Error_L (line 466)
*** DataHash: 1 or 2 inputs required.

Error in DataHash (line 177)
   Error_L('BadNInput', '1 or 2 inputs required.');

Daniel Golden

Great submission, Jan. Consider putting the code on GitHub so that users can more easily comment and contribute.

Jan Simon

Jan Simon (view profile)

Unfortunately there are some collisions (same hash for different data). E.g.: The scalar 0 and the empty double array [1 x 1 x 0] get the same hash. I'm going to publish a new version, which fixes these bugs, but produce other hashs in consequence. In addition sparse arrays are considered.

The submission CalcMD5 is updated also, such that (nested) cells and structs are considered also. Because the overhead of calling Java is omitted, this is dramatically faster than DataHash, I get a speedup factor > 100 for nested structs.

Aslak Grinsted

Aslak Grinsted (view profile)

Well written, good running time

Giorgio

Well written!

Clemens

Nath

Nath (view profile)

function handle refering to struct containing the function will create infinite loop. Is there any workaround ?

Exemple:

d= dynamicprops();
addprop(d,'f');
d.f= @(varargin) struct2cell(d);
DataHash(d.f) % infinite loop

Arindam Bose

Arindam Bose (view profile)

Hi, Do anyone have any idea, how to decrypt an MD5 code?

Igor

Igor (view profile)

David

David (view profile)

Hi, I'm just wondering if this function works for p-files? Line 144-147 tells the function to stop if the file is not an m-file. When I comment these lines out, it seems to provide the correct hash for the p-file.

Jan Simon

Jan Simon (view profile)

Thanks, Jan, a good idea. I will include this in the next version.

Great function! Just as a little improvement, I added support for Java and MATLAB objects by calling their hashCode function. Just insert

elseif (isobject(Data) || isjava(Data))....
        && ismethod(Data, 'hashCode')
        Engine = CoreHash(Data.hashCode, Engine);

into the CoreHash() and CoreHash_() functions (in the main if branches).

Regards, Jan

Jan Simon

Jan Simon (view profile)

The examples use 8-bit CHAR strings, while Matlab uses 16 bits per CHAR. In addition DataHash includes information about the type and dimension of the input, as described in the help text. Because e.g. Mido asked for a version, which considers the actual data only last September, I've included this feature last year. Note the conversion to 8-bits:
Opt.Input = 'bin'; Opt.Method = 'MD5';
DataHash(uint8('The quick brown fox jumps over the lazy dog.'), Opt)
% >> e4d909c290d0fb1ca068ffaddf22cbd0
as expected.
But DataHash fails for the empty string and binary input! Thanks for finding this bug, it will be fixed soon.

It would be nice to be able to check the returned values against some public results. For example Wikipedia mentions MD5("") = d41d8cd98f00b204e9800998ecf8427e and MD5("The quick brown fox jumps over the lazy dog.")
= e4d909c290d0fb1ca068ffaddf22cbd0, and I don't know how to get the same from DataHash to check it.

Jan Simon

Jan Simon (view profile)

@Oyvind: I have posted an MD5-calculator also as MEX-function. But it does not allow the direct accumulation of the hash for nested cells or structs. But you could do a XORing of the partial hashes. But this will not be trivial.

Oyvind

Oyvind (view profile)

Hi,
Is it possible to run Datahash without Java (I am stuck with the limitation of excel 2003 using a compiled version of datahash tha requires JVM).
Oyvind

Jan Simon

Jan Simon (view profile)

@Mido: Now you can use the Opt.Input='bin' method to create the hash for the raw data.

Jan Simon

Jan Simon (view profile)

@Mido: DataHash considers the class and dimensions of the inputs, otherwise UINT8([0,0]) and UINT16(0) would have the same hash.
I'm going to add the option, that only the contents of the input is considered. But even then the MD5 of 'I am not happy' will not be '59b4...', because Matlab uses 16 bits for a CHAR - you look for UINT8('I am not happy').
You can find a fast Md5 tool here: http://www.mathworks.com/matlabcentral/fileexchange/25921-calcmd5

Mido Mido

It does not work with me!!
When i tried:

Opt.Format = 'HEX';
Opt.Method = 'MD5';
DataHash('I am not happy', Opt)

it gives me : "7C23124A8F69D72A65C0E86A4B9075CF"
although the correct is:
"59b469ea3ffbe72cf4983facf13cbe1f"

The same also for SHA-1!!

Could any one help?

Martin

Martin (view profile)

Aaron

Aaron (view profile)

Extremely well made function.
Very easy to use.
Works correctly.

Very useful for my simulations parameters sets !

Thank you Jan,

Regards,

François.

Updates

1.5

Bugfix: Failed for empty files.

1.5

No need to use TYPECASTX in modern Matlab versions. Hashes have changed for 'Array' mode to reply the same output as GetMD5. Speed improvements for STRUCT arrays.

1.4

In the version 30-Mar-2015 the results have been changed, but the help section has not been adjusted. This is fixed now.
Structs arrays are processed much faster now and the checksum differs from earlier versions.

1.4

Fixed bugs: Strings and empty array for "binary" mode. For "array" mode [] and zeros(1,1,0) had the same hash before.

1.3

Accept empty input for binary mode now.

1.2

Binary mode add to consider only the contents of the data.

MATLAB Release
MATLAB 8.6 (R2015b)

Download apps, toolboxes, and other File Exchange content using Add-On Explorer in MATLAB.

» Watch video