File Exchange

image thumbnail

Rapid lossless data compression of numerical or string variables

version 1.0 (1.86 KB) by

Rapidly compresses (or decompresses) Matlab variables in memory

23 Downloads

Updated

No License

Using the public domain ZLIB Deflator algorithm, these two functions (DZIP and DUNZIP) losslessly compress or decompress MATLAB variables of most data types so that they occupy less space. Class type and variable size and shape are stored within the compressed data.

NOTES:
(1) The input variable can be a scalar, vector, matrix, or n-dimensional matrix
(2) The input variable must be a non-complex and full (meaning matrices declared as type "sparse" are not allowed)
(4) In testing, DZIP compresses several megabytes of data per second.
(5) In testing, sparsely populated matrices or matrices with regular structure can compress to less than 10% of their original size. The realized compression ratio is heavily dependent on the data. (For example, a large stream of truly random data is theoretically impossible to compress.)
(6) Variables originally occupying very little memory (less than about half of one kilobyte) are handled correctly, but the compression requires some overhead and may actually increase the storage size of such small data sets. One exception to this rule is noted next.
(7) LOGICAL variables are compressed to a small fraction of their original sizes.
(8) The DUNZIP function decompresses the output of this function and restores the original data, including size and class type.
(9) This function uses the public domain ZLIB Deflater algorithm.
(10) Carefully tested, but no warranty; use at your own risk.
(11) Michael Kleder, Nov 2005

Comments and Ratings (17)

Loving this but not quite there. Noob question: to open/read the original archive, which is correct? Thx

d=fread(fid,'int8')?
d=fread(fid,'uint8')?
d=fread(fid,'uint8=>uint8')?

Todd Karin

Jasper

Jasper (view profile)

Works great. Clean code as well.

osama bushnaq

Xiaohu

Xiaohu (view profile)

Charlie

I have data that I believe was compressed using the zlib library (http://zlib.net/).

Using this script I get the error:

Subscript indices must either be real positive integers or logicals.

Error in dunzip (line 37)
    M = typecast(Q,ct{cn});

This error is produced because c at line 23 is not assigned to any value.

I tried making the change suggested by Jesse Hopkins above, but got a different error as follows:
Java exception occurred:
java.io.IOException: Not in GZIP format

at java.util.zip.GZIPInputStream.readHeader(Unknown Source)

at java.util.zip.GZIPInputStream.<init>(Unknown Source)

at java.util.zip.GZIPInputStream.<init>(Unknown Source)

Error in dunzip_ch (line 23)
b=java.util.zip.GZIPInputStream(a);

Do you have any suggestions of what my mistake is?

ali reza

very nice

Fabian

Fabian (view profile)

Is there a way to use these functions to load gzpped ascii-csv-files into Matlab?

Jim Crozier

Jesse Hopkins

Jesse Hopkins (view profile)

Needed to make the following changes to eliminate above java exceptions (in R2008b)

dzip.m line 48: change "DeflaterOutputStream" to "GZIPOutputStream"

dunzip.m line 21: change "InflaterInputStream" to "GZIPInputStream"

Jesse Hopkins

Jesse Hopkins (view profile)

Using R2008b an exception occurs at line 24 of dunzip:
??? Java exception occurred:
java.util.zip.ZipException: unknown compression method

at java.util.zip.InflaterInputStream.read(Unknown Source)

at java.io.FilterInputStream.read(Unknown Source)

at com.mathworks.mlwidgets.io.InterruptibleStreamCopier.copyStream(InterruptibleStreamCopier.java:97)

at com.mathworks.mlwidgets.io.InterruptibleStreamCopier.copyStream(InterruptibleStreamCopier.java:76)

Any thoughts on what needs to be changed?

rajchandar k

hello every one.. how to use this files with Rapid AES algorithm explain with procedure...

thank U......

Zack Voulgaris

An exceptional piece of code. Short, fast and effective.

Matlab User

A simple, easy to use file that made my application not eat up 1.5 gigs+ of memory. Thumbs up.

Pranas Baliuka

Thanks! KISS. Excellent. I wonder why Mathworks did not create as standart e.g. in communication toolbox or even in core ...

Matlab User

Very clever, very simple
Thanks a lot!

Updates

comment clarifications

MATLAB Release
MATLAB 7.0.4 (R14SP2)

Download apps, toolboxes, and other File Exchange content using Add-On Explorer in MATLAB.

» Watch video