File Exchange

image thumbnail

Directory Traversal and Duplicate File Deletion using the SHA-256 Hash

version 1.0.0.0 (23.4 KB) by Michael Kleder
On Windows systems, finds duplicate files in a directory tree using SHA-256, then rapidly deletes.

0 Downloads

Updated 03 Nov 2005

No License

PURGETREE - On Windows systems, rapidly identifies duplicate files in an entire directory tree using the SHA-256 hash algorithm, then creates and executes a batch file to rapidly delete all but one of the duplicates.

USAGE:

cd('dirname'); purgetree
OR
cd('dirname');purgetree('displayonly')

dirname = The name of the highest ("parent") directory in the directory tree within which you wish to purge duplicate files.

Notes:

This function rapidly compares large numbers of files for identical content by computing the SHA-256 hash of each file and detecting duplicates. The probablility of two non-identical files having the same hash, even in a hypothetical directory containing millions of files, is exceedingly remote. Since hashes rather than file contents are compared, the process of detecting duplicates is greatly accelerated.

If PURGETREE is called with any arguments, for example purgetree('displayonly') or simply purgetree(1), then the file deletion commands are displayed in the Matlab window but NOT executed (i.e., no files are deleted.)

You must have the freeware file sha256deep.exe on your Matlab path to use this function. The sha256deep.exe file is included in the download, but is also available as part of the md5deep family of functions at:
http://md5deep.sourceforge.net/
The SHA-256 hash algorithm was developed by the National Institute of Standards and Technology (NIST) and is described at:
http://csrc.nist.gov/publications/fips/fips180-2/fips180-2.pdf

When duplicate files are found, the file that has the first alphabetically sorted name is retained; others are deleted.

Carefully tested, but no warranty; use at your own risk.

Michael Kleder, Sep 2005

Cite As

Michael Kleder (2020). Directory Traversal and Duplicate File Deletion using the SHA-256 Hash (https://www.mathworks.com/matlabcentral/fileexchange/8495-directory-traversal-and-duplicate-file-deletion-using-the-sha-256-hash), MATLAB Central File Exchange. Retrieved .

Comments and Ratings (2)

The Author

Thank you for your comments. The SHA-256 hash is slower than the MD5 hash. At twice the bit length, it is (to date) untarnished by a collision "break" of the kind reported by Wang et al in August 2004. (See article below.) I agree that return statements at the end of functions may be unnecessary, but I try to maintain the habit of including them for code clarity and to avoid problems with functions sometimes being interpreted as nested functions when I didn't intend them to be. (It may not be the tightest possible coding style, but I don't think it slows down the execution and for me it helps with readability.)
Cheers, Mike.
http://eprint.iacr.org/2004/199.pdf

A. Belani

-The 'return' statement at the end of each of the subfunctions seem unnecessary.
-Please document whether the SHA-256 hash offer any speed improvement over MD5.

Updates

1.0.0.0

Comment clarification.

MATLAB Release Compatibility
Created with R14SP2
Compatible with any release
Platform Compatibility
Windows macOS Linux
Acknowledgements

Inspired by: Using the MD5 Hash for Duplicate File Deletion