FINDDUPE: Duplicate file detector and eliminator

FINDDUPE: Duplicate file detector and eliminator

Version 1.23   August 25 2010

Finddupe is a tool for quick detecting of duplicate files on a hard drive under Windows. Duplicate files can be detected, hardlinked, or deleted. Finddupe can also search for hardlinked files on your computer.

Finddupe is a command line program. If you aren't familiar with the "command prompt" under windows, this program might be a little challenging for you to use.

Finddupe has several possible uses:

Deleting duplicate files
When working thru somebody else's photo collection, or MP3 collection, this tool is useful for deleting the files that are duplicate. Depending on how the media is organized, there can be a lot of duplicate files in a collection.

Freeing hard drive space
Sometimes its intentional to have certain media in multiple places. By running finddupe, and hard linking the identical files, you can keep the files in multiple places, while only having one physical copy on the hard drive.

Detecting changed files for backup
Finddupe is useful for detecting which files have changed and need backing up. Simply back up the media, and then run finddupe to eliminate those files in the copy that are already contained in a previous backup.

Detecting hardlinked groups of files
Unknown to most, the winodws NTFS file system has the ability to hard link files together. If you use finddupe with the -listlink option, finddupe will search for files that appear in multiple places as hard links.

finddupe command line options

finddupe [options] [-ref] <filepat> [filepat]...

Example uses

If you have a previous backup in a directory tree on c:\prev_backup, and just copied your work files to a directory tree on c:\new_backup, you can remove any files that are already in the previous backup with the following incantation:
    finddupe -del -ref c:\prev_backup c:\new_backup


If you have a large photo collection on c:\photos, and you wish to replace duplicates with hard links, you can run:

    finddupe -hardlink c:\photos
Note that this only works on NTFS file systems (such as the C drive under Windows XP). It won't work on FAT file systems, like the ones used on most external hard disks or USB flash drives.

If you want to know which files within a directory tree are hardlinked together, such as after running the above command, you can run:

    finddupe -listlink c:\photos

If you just want to know which files are common between two directory trees, you can run:
    finddupe -bat work.bat -del c:\media\** c:\media2\**
This will create the file "work bat" with file delete commands in it. The '-bat' option tells finddupe to not do anything, but rather store the actions to a batch file. This allows you to review what finddupe would do before taking any action. The '**' tells it to recursively do all the files.

"Screenshot" - finddupe looks like while running:

Command Prompt - finddupe
C:\>finddupe testdir\*
Duplicate: 'testdir\aab.txt'
With:      'testdir\aab.zzz'
Duplicate: 'testdir\aab.bak'
With:      'testdir\aac.txt'
Duplicate: 'testdir\dup1'
With:      'testdir\foo2'
Duplicate: 'testdir\foo'
With:      'testdir\makefile'
Duplicate: 'testdir\foo'
With:      'testdir\makefile.bak'
Duplicate: 'testdir\dup1'
With:      'testdir\myglob.bak'
Duplicate: 'testdir\longdiff.bak'
With:      'testdir\nadine.txt'
Files:    23285 kBytes in    23 files
Dupes:     6971 kBytes in     7 files

Compatibility

Finddupe has been tested on Windows 2000, XP and Vista. Hard linking does not work on Windows versions prior to Windows 2000.

Why I wrote this program

I wanted to eliminate some duplicate files on my windows computer. Naturally, I searched the internet. But mostly, I could just find fancy payware, whereas all I wanted was a really simple command line based utility. So I eventually wrote one.

I also wrote it to be very fast. For large media files, this helps a lot. Finddupe will only read the first 32k of a file and compute a hash based on that. Only if that matches with another file will it even read the entire files. I use it mostly on various media, like jpegs and mp3s to find and eliminate duplicates I may have

License

Finddupe is totally free. Do whatever you like with it. You can integrate it into GPL or BSD style licensed programs if you would like to.

Bugs

Presently, finddupe does not check for NTFS file systems before attempting to hard link. If you run it on a non NTFS file system, it will stop on the first failed hardlink attempt, but not before deleting the file it meant to replace with a hardlink.

On modifying one file of a hardlinked pair or set, windows may not update the size of the other linked instances right away. Also, programs that move the old file to a backup copy tend to break (unlink) the hard link on saving. This behaviour may or may not be desirable.

Downloads

Got questions? Email me: The address is in the PNG file so no robot can pick it up

Other handy free utilities by Matthias Wandel:


To Matthias Wandel's home page