[Perl] Duplicate File Finder - MD5 hash compare method (Subdirectory Recursive)

ArkPhaze · 11-27-2011, 07:35 AM

Here's a script I created some time ago in perl which will compare all files within the same location of the script to check for any file duplicates using an md5 checksum compare method. It's recursive, meaning it will find duplicates between different directories as well inside the parent directory that you run the script from. So put the script where you would like to scan, and run it from there, otherwise I can add a method to input a directory to scan in the future, it's honestly a minute edit to make that happen.

Code:
#!/usr/bin/perl
use strict;
use File::Find;
use Digest::MD5;

my %files;
my $wasted = 0;
find(\&f_check, $ARGV[0] || ".");

local $" = "\n";
foreach my $size (sort {$b <=> $a} keys %files) {
  next unless @{$files{$size}} > 1;
  my %md5;
  foreach my $file (@{$files{$size}}) {
    open(FILE, $file) or next;
    binmode(FILE);
    push @{$md5{Digest::MD5->new->addfile(*FILE)->hexdigest}},$file;
  }
  foreach my $hash (keys %md5) {
    next unless @{$md5{$hash}} > 1;
	print "\n";
	print "\n";
    print "($size bytes) Duplicate Files: [MD5 = $hash]\n";
	print "@{$md5{$hash}}\n";
	print "\n";
    $wasted += $size * (@{$md5{$hash}} - 1);
  }
}

1 while $wasted =~ s/^([-+]?\d+)(\d{3})/$1,$2/;
print "\n";
print "----------------------------------------------------\n";
print "                                                    \n";
print "  You have $wasted bytes total in duplicate files   \n";
print "                                                    \n";
print "----------------------------------------------------\n";
print "\n";

sub f_check {
  -f && push @{$files{(stat(_))[7]}}, $File::Find::name;
}

What we're doing here is creating a hash array %files, and sorting it by byte size with $b <=> $a so that we compare files of similar file sizes ONLY when we compare each MD5 checksum. This will make the script a lot faster because now we don't compare every file to every file...

The reason for this is, if the filesize is differen't, then you're not going to have the same hash, and it's probably not the same file. But we compare by hash obviously because filenames would be pointless. You can't have a duplicate filename with same extensions in a directory, windows won't allow it, and it would be easy enough to locate duplicates manually that way too if that was your method.