fdf

README

File Duplicate Finder (fdf)

A cross-platform duplicate file finder supporting deduplication via copy-on-write clones and hard links. Inspired by Olof Laderkvist's Windows-only fdf utility.

Installation

go install github.com/josephvusich/fdf@latest

System Requirements

• Go (with CGO support enabled on non-Windows platforms)
• One of the following platforms:

Platform	Minimum version	Reason
Linux	2.6.33+	File-to-file sendfile() support
Mac OS X	Sierra 10.12+	clonefile() and APFS support
Windows	Windows 10 or Windows Server 2016+	FSCTL_DUPLICATE_EXTENTS_TO_FILE support

Usage

usage: fdf [--clone | --copy | --delete | --link] [-hqrtv]
        [-m FIELDS] [-z BYTES] [-n LENGTH]
        [--protect PATTERN] [--unprotect PATTERN] [directory ...]

  -a, --clone               (verb) create copy-on-write clones instead of hardlinks (not supported on all filesystems)
  -c, --copy                (verb) split existing hardlinks via copy
                            mutually exclusive with --ignore-hardlinks
      --copy-unlinked       always copy over matching files even if not hardlinked
  -d, --delete              (verb) delete duplicate files
  -t, --dry-run             don't actually do anything, just show what would be done
      --exclude GLOB        exclude files matching GLOB from scanning
      --exclude-dir DIR     exclude DIR from scanning, throws error if DIR does not exist
      --help                show this help screen and exit
      --if-kept GLOB        only remove files if the 'kept' file matches the provided GLOB
      --if-kept-dir DIR     only remove files if the 'kept' file is a descendant of DIR
      --ignore-content      allow --match without 'content'
  -h, --ignore-hardlinks    ignore existing hardlinks
                            mutually exclusive with --copy
      --include GLOB        include GLOB, opposite of --exclude
      --include-dir DIR     include DIR, throws error if DIR does not exist
      --json-report FILE    on completion, dump JSON match data to FILE
  -l, --link                (verb) hardlink duplicate files
  -m, --match FIELDS        Evaluate FIELDS to determine file equality, where valid fields are:
                              name (case insensitive)
                                range notation supported: name[offset:len,offset:len,...]
                                  name[0:-1] whole string
                                  name[0:-2] all except last character
                                  name[1:2]  second and third characters
                                  name[-1:1] last character
                                  name[-3:3] last 3 characters
                              copyname (case insensitive)
                                'foo.bar' == 'foo (1).bar' == 'Copy of foo.bar', also requires +size or +content
                              namesuffix (case insensitive)
                                one filename must end with the other, e.g.: 'foo-1.bar' and '1.bar'
                              nameprefix (case insensitive)
                                one filename must begin with the other, e.g., 'foo-1.bar' and 'foo.bar'
                              parent (case insensitive name of immediate parent directory)
                                range notation supported: see 'name' for examples
                              path
                                match parent directory path
                              relpath
                                match parent directory path relative to input dir(s)
                              size
                              content (default, also implies size)
                            specify multiple fields using '+', e.g.: name+content
  -z, --minimum-size BYTES  skip files smaller than BYTES, must be greater than the sum of --skip-header and --skip-footer (default 1)
      --preserve PATTERN    (deprecated) alias for --protect PATTERN
  -p, --protect PATTERN     prevent files matching glob PATTERN from being modified or deleted
                            may appear more than once to support multiple patterns
                            rules are applied in the order specified
      --protect-dir DIR     similar to --protect 'DIR/**/*', but throws error if DIR does not exist
  -q, --quiet               don't display current filename during scanning
  -r, --recursive           traverse subdirectories
      --skip-footer LENGTH  skip LENGTH bytes at the end of each file when comparing
  -n, --skip-header LENGTH  skip LENGTH bytes at the beginning of each file when comparing
      --timestamps MODE     MODE must be one of ignore, prefer-newer, prefer-older (default "prefer-older")
      --unprotect value     remove files added by --protect
                            may appear more than once
                            rules are applied in the order specified
      --unprotect-dir DIR   similar to --unprotect 'DIR/**/*', but throws error if DIR does not exist
  -v, --verbose             display additional details regarding protected paths

Copy-on-write Cloning

The --clone flag enables copy-on-write clones on compatible filesystems. Common filesystems with support include APFS, ReFS, and Btrfs. See Comparison of file systems on Wikipedia for more. Note that --copy may also create clones when using Mac OS X with an APFS filesystem.

License

Licensed under the Apache 2.0 license.

• clonefile_windows.go is adapted from git-lfs, licensed under the MIT license.
• path_windows.go is adapted from the Go standard library, licensed under a BSD-style license.