A high-performance image duplicate detection CLI tool built with TypeScript. Scans directories for duplicate images and safely moves them to a duplicate folder.

日本語版 README はこちら

• 🚀 Standalone Executable: Single-file executable without Node.js dependency (~397KB)
• 🔍 High-Precision Detection: Reliable duplicate detection using hash-based + byte-level comparison
• 🎯 Multiple Image Format Support: JPG, JPEG, PNG, GIF, BMP, WebP, TIFF
• 🛡️ Safe Operations: Dry-run mode, filename collision avoidance, error handling
• ⚡ High Performance: Caching, memory optimization, large file support
• 🧪 Quality Assurance: TypeScript, ESLint, comprehensive tests (42 tests), CI/CD ready
• 🔧 Robustness: Automatic fallback functionality for Sharp dependencies

# Clone repository
git clone <repository-url>
cd dupf

# Install dependencies
pnpm install

# Build standalone executable
pnpm build:standalone

# Basic usage
./dupf /path/to/images

# Dry run (show what would be moved without actually moving)
./dupf /path/to/images --dry-run --verbose

# Custom duplicate folder name
./dupf /path/to/images --output-dir duplicates

# Show help
./dupf --help

# Build only
pnpm install
pnpm build:standalone

# Copy executable to other machines
cp ./standalone/dupf /usr/local/bin/  # For system-wide use

# Development mode
pnpm dev /path/to/images

# Run after TypeScript build
pnpm build
node dist/bin/dupf.js /path/to/images

# Run on test image folder
./standalone/dupf ./test-images --dry-run --verbose

# Output example:
# Scanning directory: /home/user/test-images
# Duplicate folder: duplicate
# Dry run: Yes
#
# Found 10 image files.
# Comparing images for duplicates...
# Found 2 duplicate image(s).
#
# Duplicate found:
#   Original: /home/user/test-images/photo1.jpg
#   Duplicate: /home/user/test-images/photo1_copy.jpg
#   Would move to: /home/user/test-images/duplicate/photo1_copy.jpg
#
# Dry run completed. 2 duplicate(s) would be moved.

1. File Discovery: Recursively scan target directory
2. Hash Generation:
   • Primary method: 8x8 grayscale image MD5 hash using Sharp (high precision)
   • Fallback method: File-based MD5 hash for Sharp-incompatible environments
3. Fast Comparison: Initial screening using hash-based comparison
4. Strict Verification: Byte-level file content comparison
5. Safe Moving: File movement with collision avoidance functionality

• JPEG (.jpg, .jpeg)
• PNG (.png)
• GIF (.gif)
• BMP (.bmp)
• WebP (.webp)
• TIFF (.tiff, .tif)

• Memory Efficiency: Low memory usage through streaming processing
• Cache Functionality: Memory cache for image hashes
• Large File Support: Efficient processing of tens of thousands of images

# Install dependencies
pnpm install

# Run in development mode
pnpm dev /path/to/images

# TypeScript build
pnpm build

# Run tests
pnpm test:run

# Code quality checks
pnpm lint
pnpm format

# Run all tests
pnpm test:run

# Watch mode
pnpm test

# Test UI
pnpm test:ui

# Generate test images (required before first test run) - runs TypeScript directly
pnpm test:setup

# Complete test workflow (setup + test execution) - faster
pnpm test:full

# Integration tests only
pnpm test tests/integration

# Unit tests only
pnpm test tests/unit

• Single File: ./standalone/dupf
• No Dependencies: No Node.js required, runs immediately
• Portable: Can be copied and run on any machine
• Simple: No installation or configuration required
• Robustness: Automatic fallback functionality for Sharp-incompatible environments

# Build
pnpm build:standalone

# Copy file to target machine
scp ./standalone/dupf user@target:/usr/local/bin/

# Verify execution permissions
chmod +x /usr/local/bin/dupf

Q: Permission denied error occurs

chmod +x ./standalone/dupf

Q: Memory shortage with large number of files

# Split directories to process one at a time
./standalone/dupf /path/to/photos/2023 --verbose
./standalone/dupf /path/to/photos/2024 --verbose

Q: Sharp-related errors occur

• The tool automatically uses fallback functionality
• When Sharp is unavailable, it switches to file-based hashing
• Warning messages will be displayed, but functionality is not affected

Q: Specific image formats are not detected

• Check supported formats: JPG, JPEG, PNG, GIF, BMP, WebP, TIFF
• Verify file extensions are correct