feat!: hardlink deduplications (#291)

* feat!: hardlink deduplications (wip)

* test: multiple hardlinks to a single file

* fix: algorithm

* test: children

* refactor: allow customization

* feat(api): re-export `dashmap`

* feat: report deduplication results

* refactor: use `0.into()` instead

* refactor: just use `default`

* feat(api): ability to cause errors

* feat!: report hardlinks to progress

* feat(api): stop exposing `dashmap`

* feat(api): obscure container type

* docs: fix

* feat(api): newtype for inode numbers

* feat(api): make `Event` `non_exhaustive`

* feat(api): rename a struct

* feat: show shared links before errors

* feat(api): `ProgressReport::shared` should be `Size`

* perf: reduce string resizing

* feat(api): rename some structs

* feat(api): move `HardlinkList` to `hardlink`

* docs: fix

* feat(api): reflection types for hardlink types

* fix(windows): dependencies

* feat(json)!: add optional `shared-inodes`

* feat(api): add some aliases

* docs: correction

* docs: correction

* feat(api): `into_reflection`

* feat(api): `HardlinkList::{len,is_empty}`

* feat(json): output the hardlink record

* chore(git): revert a wrong change

This reverts commit 4efa9b9.

* fix(json): move `"shared-inodes"` out of `"tree"`

* feat(api)!: rename `UnitAndTree` to `JsonDataBody`

* feat(windows): error on unsupported feature

* fix(json): `--json-input` with `--shared-inodes`

* feat(json): rename `"shared-inodes"` to `"shared"`

* refactor: this looks better

* feat(json): always sort `shared` by `ino`

* fix: git merge artifact

* fix: `TEXT_MAX_LEN`

* feat(api)!: remove `hook`

* feat(api): add some aliases

* feat(api): re-export `RecordHardlinksArgument`

* feat(api): create some aliases

* feat(api): expose `aware` and `ignorant`

* feat(api): simplify, eliminate `.clone()`

* refactor: shorten, consistency

* refactor: rename a trait to match its sole method

* refactor: shorten a name

* docs: `HardlinkAware::new`

* refactor: use `SmartDefault`

* feat(api): `DeduplicateSharedSize`

* refactor: move POSIX-exclusive code to its own module

* feat(api): `Reflection::{new,len,is_empty}`

* feat(api): `Reflection::iter`

* feat(api): summarize `HardlinkList`

* fix: compilation conditionals

* feat(api): `Reflection::{len,is_empty}`

* feat(json): `.shared.{details,summary}`

* feat(api): `Default` for `JsonShared`

* fix: report hardlinks

* feat(api): shorten a method name

* test: fix

* feat(json): print hardlinks summary

* fix(json): missing `shared`

* fix: clippy

* feat(json): omit `.shared.{details,summary}`

* feat(cli): forbid `--json-input` + `--deduplicate-hardlinks`

* feat(cli): define some aliases

* chore: generate completions

* refactor: prefer arrays

* docs(cli): make some aliases visible

* refactor: move iterators to its own module

* refactor: move iterator to its own module

* feat(api)!: pass params into `Sub::json_output`

* feat(api): convert `Infallible` to `RuntimeError`

* docs: add some text

* feat(api): make `RecordHardlinks` fallible

* feat(api): move instead of mutate

* feat: output json despite deduplication failures

* refactor: replace qualified with use

* feat: report the number of unrecorded links (wip)

* feat: report the number of unrecorded links

* fix: message

* fix: grammar

* feat: remove extra trailing newline

* feat(api): stop relying on `Into`

* feat(api): remove unneeded impls

* feat(api): better names

* docs: be more specific

* feat(api): revert back to mutations

This reverts commit 8f79037.

* docs: real solutions

* docs: I have decided that I don't care

There's just too many ways to create invalid tree (such as implementing
a custom trait incorrectly for example) that it's not worth the efforts

* fix: windows

* test: fix windows

* fix: windows

* fix: windows

* test: fix windows

* test: fix windows

* test: fix windows

* test: fix windows

* test: fix windows

* chore: remove a lint silencer

* docs: resolve a todo

* feat(api): summary on no hardlinks

* docs: resolve a todo

* feat(api): remove eq traits from `LinkPathList`

The order of items is non-deterministic

* docs: more details

* docs: correction

* test: reflections' equality

* feat: assert links equal

* test: change detection

* fix: windows

* ci(benchmark): add `deduplicate-hardlinks`

* docs(cli): add examples

* docs(cli): consistent wording

* docs(cli): more accurate description

* chore: update completions

* docs(readme): update regarding hardlinks

* fix: windows

* refactor: longer dot chain

* test: prepare some functions

* refactor: word wrap consistently

* test: less confusing names

* feat(api): add missing `Mul<usize>`

* fix(test): create the files

* docs: fix grammar

* test: complex tree

* style: add an empty line

* refactor: rearrange

* test: hardlinks summary

* refactor: use `.into_sorted_by`

* test: hardlinks details

* docs: plan

* refactor: rename a function

* refactor: add `.as_ref`

* test: add some (wip)

* test: sorted

* test: unique inodes

* test: combine

* test: correct setup function

* test: more even number

* test: summary

* test: use `assert_eq`

* test: without deduplication

* test: consistent convention

* test: create fewer files

* refactor: better formula

* docs: remove irrelevant comment

* test: `into_sorted_by_key`

* test: some hardlinks with deduplication

* refactor: remove unnecessary `vec!`

* test: hardlinks summary text

* refactor: shorten the code

* test: some hardlinks without deduplication

* refactor: rearrange

* refactor: split a test file

* refactor: rename a test file

* refactor: remove repeating suffices from test names

* feat(api): immutable setter methods

* chore(deps): add `derive_setters`

* feat(api): immutable setters for `Summary`

* test: exclusive hardlinks only

* test: mixing exclusive-only and external-only

* test: add a missing assertion

* test: external hardlinks only

* refactor: consistent naming

* test: simple tree with some hardlinks

* test: visualization

* style: remove some empty lines

* refactor: remove unnecessary cast

* feat: multiple arguments hardlinks deduplication

* test: multiple arguments hardlinks deduplication

* refactor: rename a test file

* feat: deduplicate arguments

* test: `remove_items_from_vec_by_indices`

* docs: quirk

* test: posix only

* refactor: use mockable trait

* test: add some tests

* test: correct `MockedApi::canonicalize`

* feat: spare symlinks from deduplication

* test: forgot to normalize

* test: fix `MockedApi::is_real_dir`

* test: fix a typo

* test: clear assertion message

* test: fix `resolve_symlink`

* test: add some tests

* style: missing trailing comma

* refactor: move tests to their own files

* test: deduplicate arguments

* chore: apply some suggestions

* style: apply copilot suggestion

* docs(cli): improve flag description

* refactor: use better terms

Author: KSXGitHub

Cargo.lock

@@ -49,6 +49,7 @@ cli = ["clap/derive", "clap_complete", "clap-utilities", "json"]
 cli-completions = ["cli"]
 
 [dependencies]
+dashmap = "^6.1.0"
 pipe-trait = "^0.4.0"
 smart-default = "^0.7.1"
 derive_more = { version = "^2.0.1", features = ["full"] }
@@ -72,5 +73,6 @@ sysinfo = "^0.35.2"
 build-fs-tree = "^0.7.1"
 command-extra = "^1.0.0"
 maplit = "^1.0.2"
+normalize-path = "^0.2.1"
 pretty_assertions = "^1.4.1"
 rand = "^0.9.1"

Cargo.toml

@@ -50,13 +50,14 @@ The benchmark was generated by [a GitHub Workflow](https://github.com/KSXGitHub/
 * Very fast.
 * Relative comparison of separate files.
 * Extensible via the library crate or JSON interface.
+* Unbiased regarding hardlinks: All hardlinks are treated as equally real.
+* Optional hardlink detection and deduplication (would make `pdu` proportionally slower).
 * Optional progress report (would make `pdu` slightly slower).
 * Customize tree depth.
 * Customize chart size.
 
 ## Limitations
 
-* Ignorant of hard links: All hard links are counted as real files.
 * Do not follow symbolic links.
 * Do not differentiate filesystem: Mounted folders are counted as normal folders.
 * The runtime is optimized at the expense of binary size.

README.md

@@ -111,6 +111,17 @@ export const COMPETING_BENCHMARK_MATRIX: readonly CompetingBenchmarkCategory[] =
       ['du', '--count-links'],
     ],
   },
+  {
+    id: 'deduplicate-hardlinks',
+    pduCliArgs: ['--deduplicate-hardlinks'],
+    competitors: [
+      ['dust', '--no-progress'],
+      ['dua'],
+      ['ncdu', '-o', '/dev/stdout', '-0'],
+      ['gdu', '--non-interactive', '--no-progress'],
+      ['du'],
+    ],
+  },
   {
     id: 'top-down',
     pduCliArgs: ['--top-down'],

ci/github-actions/benchmark/matrix.ts

@@ -23,7 +23,7 @@ _pdu() {
 
     case "${cmd}" in
         pdu)
-            opts="-h -V --json-input --json-output --bytes-format --top-down --align-right --quantity --depth --max-depth --width --total-width --column-width --min-ratio --no-sort --no-errors --silent-errors --progress --threads --help --version [FILES]..."
+            opts="-h -V --json-input --json-output --bytes-format --detect-links --dedupe-links --deduplicate-hardlinks --top-down --align-right --quantity --depth --max-depth --width --total-width --column-width --min-ratio --no-sort --no-errors --silent-errors --progress --threads --omit-json-shared-details --omit-json-shared-summary --help --version [FILES]..."
             if [[ ${cur} == -* || ${COMP_CWORD} -eq 1 ]] ; then
                 COMPREPLY=( $(compgen -W "${opts}" -- "${cur}") )
                 return 0

exports/completion.bash

@@ -29,12 +29,17 @@ set edit:completion:arg-completer[pdu] = {|@words|
             cand --threads 'Set the maximum number of threads to spawn. Could be either "auto", "max", or a number'
             cand --json-input 'Read JSON data from stdin'
             cand --json-output 'Print JSON data instead of an ASCII chart'
+            cand --deduplicate-hardlinks 'Detect and subtract the sizes of hardlinks from their parent directory totals'
+            cand --detect-links 'Detect and subtract the sizes of hardlinks from their parent directory totals'
+            cand --dedupe-links 'Detect and subtract the sizes of hardlinks from their parent directory totals'
             cand --top-down 'Print the tree top-down instead of bottom-up'
             cand --align-right 'Set the root of the bars to the right'
             cand --no-sort 'Do not sort the branches in the tree'
             cand --silent-errors 'Prevent filesystem error messages from appearing in stderr'
             cand --no-errors 'Prevent filesystem error messages from appearing in stderr'
             cand --progress 'Report progress being made at the expense of performance'
+            cand --omit-json-shared-details 'Do not output `.shared.details` in the JSON output'
+            cand --omit-json-shared-summary 'Do not output `.shared.summary` in the JSON output'
             cand -h 'Print help (see more with ''--help'')'
             cand --help 'Print help (see more with ''--help'')'
             cand -V 'Print version'

exports/completion.elv

@@ -11,10 +11,13 @@ complete -c pdu -l min-ratio -d 'Minimal size proportion required to appear' -r
 complete -c pdu -l threads -d 'Set the maximum number of threads to spawn. Could be either "auto", "max", or a number' -r
 complete -c pdu -l json-input -d 'Read JSON data from stdin'
 complete -c pdu -l json-output -d 'Print JSON data instead of an ASCII chart'
+complete -c pdu -l deduplicate-hardlinks -l detect-links -l dedupe-links -d 'Detect and subtract the sizes of hardlinks from their parent directory totals'
 complete -c pdu -l top-down -d 'Print the tree top-down instead of bottom-up'
 complete -c pdu -l align-right -d 'Set the root of the bars to the right'
 complete -c pdu -l no-sort -d 'Do not sort the branches in the tree'
 complete -c pdu -l silent-errors -l no-errors -d 'Prevent filesystem error messages from appearing in stderr'
 complete -c pdu -l progress -d 'Report progress being made at the expense of performance'
+complete -c pdu -l omit-json-shared-details -d 'Do not output `.shared.details` in the JSON output'
+complete -c pdu -l omit-json-shared-summary -d 'Do not output `.shared.summary` in the JSON output'
 complete -c pdu -s h -l help -d 'Print help (see more with \'--help\')'
 complete -c pdu -s V -l version -d 'Print version'

exports/completion.fish

@@ -32,12 +32,17 @@ Register-ArgumentCompleter -Native -CommandName 'pdu' -ScriptBlock {
             [CompletionResult]::new('--threads', '--threads', [CompletionResultType]::ParameterName, 'Set the maximum number of threads to spawn. Could be either "auto", "max", or a number')
             [CompletionResult]::new('--json-input', '--json-input', [CompletionResultType]::ParameterName, 'Read JSON data from stdin')
             [CompletionResult]::new('--json-output', '--json-output', [CompletionResultType]::ParameterName, 'Print JSON data instead of an ASCII chart')
+            [CompletionResult]::new('--deduplicate-hardlinks', '--deduplicate-hardlinks', [CompletionResultType]::ParameterName, 'Detect and subtract the sizes of hardlinks from their parent directory totals')
+            [CompletionResult]::new('--detect-links', '--detect-links', [CompletionResultType]::ParameterName, 'Detect and subtract the sizes of hardlinks from their parent directory totals')
+            [CompletionResult]::new('--dedupe-links', '--dedupe-links', [CompletionResultType]::ParameterName, 'Detect and subtract the sizes of hardlinks from their parent directory totals')
             [CompletionResult]::new('--top-down', '--top-down', [CompletionResultType]::ParameterName, 'Print the tree top-down instead of bottom-up')
             [CompletionResult]::new('--align-right', '--align-right', [CompletionResultType]::ParameterName, 'Set the root of the bars to the right')
             [CompletionResult]::new('--no-sort', '--no-sort', [CompletionResultType]::ParameterName, 'Do not sort the branches in the tree')
             [CompletionResult]::new('--silent-errors', '--silent-errors', [CompletionResultType]::ParameterName, 'Prevent filesystem error messages from appearing in stderr')
             [CompletionResult]::new('--no-errors', '--no-errors', [CompletionResultType]::ParameterName, 'Prevent filesystem error messages from appearing in stderr')
             [CompletionResult]::new('--progress', '--progress', [CompletionResultType]::ParameterName, 'Report progress being made at the expense of performance')
+            [CompletionResult]::new('--omit-json-shared-details', '--omit-json-shared-details', [CompletionResultType]::ParameterName, 'Do not output `.shared.details` in the JSON output')
+            [CompletionResult]::new('--omit-json-shared-summary', '--omit-json-shared-summary', [CompletionResultType]::ParameterName, 'Do not output `.shared.summary` in the JSON output')
             [CompletionResult]::new('-h', '-h', [CompletionResultType]::ParameterName, 'Print help (see more with ''--help'')')
             [CompletionResult]::new('--help', '--help', [CompletionResultType]::ParameterName, 'Print help (see more with ''--help'')')
             [CompletionResult]::new('-V', '-V ', [CompletionResultType]::ParameterName, 'Print version')

exports/completion.ps1

@@ -28,14 +28,19 @@ block-count\:"Count numbers of blocks"))' \
 '*--column-width=[Maximum widths of the tree column and width of the bar column]:TREE_WIDTH:_default:TREE_WIDTH:_default' \
 '--min-ratio=[Minimal size proportion required to appear]:MIN_RATIO:_default' \
 '--threads=[Set the maximum number of threads to spawn. Could be either "auto", "max", or a number]:THREADS:_default' \
-'(--quantity)--json-input[Read JSON data from stdin]' \
+'(--quantity --deduplicate-hardlinks)--json-input[Read JSON data from stdin]' \
 '--json-output[Print JSON data instead of an ASCII chart]' \
+'--deduplicate-hardlinks[Detect and subtract the sizes of hardlinks from their parent directory totals]' \
+'--detect-links[Detect and subtract the sizes of hardlinks from their parent directory totals]' \
+'--dedupe-links[Detect and subtract the sizes of hardlinks from their parent directory totals]' \
 '--top-down[Print the tree top-down instead of bottom-up]' \
 '--align-right[Set the root of the bars to the right]' \
 '--no-sort[Do not sort the branches in the tree]' \
 '--silent-errors[Prevent filesystem error messages from appearing in stderr]' \
 '--no-errors[Prevent filesystem error messages from appearing in stderr]' \
 '--progress[Report progress being made at the expense of performance]' \
+'--omit-json-shared-details[Do not output \`.shared.details\` in the JSON output]' \
+'--omit-json-shared-summary[Do not output \`.shared.summary\` in the JSON output]' \
 '-h[Print help (see more with '\''--help'\'')]' \
 '--help[Print help (see more with '\''--help'\'')]' \
 '-V[Print version]' \