Skip to content

Performant hash trees to dedup large collections of files.

License

Notifications You must be signed in to change notification settings

jefdaj/bigtrees

Repository files navigation

BigTrees

A rewrite of gander focusing more on usability of the data structures as a library, rather than on my own "dedup backups" use case.

Quick Start

git clone https://github.com/jefdaj/bigtrees
cd bigtrees

# old way, still works:
nix-shell
stack test

# new way, static build in progress:
nix build

# benchmarking
stack bench --ba --baseline=test/bench/bench.csv --timeout=60s

Done

  • Moved Gander.Cmd -> BigTrees.OldCmd, leaving old commands functional during the rewrite
  • Wrote a meta lint script (hlint, stan, stylish-haskell, weeder) and applied some basic suggestions
  • Some initial work in progress writing haddocs
  • Moved tests into lib/ + app/ alongside the functions they test, wrote more of them
  • Broke HashTree into smaller modules by operation: Build, Write, etc
  • Rewrote my old directory-tree code using a typeclass, started a PR upstream
  • Wrote comparison of text vs binary format file sizes, realized binary is always larger, removed it
  • Added mod time, size (bytes), n files (nodes) to tree data
  • Added header + footer to hashes describing filters, version used, start/end time, table format
  • Rename data structures: Depth, NFiles, NBytes

Todo

  • Static build so it can be used offline without Nix
  • "find mode": list full paths, filter by metadata and glob/regex
  • Rewrite command line interface
  • Add Graft nodes that import other tree files
  • Add Link nodes that indicate whether their target data is present in the tree
  • Add Error nodes to wrap errors, the same way directory-tree does it
  • Intelligent re-hashing of only the files whose mod times have changed
  • Clean up: write haddocks, hide partial constructors, etc
  • Upload to Hackage
  • Example screencasts of using the binary + data structures in repl
bigtrees hash   <src> [-o <tree>]
bigtrees update <tree> [-i <src>]
bigtrees cut    <tree> <branch> [-o <tree>]
bigtrees rm     <tree> <branch>
bigtrees graft  <tree> <branch> [-i <tree>]
bigtrees mv     <tree> <oldbranch> <newbranch>
bigtrees diff   <oldtree> <newtree>
bigtrees dupes  <tree> [<condition>..] [-s <sortby>] [-n <nhits>] [-p <branch>] [-d <script>]

About

Performant hash trees to dedup large collections of files.

Resources

License

Stars

Watchers

Forks

Packages

No packages published