deduped always 0 when deduped??

[i00]

When I run it using:

dduper --device /dev/sdb1 --dir /Databases/_/BNE/

files appear to be deduped (used space goes down) ... but ...
the output is always "deduped: 0" (see below)

[Lakshmipathi]

That's strange. Are you using some type of btrfs RAID?
Can you share that output for dduper --device /dev/sdb1 --files /path/to/backup1.bak /path/to/backup2.bak --dry-run ?

[i00]

Just standard BTRFS (no raid)

The Dry-run:

... Also the size does not appear to go down by the amount that it should (basically most of the file is the same).
... I should probably also mention that the command without the dry run takes AGES to run (... over 10 mins!)
The run:

[Lakshmipathi]

Thanks dry-run seems to indicate we have 1192600 KB (approx 11.03 GB) duplicate data and both files are around 11.05 GB. Is that correct? But I still can't figure out why this shows deduped: 0 . Please let me know following details:

1. How did you install dduper ?
2. Can you share output for uname -a and dduper --version

Also the size does not appear to go down by the amount that it should

What's the disk usage before and after you ran dduper command?

that the command without the dry run takes AGES to run (... over 10 mins!)

Its because we are using 32kb chunk size for 11GB file. So it does lot of syscalls, can you try increasing chunk size to 1M ?
dduper --device /dev/sdb1 --files /path/to/backup1.bak /path/to/backup2.bak --chunk-size 1024

Also you can run --analyze option to decide which chunk size gets optimal disk saving and performance.
dduper --device /dev/sdb1 --files /path/to/backup1.bak /path/to/backup2.bak --analyze https://github.com/Lakshmipathi/dduper#analyze-with-different-chunk-size

PS: Please use this suggestion with care: If you already have a backup of these files in other system, you may want to checkout --fast-mode option it is faster than default mode. https://github.com/Lakshmipathi/dduper#dedupe-files-faster-fast-mode

[i00]

dduper is installed in a docker container from laks\dduper

It's hard to tell the space reclaimed ... since this server gets use from others ... but the volume usage only seemed to drop by ~100MB

[Lakshmipathi]

Thanks for the details. I didn't update the docker image with recent changes. Let me push of those change to docker and re-create the image and ping here. After that you can pull the latest image and give it a try.

[Lakshmipathi]

I was testing the issue without docker image - let me test there and reproduce the issue too

[Lakshmipathi]

This is a bug, I managed to reproduce the issue. Fixed it (hopefully :P) - currently running some basic tests

[Lakshmipathi]

@i00 , Changes seems to be working, please pull the latest docker image and give it a try.

[i00]

Ok ... thanks ... appears to be working now.
Thanks ... :)

Just wondering why when I run it now it takes ~399 seconds the first time (2 files) ... then when I re-run it it takes ~274 seconds ...
The --dry-run on these takes ~1 second ... can it not tell that it has already been deduped?

[i00]

Also...
I ran the dedupe 2x (below), then dry-ran it...

1. Why did the matched / unmatched chunks change between one run and the next?
2. Why did the 2nd run dedupe another ~54MB?

Thanks

[i00]

Also...
Just copied 2 files (Total size: 24,887,927,808 bytes)
Then deduped:

Then checked the space on the drive (268.9 GB free in System Monitor)
Then deleted both files (246.9 GB free in System Monitor)
... the space was reduced by ~22GBs? ... how is this when almost 12GB of the file was dedupped?

Thanks

[Lakshmipathi]

. can it not tell that it has already been deduped?

dduper doesn't maintain any states on-disk (like storing details in db).
I plan to add this check in upcoming days.

Why did the matched / unmatched chunks change between one run and the next?

Can you try something like this:

run-1

sleep 10 && sync && sleep 10

run-2

and verify the results.

... the space was reduced by ~22GBs? ... how is this when almost 12GB of the file was dedupped?

Sorry, I didn't understand this issue. could you please provide psudeo-code of it?
thanks

[Lakshmipathi]

filefrag -e /path/to/file should display extents which are shared or not.

#### 2mb file with extents 

$ filefrag -e /mnt/fn_aa_1_2mb 
Filesystem type is: 9123683e
File size of /mnt/fn_aa_1_2mb is 2097152 (512 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..     511:      35888..     36399:    512:             last,eof
/mnt/fn_aa_1_2mb: 1 extent found

#### perform dedupe on 2mb file

$  sudo ./dduper --device /dev/loop12 --files /mnt/fn_a_1_1mb /mnt/fn_aa_1_2mb --chunk-size 1024


#### now filefrag shows it has 2 extents and both of them are `shared` 

$ filefrag -e /mnt/fn_aa_1_2mb 
Filesystem type is: 9123683e
File size of /mnt/fn_aa_1_2mb is 2097152 (512 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..     255:       4896..      5151:    256:             shared
   1:      256..     511:       4896..      5151:    256:       5152: last,shared,eof
/mnt/fn_aa_1_2mb: 2 extents found

[Lakshmipathi]

Another simple option:

btrfs fi du /path/to/deduped files

below says 30MB deduped/shared content on the file.

btrfs fi du /mnt/fn_xbyczd_10_60mb 
     Total   Exclusive  Set shared  Filename
  60.00MiB    30.00MiB    30.00MiB  /mnt/fn_xbyczd_10_60mb

[i00]

Hey sorry; project has been on hold for a while; hope to investigate this further next week.

[Lakshmipathi]

Sure no problem.