Feature request: zero-detection

[tkn777]

Some or most of the device, I synchronize with blocksync-fast is empty space.

Therefore it would be great, if blocksync-fast could detect this empty blocks and writes some kind of null-pointer (ref-links) to destination for those empty blocks.

[nethappen]

Hello,
thank you for your interest in the blocksync-fast program.

Your request looks very interesting, but I need to better understand your use case and how you are using the program.

Key question:
Where do you store the destination copies? Are they image files in a Copy-On-Write filesystem (e.g., BTRFS or similar), or do you copy data directly to a block device?

I) If you are using files in a Copy-On-Write filesystem:
You can handle this as follows:

Create an empty file matching the size of the source device:
For example, if your /dev/sda1 is 100 GB:

$ truncate -s 100G dst.img
The file dst.img will be empty and will automatically leverage deduplication for zero blocks, thanks to the Copy-On-Write features.

Synchronize the data using blocksync-fast:

$ blocksync-fast -s /dev/sda1 -d dst.img
As a result, the file dst.img will store only the actual data, while empty blocks will be represented as "empty space."

Verify the result:
You can check the actual data usage in the file with:
$ sync; btrfs fi du dst.img

II) If the destination system does not support Copy-On-Write:
You can store only the actual data (without empty blocks), but in this case, the result will not be an exact copy of the source device but rather a Delta file. Recreating the original data would require using blocksync-fast again.

Example:

Create an empty file matching the size of the source device:
$ truncate -s 100G zero.file

Generate a digest file for the empty file:
$ blocksync-fast -s zero.file --make-digest -f zero.digest

Create a Delta file with the actual data:
$ blocksync-fast --make-delta -s /dev/sda1 -D sda1_only_real_data.delta
As a result, the file sda1_only_real_data.delta will store only blocks with actual data (non-zero).

Additional note:
Currently, the program does not yet support the simplified command:
$ blocksync-fast -s /dev/sda1 -d /dev/zero.file --make-delta -D sda1_only_real_data.delta
I consider this a bug to fix in the next release.

Both I and II methods can be improved or automated in future versions of the program. I’d appreciate it if you could provide a more detailed description of your use case, particularly about the source and destination data storage.

[ollivers]

Hello nethappen,

discussed this issue with tkn777 - we both use blocksync-fast to copy backup images to XFS filesystems in conjunction with reflink versioning.

The zero detection question was only related to the first initial run of blocksync-fast. In case there is no sparse file in place, a non-sparse file (zero filled) seems to be created by blocksync-fast during initial run when using XFS for storing image files.

We just added the truncate part to our backup scripts and it works as described. I highly appreciate your effort in providing this great tool.

Regards,
Oliver

[nethappen]

Thank you for your feedback and additional details about how you use the blocksync-fast tool.
The proposed solution is temporary — it works but may lead to performance loss because empty blocks from the destination device are still being read and compared with the source.
In the next version, I plan to add an option/attribute to skip empty blocks (0x000) or possibly any user-defined blocks.
I will also consider how to handle subsequent synchronizations so that empty blocks being overwritten in places with data are deduplicated in the same way as during the initial synchronization.