diskrsync

README

diskrsync -- rsync for block devices and disk images

This is a utility for remote backup of disk image files or devices. It uses ssh as a transport and tries to minimise the amount of data transferred. This is done by building a Merkle tree of blake2 hashes on both the source and the target hosts and then traversing the tree only transmitting those blocks where the hashes differ.

It is important that the source file does not change during the process. Typically this would be achieved by creating an LVM snapshot or by freezing the VM if that's not possible.

By default the resulting file is compressed using the spgz library (this can be disabled by using --no-compress flag). Note this only works on filesystems that support punching holes in files like xfs or ext4.

The utility handles sparse files (or just files with a lot of zeros) efficiently. The resulting file will be sparse (even if not compressed).

Size changes are also supported (both shrinks and expansions).

Installation

1. Install go. Version 1.16 is minimum required but the newer the better. If your distribution lacks the required version, check backports or updates (e.g. for debian or ubuntu) Alternatively, install manually.

2. Run the following commands:

mkdir workspace
cd workspace
GOPATH=$(pwd) go install github.com/dop251/diskrsync/diskrsync@latest
sudo cp -a bin/diskrsync /usr/local/bin

1. Make sure the binary is copied to the remote machine as well. If the remote machine has a different CPU or OS you may want to use cross-compilation. For example if you are want to build a binary for ARM:

GOPATH=$(pwd) GOARCH=arm go install github.com/dop251/diskrsync/diskrsync@latest
ls -l bin/linux_arm/diskrsync

Usage examples

diskrsync /dev/vg00/lv_snap ruser@backuphost:/mnt/backup/disk

This ensures that /mnt/backup/disk is up-to-date with the LV snapshot. The file will be compressed using spgz and can be recovered using the following command:

spgz -x /mnt/backup/disk /dev/vg00/....

diskrsync --verbose --no-compress --ssh-flags="-i id_file" /var/lib/libvirt/images/disk.img ruser@rbackuphost:/mnt/backup/

This ensures that /mnt/backup/disk.img is up-to-date with the source file.

Documentation

Index

• Constants
• Variables
• func Source(reader io.ReadSeeker, size int64, cmdReader io.Reader, cmdWriter io.Writer, ...) (err error)
• func Target(writer spgz.SparseFile, size int64, cmdReader io.Reader, cmdWriter io.Writer, ...) (err error)
• type CountingReader
  • func (c *CountingReader) Count() int64
  • func (r *CountingReader) Read(buf []byte) (n int, err error)
• type CountingWriteCloser
  • func (c *CountingWriteCloser) Count() int64
  • func (r *CountingWriteCloser) Write(buf []byte) (n int, err error)
• type FixingSpgzFileWrapper
  • func (rw *FixingSpgzFileWrapper) Read(p []byte) (n int, err error)
  • func (rw *FixingSpgzFileWrapper) Seek(offset int64, whence int) (int64, error)
• type ProgressListener
• type TargetFile

Constants

const (
	DefTargetBlockSize = 128 * 1024
)

Variables

var (
	ErrInvalidFormat = errors.New("invalid data format")
)

Functions

func Source

func Source(reader io.ReadSeeker, size int64, cmdReader io.Reader, cmdWriter io.Writer, useBuffer bool, verbose bool, calcPl, syncPl ProgressListener) (err error)

func Target

func Target(writer spgz.SparseFile, size int64, cmdReader io.Reader, cmdWriter io.Writer, useReadBuffer bool, verbose bool, calcPl, syncPl ProgressListener) (err error)

Types

type CountingReader

type CountingReader struct {
	io.Reader
	// contains filtered or unexported fields
}

func (*CountingReader) Count

func (c *CountingReader) Count() int64

func (*CountingReader) Read

func (r *CountingReader) Read(buf []byte) (n int, err error)

type CountingWriteCloser

type CountingWriteCloser struct {
	io.WriteCloser
	// contains filtered or unexported fields
}

func (*CountingWriteCloser) Count

func (c *CountingWriteCloser) Count() int64

func (*CountingWriteCloser) Write

func (r *CountingWriteCloser) Write(buf []byte) (n int, err error)

type FixingSpgzFileWrapper

type FixingSpgzFileWrapper struct {
	*spgz.SpgzFile
}

FixingSpgzFileWrapper conceals read errors caused by compressed data corruption by re-writing the corrupt blocks with zeros. Such errors are usually caused by abrupt termination of the writing process. This wrapper is used as the sync target so the corrupt blocks will be updated during the sync process.

func (*FixingSpgzFileWrapper) Read

func (rw *FixingSpgzFileWrapper) Read(p []byte) (n int, err error)

func (*FixingSpgzFileWrapper) Seek

func (rw *FixingSpgzFileWrapper) Seek(offset int64, whence int) (int64, error)

type ProgressListener

type ProgressListener interface {
	Start(size int64)
	Update(position int64)
}

type TargetFile

type TargetFile interface {
	io.ReadWriteSeeker
	io.WriterAt
	io.Closer
	spgz.Truncatable
}