rsync

package
v0.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 30, 2018 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package rsync provides an implementation of the rsync algorithm as described in Andrew Tridgell's thesis (https://www.samba.org/~tridge/phd_thesis.pdf) and the rsync technical report (https://rsync.samba.org/tech_report). Rsync algorithmic functionality is provided by the Engine type, and a transport protocol for pipelined rsync operations is provided by the Client and Server types.

Index

Constants

View Source 
const (

	// DefaultBlockSize is the default block size that will be used if a zero
	// value is passed into Engine.Signature for the blockSize parameter.
	DefaultBlockSize = 1 << 13
	// DefaultMaximumDataOperationSize is the default maximum data size
	// permitted per operation. The optimal value for this isn't at all
	// correlated with block size - it's just what's reasonable to hold
	// in-memory and pass over the wire in a single transmission. This value
	// will be used if a zero value is passed into Engine.Deltafy for the
	// maxDataOpSize parameter.
	DefaultMaximumDataOperationSize = 1 << 16
)

Variables

This section is empty.

Functions

func DecodeToReceiver 

func DecodeToReceiver(decoder Decoder, count uint64, receiver Receiver) error

DecodeToReceiver decodes messages from the specified Decoder and forwards them to the specified receiver. It must be passed the number of files to be received so that it knows when forwarding is complete. It is designed to be used with an encoding receiver, such as that returned by NewEncodingReceiver. It finalizes the provided receiver before returning.

func OptimalBlockSizeForBase 

func OptimalBlockSizeForBase(base io.Seeker) (uint64, error)

OptimalBlockSizeForBase is a convenience function that will determine the optimal block size for a base that implements io.Seeker. It calls down to OptimalBlockSizeForBaseLength. After determining the base's length, it will attempt to reset the base to its original position.

func OptimalBlockSizeForBaseLength 

func OptimalBlockSizeForBaseLength(baseLength uint64) uint64

OptimalBlockSizeForBaseLength uses a simpler heuristic to choose a block size based on the base length. It starts by choosing the optimal block length using the formula given in the rsync thesis. It then enforces that the block size is within a sensible range. TODO: Should we add rounding to "nice" values, e.g. the nearest multiple of 1024 bytes? Would this improve read throughput?

func Transmit 

func Transmit(root string, paths []string, signatures []Signature, receiver Receiver) error

Transmit performs streaming transmission of files (in rsync deltafied form) to the specified receiver.

Types

type BlockHash 

type BlockHash struct {
	// Weak is the weak hash for the block.
	Weak uint32
	// Strong is the strong hash for the block.
	Strong [sha1.Size]byte
}

BlockHash represents a pair of weak and strong hash for a base block.

type Decoder 

type Decoder interface {
	// Decode decodes a value and should adhere to gob semantics.
	Decode(value interface{}) error
}

Decoder is the decoding interface used by DecodeToReceiver. It should adhere to gob semantics.

type Encoder 

type Encoder interface {
	// Encode encodes a value and should adhere to gob semantics. It should not
	// retain the value across calls because the value's internal buffers will
	// be re-used.
	Encode(value interface{}) error
}

Encoder is the encoding interface used by encoding receivers. It should adhere to gob semantics.

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine provides rsync functionality without any notion of transport. It is designed to be re-used to avoid heavy buffer allocation.

func NewEngine 

func NewEngine() *Engine

NewEngine creates a new rsync engine.

func (*Engine) BytesSignature 

func (e *Engine) BytesSignature(base []byte, blockSize uint64) Signature

func (*Engine) Deltafy 

func (e *Engine) Deltafy(target io.Reader, base Signature, maxDataOpSize uint64, transmit OperationTransmitter) error

TODO: We should document that the internal engine buffer will be resized to greater than maxDataOpSize and retained for the lifetime of the engine, so a reasonable value should be provided.

func (*Engine) DeltafyBytes 

func (e *Engine) DeltafyBytes(target []byte, base Signature, maxDataOpSize uint64) []Operation

func (*Engine) Patch 

func (e *Engine) Patch(destination io.Writer, base io.ReadSeeker, signature Signature, operation Operation) error

func (*Engine) PatchBytes 

func (e *Engine) PatchBytes(base []byte, signature Signature, delta []Operation) ([]byte, error)

func (*Engine) Signature 

func (e *Engine) Signature(base io.Reader, blockSize uint64) (Signature, error)

type Monitor 

type Monitor func(*ReceiverStatus) error

Monitor is the interface that monitors must implement to capture status information from a monitoring receiver. The argument provided to this function will be allocated on each update and can be kept by the monitoring callback. There's no point in attempting to re-use the allocated argument because (a) it would be complicated and the callback would most likely just copy it anyway and (b) it will only be allocated once per received file, and the per-file allocations are already significantly higher.

type Operation 

type Operation struct {
	// Data contains data for data operations. If its length is 0, the operation
	// is assumed to be a non-data operation. Operation transmitters and
	// receivers may thus treat a length-0 buffer as semantically equivalent to
	// a nil buffer and utilize that fact to efficiently re-use buffer capacity,
	// e.g. by truncating the buffer and doing a gob receive into it.
	Data []byte
	// Start is the 0-indexed starting block for block operations.
	Start uint64
	// Count is the number of blocks for block operations.
	Count uint64
}

type OperationTransmitter 

type OperationTransmitter func(Operation) error

OperationTransmitter transmits an operation. Operation data buffers are re-used between calls to the transmitter, so the transmitter should not return until it has either transmitted the data buffer (if any) or copied it for later transmission.

type Receiver 

type Receiver interface {
	// Receive processes a single message in a transmission stream.
	Receive(message Transmission) error
	// contains filtered or unexported methods
}

Receiver manages the streaming reception of multiple files. It should be used in conjunction with the Transmit function.

func NewEncodingReceiver 

func NewEncodingReceiver(encoder Encoder) Receiver

NewEncodingReceiver creates a new receiver that handles messages by encoding them with the specified Encoder. It is designed to be used with DecodeToReceiver.

func NewMonitoringReceiver 

func NewMonitoringReceiver(receiver Receiver, paths []string, monitor Monitor) Receiver

NewMonitoringReceiver wraps a receiver and provides monitoring information via a callback.

func NewPreemptableReceiver 

func NewPreemptableReceiver(receiver Receiver, run context.Context) Receiver

NewPreemptableReceiver wraps a receiver and aborts on Receive if the specified context has been cancelled.

func NewReceiver 

func NewReceiver(root string, paths []string, signatures []Signature, sinker Sinker) (Receiver, error)

NewReceiver creates a new receiver that stores files on disk.

type ReceiverStatus 

type ReceiverStatus struct {
	// Path is the path currently being received.
	Path string `protobuf:"bytes,1,opt,name=path" json:"path,omitempty"`
	// Received is the number of paths that have already been received.
	Received uint64 `protobuf:"varint,2,opt,name=received" json:"received,omitempty"`
	// Total is the total number of paths expected.
	Total                uint64   `protobuf:"varint,3,opt,name=total" json:"total,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

ReceivingStatus encodes that status of an rsync receiver.

func (*ReceiverStatus) Descriptor 

func (*ReceiverStatus) Descriptor() ([]byte, []int)

func (*ReceiverStatus) EnsureValid 

func (s *ReceiverStatus) EnsureValid() error

func (*ReceiverStatus) GetPath 

func (m *ReceiverStatus) GetPath() string

func (*ReceiverStatus) GetReceived 

func (m *ReceiverStatus) GetReceived() uint64

func (*ReceiverStatus) GetTotal 

func (m *ReceiverStatus) GetTotal() uint64

func (*ReceiverStatus) ProtoMessage 

func (*ReceiverStatus) ProtoMessage()

func (*ReceiverStatus) Reset 

func (m *ReceiverStatus) Reset()

func (*ReceiverStatus) String 

func (m *ReceiverStatus) String() string

func (*ReceiverStatus) XXX_DiscardUnknown 

func (m *ReceiverStatus) XXX_DiscardUnknown()

func (*ReceiverStatus) XXX_Marshal 

func (m *ReceiverStatus) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ReceiverStatus) XXX_Merge 

func (dst *ReceiverStatus) XXX_Merge(src proto.Message)

func (*ReceiverStatus) XXX_Size 

func (m *ReceiverStatus) XXX_Size() int

func (*ReceiverStatus) XXX_Unmarshal 

func (m *ReceiverStatus) XXX_Unmarshal(b []byte) error

type Signature 

type Signature struct {
	// BlockSize is the block size used to compute the signature.
	BlockSize uint64
	// LastBlockSize is the size of the last block in the signature.
	LastBlockSize uint64
	// Hashes are the hashes of the blocks in the base.
	Hashes []BlockHash
}

Signature represents an rsync base signature. It encodes the block size used to generate the signature, the size of the last block in the signature (which may be smaller than a full block), and the hashes for the blocks of the file.

type Sinker 

type Sinker interface {
	// Sink should return a new io.WriteCloser for staging the given path. Each
	// result it returns will be closed before Sink is invoked again.
	Sink(path string) (io.WriteCloser, error)
}

Sinker provides the interface for a receiver to store incoming files.

type Transmission 

type Transmission struct {
	// Done indicates that the operation stream for the current file is
	// finished. If set, there will be no operation in the response, but there
	// may be an error.
	Done bool
	// Operation is the next operation in the stream for the current file.
	Operation Operation
	// Error indicates that a non-terminal error has occurred. It will only be
	// present if Done is true.
	Error string
}

Transmission represents a single message in a transmission stream. Its internals are public to allow for transmission using a reflection-based encoder (such as gob), but it should otherwise be treated as an opaque type with a private implementation.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.