spgz

package module
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 18, 2023 License: MIT Imports: 10 Imported by: 5

README

spgz ('Sparse gzip')

Spgz is a Go library for handling compressed files optimized for random read and write access.

A file is split into blocks of equal size. Each block can be stored either uncompressed or compressed using gzip (it compresses the data and only stores it compressed if it saves at least 8KB). If the block is compressed a hole is punched at the reminder of the block and this is how disk space is saved. The position of each block remains fixed allowing efficient random access.

Because it uses Fallocate(FALLOC_FL_PUNCH_HOLE), it only works on filesystems that support it (see http://man7.org/linux/man-pages/man2/fallocate.2.html).

There is an overhead of one byte per block and a 4KB header.

This library was developed for the disk image backup tool.

Documentation

Index

Constants

View Source 
const (
	DefBlockSize = 128*1024 - 1
	MinBlockSize = 3*4096 - 1
)
View Source 
const (
	FALLOC_FL_KEEP_SIZE  = 0x01 /* default is extend size */
	FALLOC_FL_PUNCH_HOLE = 0x02 /* de-allocates range */
)
View Source 
const (
	BUFSIZE = 32768
)

Variables

View Source 
var (
	ErrInvalidFormat         = errors.New("invalid file format")
	ErrFileIsDirectory       = errors.New("file cannot be a directory")
	ErrWriteOnlyNotSupported = errors.New("write only mode is not supported")
)
View Source 
var (
	ErrPunchHoleNotSupported = errors.New("this filesystem does not support punching holes. Use xfs, ext4, btrfs or such")
)

Functions

func IsBlockZero 

func IsBlockZero(buf []byte) (ret bool)

func NewSparseFile 

func NewSparseFile(f *os.File) *sparseFile

Types

type BuseDevice 

type BuseDevice struct {
	*SpgzFile
}

func NewBuseDevice 

func NewBuseDevice(f *SpgzFile) *BuseDevice

func (*BuseDevice) Trim 

func (d *BuseDevice) Trim(off, length int64) error

type ErrCorruptCompressedBlock 

type ErrCorruptCompressedBlock struct {
	// contains filtered or unexported fields
}

ErrCorruptCompressedBlock is returned when a compressed block involved in a read or a write operation is corrupt (for example as a result of a partial write). Note, this is not a media error.

Any read operation intersecting with the range specified by Offset() and Size(() will fail. Any write operation intersecting, but not fully covering the range, will also fail (therefore in order to make the file usable again one must overwrite the entire range in a single write operation).

func (*ErrCorruptCompressedBlock) Error 

func (e *ErrCorruptCompressedBlock) Error() string

func (*ErrCorruptCompressedBlock) Offset 

func (e *ErrCorruptCompressedBlock) Offset() int64

func (*ErrCorruptCompressedBlock) Size 

func (e *ErrCorruptCompressedBlock) Size() int64

func (*ErrCorruptCompressedBlock) Unwrap 

func (e *ErrCorruptCompressedBlock) Unwrap() error

type SparseFile 

type SparseFile interface {
	io.ReadWriteSeeker
	io.ReaderAt
	io.WriterAt
	io.Closer
	Truncatable

	PunchHole(offset, size int64) error
	Sync() error
}

type SparseFileWithFallback 

type SparseFileWithFallback struct {
	SparseFile
	// contains filtered or unexported fields
}

func NewSparseFileWithFallback 

func NewSparseFileWithFallback(f *os.File) *SparseFileWithFallback

NewSparseFileWithFallback creates a sparse file that will fall back to writing zeros if the first call to SparseFile.PunchHole() returns ErrPunchHoleNotSupported.

func NewSparseFileWithoutHolePunching 

func NewSparseFileWithoutHolePunching(f *os.File) *SparseFileWithFallback

NewSparseFileWithoutHolePunching creates a sparse file that will write zeros instead of calling SparseFile.PunchHole()

func (*SparseFileWithFallback) PunchHole 

func (f *SparseFileWithFallback) PunchHole(offset, size int64) error

type SparseWriter 

type SparseWriter struct {
	SparseFile
}

func NewSparseWriter 

func NewSparseWriter(f SparseFile) *SparseWriter

func (*SparseWriter) Write 

func (w *SparseWriter) Write(p []byte) (int, error)

type SpgzFile 

type SpgzFile struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func NewFromFile 

func NewFromFile(file *os.File, flag int) (f *SpgzFile, err error)

NewFromFile creates a new SpgzFile from the given *os.File with desired block size of DefBlockSize. See NewFromFileSize for more details.

func NewFromFileSize 

func NewFromFileSize(file *os.File, flag int, blockSize int64) (f *SpgzFile, err error)

NewFromFileSize creates a new SpgzFile from the given *os.File. File position must be 0.

The blockSize parameter specifies the desired block size for newly created files. The actual block size will be DefBlockSize if the supplied value is 0, otherwise ((blockSize+1) &^ 0xFFF)-1 or MinBlockSize whichever is greater. For optimal I/O performance reads and writes should be sized in multiples of the actual block size and aligned at the block size boundaries. This can be achieved by using bufio.Reader and bufio.Writer. This parameter is ignored for already initialised files. The current block size can be retrieved using SpgzFile.BlockSize().

The flag parameter is similar to that of os.OpenFile(). If the file is empty it will be initialised if os.O_CREATE and os.O_RDWR are set, otherwise ErrInvalidFormat is returned. Write-only mode (os.O_WRONLY) is not supported.

Returns ErrPunchHoleNotSupported if opening for writing and the underlying OS or filesystem does not support punching holes (as space-saving will not be possible in this case).

func NewFromSparseFile 

func NewFromSparseFile(file SparseFile, flag int) (f *SpgzFile, err error)

func NewFromSparseFileSize 

func NewFromSparseFileSize(file SparseFile, flag int, blockSize int64) (f *SpgzFile, err error)

func OpenFile 

func OpenFile(name string, flag int, perm os.FileMode) (f *SpgzFile, err error)

OpenFile opens a file as SpgzFile with block size of DefBlockSize. See OpenFileSize for more details.

func OpenFileSize 

func OpenFileSize(name string, flag int, perm os.FileMode, blockSize int64) (f *SpgzFile, err error)

OpenFileSize opens a file as SpgzFile. See NewFromFileSize for more details.

func (*SpgzFile) BlockSize 

func (f *SpgzFile) BlockSize() int64

BlockSize returns the block size for optimal I/O. Reads and writes should be multiples of the block size and should be aligned with block boundaries.

func (*SpgzFile) Close 

func (f *SpgzFile) Close() error

func (*SpgzFile) PunchHole 

func (f *SpgzFile) PunchHole(offset, size int64) error

func (*SpgzFile) Read 

func (f *SpgzFile) Read(buf []byte) (n int, err error)

func (*SpgzFile) ReadAt 

func (f *SpgzFile) ReadAt(buf []byte, offset int64) (n int, err error)

func (*SpgzFile) ReadFrom 

func (f *SpgzFile) ReadFrom(rd io.Reader) (n int64, err error)

func (*SpgzFile) Seek 

func (f *SpgzFile) Seek(offset int64, whence int) (int64, error)

func (*SpgzFile) Size 

func (f *SpgzFile) Size() (int64, error)

func (*SpgzFile) Sync 

func (f *SpgzFile) Sync() error

func (*SpgzFile) Truncate 

func (f *SpgzFile) Truncate(size int64) error

func (*SpgzFile) Write 

func (f *SpgzFile) Write(buf []byte) (n int, err error)

func (*SpgzFile) WriteAt 

func (f *SpgzFile) WriteAt(buf []byte, offset int64) (n int, err error)

func (*SpgzFile) WriteTo 

func (f *SpgzFile) WriteTo(w io.Writer) (n int64, err error)

type Truncatable 

type Truncatable interface {
	Truncate(size int64) error
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.