tools

package
v0.1.17 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 27, 2019 License: Apache-2.0 Imports: 15 Imported by: 0

README

Testing Tools

WAL-G offers three prototyping programs to assist with testing and development:

compress

compress takes in a directory and minimum part size and creates a compressed tarball.

Example use cases:

Compress a local directory to disk.

./compress -out=$HOME/directory/to/be/compressed 1000000 directory/to/compress

Connect to Postgres and upload the compressed tarball to S3.

./compress -s 1000000 /dat/9.6/data
extract

extract takes in a target out directory and however many files/urls. It is often used in conjunction with generate to test the accuracy and speed of decompression and extraction.

Example use cases:

Extract local files.

./extract /target/out/directory file1 file2

Extract local files and randomly generated data.

./extract /out/directory file1                     \
https://localhost:8080/stride-10.bytes-100.tar.lzo \
https://localhost:8080/stride-100.bytes-1000.tar.lzo
generate

generate outputs randomly compressed tarballs hosted on localhost:8080. To configure the stride length N and size of the data M, use: https://localhost:8080/stride-N.bytes-M.tar.lzo

Randomly generated data can be downloaded using:

curl -k https://localhost:8080/stride-1.bytes-1.tar.lzo \
-o /path/to/new/file

The URLs can also be fed directly to extract. Currently, generate only supports LZOP compression.

NOTE: compress and extract support profiling options using the flags -p and -m. The first generates a CPU profile to cpu.prof while the latter generates a memory profile to mem.prof.

To access the profiles, use:

go tool pprof wal-g FILENAME
delta

delta runs concurrent testing of delta backups:

  • init pg_bench
  • make base backup
  • run bp_bench
  • run pg_bench and delta backup in parallel
  • make delta backup
  • restore delta chain
  • compare restoragtion results and highlight but difference
  • if critical differences found diff will exit with panic

delta must be invoked with env variables necessary for wal-g backup-push

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateTar 

func CreateTar(w io.Writer, r *io.LimitedReader)

CreateTar creates a new tarball from the passed in reader and writes to a destination writer.

func Handler 

func Handler(w http.ResponseWriter, r *http.Request)

Handler allows for generation of random bytes by configuring the URL 'https://localhost:8080/stride-N.bytes-N.tar.lzo' where byte size and stride length are customizable.

Compressed tar files are automatically generated. Grab using curl ie. 'curl -sk ...'

func MakeDir 

func MakeDir(name string)

MakeDir creates a new directory with mode 0755.

func TimeTrack 

func TimeTrack(start time.Time, name string)

TimeTrack is used to time how long functions take.

Usage Example: defer timeTrack(time.Now(), "EXTRACT ALL")

Types

type BufferTarInterpreter 

type BufferTarInterpreter struct {
	Out []byte
}

BufferTarInterpreter extracts data to a byte slice. Used for testing purposes.

func (*BufferTarInterpreter) Interpret 

func (ti *BufferTarInterpreter) Interpret(tr io.Reader, cur *tar.Header) error

Interpret handles in memory tar formats. Used for testing purposes.

type FileReaderMaker 

type FileReaderMaker struct {
	Key        string
	FileFormat string
}

FileReaderMaker decompresses lzop tarballs from the passed in file.

func (*FileReaderMaker) Format 

func (f *FileReaderMaker) Format() string

func (*FileReaderMaker) Path 

func (f *FileReaderMaker) Path() string

func (*FileReaderMaker) Reader 

func (f *FileReaderMaker) Reader() (io.ReadCloser, error)

Reader creates a new reader from the passed in file.

type FileTarBall 

type FileTarBall struct {
	// contains filtered or unexported fields
}

FileTarBall represents a tarball that is written to disk.

func (*FileTarBall) AddSize added in v0.1.8 

func (fileTarBall *FileTarBall) AddSize(i int64)

func (*FileTarBall) AwaitUploads added in v0.1.8 

func (fileTarBall *FileTarBall) AwaitUploads()

func (*FileTarBall) CloseTar 

func (fileTarBall *FileTarBall) CloseTar() error

CloseTar closes the tar writer and file, flushing any unwritten data to the file before closing.

func (*FileTarBall) FileExtension added in v0.1.11 

func (fileTarBall *FileTarBall) FileExtension() string

func (*FileTarBall) Finish 

func (fileTarBall *FileTarBall) Finish(sentinelDto *walg.S3TarBallSentinelDto) error

Finish alerts that compression is complete.

func (*FileTarBall) SetUp 

func (fileTarBall *FileTarBall) SetUp(crypter walg.Crypter, names ...string)

SetUp creates a new LZ4 writer, tar writer and file for writing bundled compressed bytes to.

func (*FileTarBall) Size 

func (fileTarBall *FileTarBall) Size() int64

func (*FileTarBall) TarWriter added in v0.1.11 

func (fileTarBall *FileTarBall) TarWriter() *tar.Writer

func (*FileTarBall) Trim 

func (fileTarBall *FileTarBall) Trim() string

type FileTarBallMaker 

type FileTarBallMaker struct {
	Trim string
	Out  string
	// contains filtered or unexported fields
}

FileTarBallMaker creates a new FileTarBall with the directory that files should be extracted to.

func (*FileTarBallMaker) Make 

func (f *FileTarBallMaker) Make(inheritState bool) walg.TarBall

Make creates a new FileTarBall.

type HTTPReaderMaker 

type HTTPReaderMaker struct {
	Client     *http.Client
	Key        string
	FileFormat string
}

HTTPReaderMaker decompresses lzop tarballs from the passed in url.

func (*HTTPReaderMaker) Format 

func (h *HTTPReaderMaker) Format() string

func (*HTTPReaderMaker) Path 

func (h *HTTPReaderMaker) Path() string

func (*HTTPReaderMaker) Reader 

func (h *HTTPReaderMaker) Reader() (io.ReadCloser, error)

Reader creates a new request to grab the data generated by the random bytes generator.

type NOPTarBall 

type NOPTarBall struct {
	// contains filtered or unexported fields
}

NOPTarBall mocks a tarball. Used for testing purposes.

func (*NOPTarBall) AddSize added in v0.1.8 

func (n *NOPTarBall) AddSize(i int64)

func (*NOPTarBall) AwaitUploads added in v0.1.8 

func (b *NOPTarBall) AwaitUploads()

func (*NOPTarBall) CloseTar 

func (n *NOPTarBall) CloseTar() error

func (*NOPTarBall) FileExtension added in v0.1.11 

func (n *NOPTarBall) FileExtension() string

func (*NOPTarBall) Finish 

func (n *NOPTarBall) Finish(sentinelDto *walg.S3TarBallSentinelDto) error

func (*NOPTarBall) SetUp 

func (n *NOPTarBall) SetUp(crypter walg.Crypter, params ...string)

func (*NOPTarBall) Size 

func (n *NOPTarBall) Size() int64

func (*NOPTarBall) TarWriter added in v0.1.11 

func (n *NOPTarBall) TarWriter() *tar.Writer

func (*NOPTarBall) Trim 

func (n *NOPTarBall) Trim() string

type NOPTarBallMaker 

type NOPTarBallMaker struct {
	Trim string
	// contains filtered or unexported fields
}

NOPTarBallMaker creates a new NOPTarBall. Used for testing purposes.

func (*NOPTarBallMaker) Make 

func (n *NOPTarBallMaker) Make(inheritState bool) walg.TarBall

Make creates a new NOPTarBall.

type NOPTarInterpreter 

type NOPTarInterpreter struct{}

NOPTarInterpreter mocks a tar extractor.

func (*NOPTarInterpreter) Interpret 

func (ti *NOPTarInterpreter) Interpret(tr io.Reader, cur *tar.Header) error

Interpret does not do anything except print the 'tar member' name.

type StrideByteReader 

type StrideByteReader struct {
	// contains filtered or unexported fields
}

StrideByteReader allows for customizable "strides" of random bytes. Creates an infinite stream.

func NewStrideByteReader 

func NewStrideByteReader(s int) *StrideByteReader

NewStrideByteReader creates a new random byte stride generator with a seed of 0.

func (*StrideByteReader) Read 

func (sb *StrideByteReader) Read(p []byte) (int, error)

Read creates randomly generated bytes of 'stride' length.

Source Files

View all Source files

Directories

Path Synopsis
cmd
compress
delta
extract
generate

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.