storage

README

Storage package for the dynamodbdump backups

This package is used to organise the code a little better.

It will hold the structures and interfaces related to the storage of the backups.

Documentation

Index

• func Close(r io.Closer)
• func MarshalDynamoAttributeMap(attrs map[string]*dynamodb.AttributeValue) ([]byte, error)
• type BackupIface
• type CustomAttributeValue
  • func (attr *CustomAttributeValue) Marshal(in *dynamodb.AttributeValue)
  • func (attr *CustomAttributeValue) Unmarshal(out *dynamodb.AttributeValue)
• type FileInput
• type Manifest
• type ManifestEntry
• type S3Backup
  • func NewS3Backup(sess client.ConfigProvider) *S3Backup
  • func (h *S3Backup) DumpBuffer(input *FileInput, buff *bytes.Buffer)
  • func (h *S3Backup) Exists(input *FileInput) (bool, error)
  • func (h *S3Backup) Flush(input *FileInput, data []byte) error
  • func (h *S3Backup) GetFile(input *FileInput) (*io.ReadCloser, error)
  • func (h *S3Backup) LoadManifest(input *FileInput) error
  • func (h *S3Backup) Scan(dataReader *io.ReadCloser) error
  • func (h *S3Backup) Write(input *FileInput, s3BufferSize int, wg *sync.WaitGroup)
  • func (h *S3Backup) WriteToDB(tableName string, batchSize int64, waitPeriod time.Duration, ...) error

Functions

func Close

func Close(r io.Closer)

Close logs errors on io.Closer if any should happen during a Close call

func MarshalDynamoAttributeMap

func MarshalDynamoAttributeMap(attrs map[string]*dynamodb.AttributeValue) ([]byte, error)

MarshalDynamoAttributeMap only purpose is to create a json-format representation of the AttributeValue map without the empty fields. We wouldn't need that if the fields of the struct was set to `json:",omitempty"` and we can't use the dynamodbattribute package because you can't really do a field for field copy with it.

Types

type BackupIface

type BackupIface interface {
	LoadManifest(*FileInput) error
	GetFile(*FileInput) (*io.ReadCloser, error)
	Exists(input *FileInput) (bool, error)
	Flush(input *FileInput, data []byte) error
	Scan(*io.ReadCloser) error
	WriteToDB(string, int64, time.Duration, *sync.WaitGroup) error
	DumpBuffer(*FileInput, *bytes.Buffer)
	Write(*FileInput, int, *sync.WaitGroup)
}

BackupIface is the interface that each storage backends implement

type CustomAttributeValue

type CustomAttributeValue struct {
	B    []byte                           `json:"b,omitempty"`
	BOOL *bool                            `json:"bool,omitempty"`
	BS   [][]byte                         `json:"bs,omitempty"`
	N    *string                          `json:"n,omitempty"`
	NS   []*string                        `json:"ns,omitempty"`
	NULL *bool                            `json:"null,omitempty"`
	S    *string                          `json:"s,omitempty"`
	SS   []*string                        `json:"ss,omitempty"`
	L    []*CustomAttributeValue          `json:"l,omitempty"`
	M    map[string]*CustomAttributeValue `json:"m,omitempty"`
}

CustomAttributeValue is here to get rid of the empty fields during the json Marshaling as the dynamodb.AttributeValue don't have any json tag set

func (*CustomAttributeValue) Marshal

func (attr *CustomAttributeValue) Marshal(in *dynamodb.AttributeValue)

Marshal translates a *dynamodb.AttributeValue into a *CustomAttributeValue field for field

func (*CustomAttributeValue) Unmarshal

func (attr *CustomAttributeValue) Unmarshal(out *dynamodb.AttributeValue)

Unmarshal translates the current CustomAttributeValue to a *dynamodb.AttributeValue

type FileInput

type FileInput struct {
	Bucket, Path *string
}

FileInput is used as input for the functions that require a file definition, be it a s3 file or a file on disk. Bucket will be used only for S3-related functions. Path will be the Key in the context of S3 functions

type Manifest

type Manifest struct {
	Name    string          `json:"name"`
	Version int             `json:"version"`
	Entries []ManifestEntry `json:"entries"`
}

Manifest represents the backup manifest

type ManifestEntry

type ManifestEntry struct {
	URL       string `json:"url"`
	Mandatory bool   `json:"mandatory"`
}

ManifestEntry represents an entry in the backup manifest

type S3Backup

type S3Backup struct {
	BackupIface

	DataPipe chan map[string]*dynamodb.AttributeValue
	// contains filtered or unexported fields
}

S3Backup is the structure that implements the backup storage for the S3 backend

func NewS3Backup

func NewS3Backup(sess client.ConfigProvider) *S3Backup

NewS3Backup initlialiaes the s3 client and returns a pointer to a S3Backup struct

func (*S3Backup) DumpBuffer

func (h *S3Backup) DumpBuffer(input *FileInput, buff *bytes.Buffer)

DumpBuffer dumps the content of the given buffer to a new randomly generated file name in the given s3 path in the given bucket and resets the said buffer

func (*S3Backup) Exists

func (h *S3Backup) Exists(input *FileInput) (bool, error)

Exists checks that a given path in s3 exists as a file

func (*S3Backup) Flush

func (h *S3Backup) Flush(input *FileInput, data []byte) error

Flush writes the content of a bytes array to the given s3 path

func (*S3Backup) GetFile

func (h *S3Backup) GetFile(input *FileInput) (*io.ReadCloser, error)

GetFile downloads a file from s3 to memory (as the files are small by default - just a few Mb).

func (*S3Backup) LoadManifest

func (h *S3Backup) LoadManifest(input *FileInput) error

LoadManifest downloads the given manifest file and load it in the Manifest attribute of the struct

func (*S3Backup) Scan

func (h *S3Backup) Scan(dataReader *io.ReadCloser) error

Scan reads the data from a backup line by line, serializes it and sends it to the struct's channel

func (*S3Backup) Write

func (h *S3Backup) Write(input *FileInput, s3BufferSize int, wg *sync.WaitGroup)

Write reads from the given channel and sends the data the given bucket in files of s3BufferSize max size

func (*S3Backup) WriteToDB

func (h *S3Backup) WriteToDB(tableName string, batchSize int64, waitPeriod time.Duration, wg *sync.WaitGroup) error

WriteToDB pulls the s3 files from S3Backup.manifest and import them inside the given table using the given batch size (and wait period between each batch)