persist

README

Persist

The persist package contains functionality for storing persisted data.

Subsystems

• appendonly
• boltdb
• json
• log
• persist

AppendOnly

Key Files

• appendonly.go

The AppendOnly Persistence System is responsible for persistence files maintained in an append-only fashion and ensures safe and performant ACID operations. An append-only structure is used with the length in bytes encoded in the metadata.

Inbound Complexities

• NewAppendOnlyPersist initializes the persistence file, either creating or loading it
• Write appends the information to the persistence file

BoltDB

Key Files

• boltdb.go

TODO

• fill out module explanation

JSON

Key Files

• json.go

TODO

• fill out module explanation

Log

Key Files

• log.go

TODO

• fill out module explanation

Persist

Key Files

• persist.go

TODO

• fill out module explanation

Documentation

Index

• Constants
• Variables
• func LoadJSON(meta Metadata, object interface{}, filename string) error
• func RandomSuffix() string
• func RemoveFile(filename string) error
• func SaveJSON(meta Metadata, object interface{}, filename string) error
• func UID() string
• type AppendOnlyPersist
  • func NewAppendOnlyPersist(dir, file string, metadataHeader, metadataVersion types.Specifier) (*AppendOnlyPersist, io.Reader, error)
  • func (aop *AppendOnlyPersist) Close() error
  • func (aop *AppendOnlyPersist) FilePath() string
  • func (aop *AppendOnlyPersist) PersistLength() uint64
  • func (aop *AppendOnlyPersist) Write(b []byte) (int, error)
• type BoltDatabase
  • func OpenDatabase(md Metadata, filename string) (*BoltDatabase, error)
  • func (db *BoltDatabase) Close() error
• type FixedMetadata
  • func VerifyMetadataHeader(r io.Reader, expected FixedMetadata) (FixedMetadata, error)
• type Logger
  • func NewFileLogger(logFilename string) (*Logger, error)
  • func NewLogger(w io.Writer) (*Logger, error)
• type Metadata

Constants

const (
	// DefaultDiskPermissionsTest when creating files or directories in tests.
	DefaultDiskPermissionsTest = 0750

	// FixedMetadataSize is the size of the FixedMetadata header in bytes
	FixedMetadataSize = 32
)

const (
	// MetadataPageSize is the number of bytes set aside for the metadata page
	// on disk. It is the length of a disk sector so that we can ensure that a
	// metadata disk write is ACID with one write and sync and we don't have to
	// worry about the persist data ever crossing the first disk sector.
	MetadataPageSize uint64 = 4096
)

Variables

var (
	// ErrWrongHeader is the wrong header error.
	ErrWrongHeader = errors.New("wrong header")
	// ErrWrongVersion is the wrong version error.
	ErrWrongVersion = errors.New("wrong version")
)

var (
	// ErrBadFilenameSuffix indicates that SaveJSON or LoadJSON was called using
	// a filename that has a bad suffix. This prevents users from trying to use
	// this package to manage the temp files - this package will manage them
	// automatically.
	ErrBadFilenameSuffix = errors.New("filename suffix not allowed")

	// ErrBadHeader indicates that the file opened is not the file that was
	// expected.
	ErrBadHeader = errors.New("wrong header")

	// ErrBadVersion indicates that the version number of the file is not
	// compatible with the current codebase.
	ErrBadVersion = errors.New("incompatible version")

	// ErrFileInUse is returned if SaveJSON or LoadJSON is called on a file
	// that's already being manipulated in another thread by the persist
	// package.
	ErrFileInUse = errors.New("another thread is saving or loading this file")
)

var (
	// MetadataVersionv150 is a common metadata version specifier to avoid
	// types.Specifier conflicts
	MetadataVersionv150 = types.NewSpecifier("v1.5.0\n")

	// MetadataVersionv156 is a common metadata version specifier to avoid
	// types.Specifier conflicts
	MetadataVersionv156 = types.NewSpecifier("v1.5.6\n")
)

Functions

func LoadJSON

func LoadJSON(meta Metadata, object interface{}, filename string) error

LoadJSON will load a persisted json object from disk.

func RandomSuffix

func RandomSuffix() string

RandomSuffix returns a 20 character base32 suffix for a filename. There are 100 bits of entropy, and a very low probability of colliding with existing files unintentionally.

func RemoveFile

func RemoveFile(filename string) error

RemoveFile removes an atomic file from disk, along with any uncommitted or temporary files.

func SaveJSON

func SaveJSON(meta Metadata, object interface{}, filename string) error

SaveJSON will save a json object to disk in a durable, atomic way. The resulting file will have a checksum of the data as the third line. If manually editing files, the checksum line can be replaced with the 8 characters "manual". This will cause the reader to accept the checksum even though the file has been changed.

func UID

func UID() string

UID returns a hexadecimal encoded string that can be used as an unique ID.

Types

type AppendOnlyPersist

type AppendOnlyPersist struct {
	// contains filtered or unexported fields
}

AppendOnlyPersist is the object responsible for creating, loading, and updating append-only persist files.

func NewAppendOnlyPersist

func NewAppendOnlyPersist(dir, file string, metadataHeader, metadataVersion types.Specifier) (*AppendOnlyPersist, io.Reader, error)

NewAppendOnlyPersist creates a new AppendOnlyPersist object and initializes the persistence file.

func (*AppendOnlyPersist) Close

func (aop *AppendOnlyPersist) Close() error

Close closes the persist file, freeing the resource and preventing further writes.

func (*AppendOnlyPersist) FilePath

func (aop *AppendOnlyPersist) FilePath() string

FilePath returns the filepath of the persist file.

func (*AppendOnlyPersist) PersistLength

func (aop *AppendOnlyPersist) PersistLength() uint64

PersistLength returns the length of the persist data.

func (*AppendOnlyPersist) Write

func (aop *AppendOnlyPersist) Write(b []byte) (int, error)

Write implements the io.Writer interface. It updates the persist file, appending the changes to the persist file on disk.

type BoltDatabase

type BoltDatabase struct {
	Metadata
	*bolt.DB
}

BoltDatabase is a persist-level wrapper for the bolt database, providing extra information such as a version number.

func OpenDatabase

func OpenDatabase(md Metadata, filename string) (*BoltDatabase, error)

OpenDatabase opens a database and validates its metadata.

func (*BoltDatabase) Close

func (db *BoltDatabase) Close() error

Close closes the database.

type FixedMetadata

type FixedMetadata struct {
	Header  types.Specifier
	Version types.Specifier
}

FixedMetadata contains the header and version of the data being stored as a fixed-length byte-array.

func VerifyMetadataHeader

func VerifyMetadataHeader(r io.Reader, expected FixedMetadata) (FixedMetadata, error)

VerifyMetadataHeader will take in a reader and an expected metadata header, if the file's header has a different header or version it will return the corresponding error and the actual metadata header

type Logger

type Logger struct {
	*log.Logger
}

Logger is a wrapper for log.Logger.

func NewFileLogger

func NewFileLogger(logFilename string) (*Logger, error)

NewFileLogger returns a logger that logs to logFilename. The file is opened in append mode, and created if it does not exist.

func NewLogger

func NewLogger(w io.Writer) (*Logger, error)

NewLogger returns a logger that can be closed. Calls should not be made to the logger after 'Close' has been called.

type Metadata

type Metadata struct {
	Header  string
	Version string
}

Metadata contains the header and version of the data being stored.