storage

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 1, 2023 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	DefaultL0CompactionFileThreshold   = 500
	DefaultL0CompactionThreshold       = 4
	DefaultL0StopWritesThreshold       = 12
	DefaultL0TargetFileSize            = 2 << 20
	DefaultFlushSplitBytes             = 0
	DefaultLBaseMaxBytes               = 64 << 20
	DefaultMaxOpenFiles                = 1000
	DefaultMemTableSize                = 4 << 20
	DefaultMemTableStopWritesThreshold = 2
	DefaultMaxConcurrentCompactions    = 1
	DefaultMetricsLogInterval          = time.Duration(0)
)

Variables

View Source 
var (
	ErrNoLogEntry                   = errors.New("storage: no log entry")
	ErrNoCommitContext              = errors.New("storage: no commit context")
	ErrInconsistentWriteCommitState = errors.New("storage: inconsistent write and commit")
)

Functions

func TestAppendLogEntryWithoutCommitContext added in v0.5.0 

func TestAppendLogEntryWithoutCommitContext(tb testing.TB, stg *Storage, llsn types.LLSN, glsn types.GLSN, data []byte)

TestAppendLogEntryWithoutCommitContext stores log entries without commit context.

func TestDeleteCommitContext added in v0.7.0 

func TestDeleteCommitContext(tb testing.TB, stg *Storage)

func TestDeleteLogEntry added in v0.7.0 

func TestDeleteLogEntry(tb testing.TB, stg *Storage, lsn varlogpb.LogSequenceNumber)

func TestGetUnderlyingDB added in v0.5.0 

func TestGetUnderlyingDB(tb testing.TB, stg *Storage) (dataDB, commitDB *pebble.DB)

TestGetUnderlyingDB returns a pebble that is an internal database in the storage.

func TestSetCommitContext added in v0.5.0 

func TestSetCommitContext(tb testing.TB, stg *Storage, cc CommitContext)

TestSetCommitContext stores only commit context.

Types

type AppendBatch added in v0.7.0 

type AppendBatch struct {
	// contains filtered or unexported fields
}

AppendBatch is a batch to put one or more log entries.

func (*AppendBatch) Apply added in v0.7.0 

func (ab *AppendBatch) Apply() error

Apply saves a batch of appended log entries to the storage.

func (*AppendBatch) Close added in v0.7.0 

func (ab *AppendBatch) Close() error

Close releases an AppendBatch.

func (*AppendBatch) SetCommitContext added in v0.7.0 

func (ab *AppendBatch) SetCommitContext(cc CommitContext) error

SetCommitContext inserts a commit context.

func (*AppendBatch) SetLogEntry added in v0.7.0 

func (ab *AppendBatch) SetLogEntry(llsn types.LLSN, glsn types.GLSN, data []byte) error

SetLogEntry inserts a log entry.

type CommitBatch 

type CommitBatch struct {
	// contains filtered or unexported fields
}

func (*CommitBatch) Apply 

func (cb *CommitBatch) Apply() error

func (*CommitBatch) Close 

func (cb *CommitBatch) Close() error

func (*CommitBatch) Set 

func (cb *CommitBatch) Set(llsn types.LLSN, glsn types.GLSN) error

type CommitContext 

type CommitContext struct {
	// Version is the unique identifier of the commit generated by the metadata
	// repository.
	Version types.Version
	// HighWatermark is the maximum GLSN in the versioned commit.
	HighWatermark types.GLSN
	// CommittedGLSNBegin is the start GLSN of this commit.
	CommittedGLSNBegin types.GLSN
	// CommittedGLSNEnd is the end GLSN of this commit exclusively. If none of
	// the log entries are committed, it equals the CommittedGLSNBegin.
	// The GLSN of the local high watermark in a log stream can be inferred by
	// CommittedGLSNEnd minus one. Even if the last commit does not contain any
	// log entries, CommitedGLSNEnd - 1 should be the GLSN of the local high
	// watermark.
	CommittedGLSNEnd types.GLSN
	// CommittedLLSNBegin is the starting LLSN of this commit.
	// The CommittedLLSNEnd can be inferred by the length of committed log
	// entries computed by the CommittedGLSNBegin and CommittedGLSNEnd and
	// CommittedLLSNBegin. Similarly, the CommittedLLSNEnd - 1 is the LLSN of
	// the local high watermark since the LLSNs are sequential.
	CommittedLLSNBegin types.LLSN
}

CommitContext is metadata that represents the environment in which the commit is issued; for instance, the commit's version, the largest GLSN having the same commit version, and the commit range of this log stream.

When a storage node restarts and, thus, its log stream replicas also restart, the log stream executor which runs the log stream replica should recover the previous status - internal/storagenode/logstream.(logStreamContext). To regain their prior status, they utilize the CommitContext.

Here are examples of the CommitContext. You can see github.com/kakao/varlog/proto/snpb.LogStreamCommitResult to refer to related commit messages. 

// LogStream 1 has not committed any log entries, and LogStream 2 has
// committed ten.
CommitContext{
	// LogStream 1
    Version: 1,
    HighWatermark: 10,
    CommittedGLSNBegin: 1,
    CommittedGLSNEnd: 1,
    CommittedLLSNBegin: 1,
}
CommitContext{
	// LogStream 2
    Version: 1,
    HighWatermark: 10,
    CommittedGLSNBegin: 1,
    CommittedGLSNEnd: 11,
    CommittedLLSNBegin: 1,
}

// LogStream 1 has not committed any log entries again, and LogStream 2 has
// committed ten.
// LogStream 1 can recover that there was no log entry committed by loading
// this commit context during restarting.
CommitContext{
	// LogStream 1
    Version: 2,
    HighWatermark: 20,
    CommittedGLSNBegin: 1,
    CommittedGLSNEnd: 1,
    CommittedLLSNBegin: 1,
}
CommitContext{
	// LogStream 2
    Version: 2,
    HighWatermark: 20,
    CommittedGLSNBegin: 11,
    CommittedGLSNEnd: 21,
    CommittedLLSNBegin: 11,
}

// LogStream 1 has committed ten log entries, and LogStream 2 has committed
// ten.
CommitContext{
	// LogStream 1
	Version: 3,
	HighWatermark: 40,
	CommittedGLSNBegin: 21,
	CommittedGLSNEnd: 31,
	CommittedLLSNBegin: 1,
}
CommitContext{
	// LogStream 2
	Version: 3,
	HighWatermark: 40,
	CommittedGLSNBegin: 31,
	CommittedGLSNEnd: 41,
	CommittedLLSNBegin: 21,
}

// LogStream 1 has committed ten log entries, and LogStream 2 has not.
// LogStream 2 can recover the GLSN and LLSN of the last committed log entry
// by loading this commit context when restarting itself.
CommitContext{
	// LogStream 1
	Version: 4,
	HighWatermark: 50,
	CommittedGLSNBegin: 41,
	CommittedGLSNEnd: 51,
	CommittedLLSNBegin: 11,
}
CommitContext{
	// LogStream 2
	Version: 4,
	HighWatermark: 50,
	CommittedGLSNBegin: 41,
	CommittedGLSNEnd: 41,
	CommittedLLSNBegin: 31,
}

func (CommitContext) Empty 

func (cc CommitContext) Empty() bool

func (CommitContext) Equal 

func (cc CommitContext) Equal(other CommitContext) bool

type DBOption added in v0.13.0 

type DBOption interface {
	// contains filtered or unexported methods
}

func WithFlushSplitBytes added in v0.13.0 

func WithFlushSplitBytes(flushSplitBytes int64) DBOption

func WithL0CompactionFileThreshold added in v0.13.0 

func WithL0CompactionFileThreshold(l0CompactionFileThreshold int) DBOption

func WithL0CompactionThreshold 

func WithL0CompactionThreshold(l0CompactionThreshold int) DBOption

func WithL0StopWritesThreshold 

func WithL0StopWritesThreshold(l0StopWritesThreshold int) DBOption

func WithL0TargetFileSize added in v0.13.0 

func WithL0TargetFileSize(l0TargetFileSize int64) DBOption

func WithLBaseMaxBytes 

func WithLBaseMaxBytes(lbaseMaxBytes int64) DBOption

func WithMaxConcurrentCompaction 

func WithMaxConcurrentCompaction(maxConcurrentCompaction int) DBOption

func WithMaxOpenFiles 

func WithMaxOpenFiles(maxOpenFiles int) DBOption

func WithMemTableSize 

func WithMemTableSize(memTableSize int) DBOption

func WithMemTableStopWritesThreshold 

func WithMemTableStopWritesThreshold(memTableStopWritesThreshold int) DBOption

type Option 

type Option interface {
	// contains filtered or unexported methods
}

func ReadOnly added in v0.5.0 

func ReadOnly() Option

ReadOnly makes storage read-only. It is helpful only for testing. Usually, users do not have to call it.

func SeparateDatabase added in v0.13.0 

func SeparateDatabase() Option

func WithCommitDBOptions added in v0.13.0 

func WithCommitDBOptions(commitDBOpts ...DBOption) Option

func WithDataDBOptions added in v0.13.0 

func WithDataDBOptions(dataDBOpts ...DBOption) Option

func WithLogger 

func WithLogger(logger *zap.Logger) Option

func WithMetrisLogInterval added in v0.12.0 

func WithMetrisLogInterval(metricsLogInterval time.Duration) Option

func WithPath 

func WithPath(path string) Option

func WithVerboseLogging 

func WithVerboseLogging() Option

func WithoutSync 

func WithoutSync() Option

func WithoutWAL 

func WithoutWAL() Option

type ReadOption 

type ReadOption interface {
	// contains filtered or unexported methods
}

func AtGLSN 

func AtGLSN(glsn types.GLSN) ReadOption

func AtLLSN 

func AtLLSN(llsn types.LLSN) ReadOption

type RecoveryPoints 

type RecoveryPoints struct {
	LastCommitContext *CommitContext
	CommittedLogEntry struct {
		First *varlogpb.LogEntryMeta
		Last  *varlogpb.LogEntryMeta
	}
}

type ScanOption 

type ScanOption interface {
	// contains filtered or unexported methods
}

func WithGLSN 

func WithGLSN(begin, end types.GLSN) ScanOption

func WithLLSN 

func WithLLSN(begin, end types.LLSN) ScanOption

type Scanner 

type Scanner struct {
	// contains filtered or unexported fields
}

func (*Scanner) Close 

func (s *Scanner) Close() error

func (*Scanner) Next 

func (s *Scanner) Next() bool

func (*Scanner) Valid 

func (s *Scanner) Valid() bool

func (*Scanner) Value 

func (s *Scanner) Value() (le varlogpb.LogEntry, err error)

type Storage 

type Storage struct {
	// contains filtered or unexported fields
}

func New 

func New(opts ...Option) (*Storage, error)

New creates a new storage.

func TestNewStorage 

func TestNewStorage(tb testing.TB, opts ...Option) *Storage

func (*Storage) Close 

func (s *Storage) Close() (err error)

Close closes the storage.

func (*Storage) DiskUsage 

func (s *Storage) DiskUsage() uint64

func (*Storage) NewAppendBatch added in v0.7.0 

func (s *Storage) NewAppendBatch() *AppendBatch

NewAppendBatch creates a batch for appending log entries. It does not put commit context.

func (*Storage) NewCommitBatch 

func (s *Storage) NewCommitBatch(cc CommitContext) (*CommitBatch, error)

NewCommitBatch creates a batch for commit operations.

func (*Storage) NewScanner 

func (s *Storage) NewScanner(opts ...ScanOption) *Scanner

NewScanner creates a scanner for the given key range.

func (*Storage) NewWriteBatch 

func (s *Storage) NewWriteBatch() *WriteBatch

NewWriteBatch creates a batch for write operations.

func (*Storage) Path 

func (s *Storage) Path() string

Path returns the path to the storage.

func (*Storage) Read 

func (s *Storage) Read(opts ...ReadOption) (le varlogpb.LogEntry, err error)

Read reads the log entry at the glsn.

func (*Storage) ReadCommitContext added in v0.5.0 

func (s *Storage) ReadCommitContext() (cc CommitContext, err error)

func (*Storage) ReadRecoveryPoints 

func (s *Storage) ReadRecoveryPoints() (rp RecoveryPoints, err error)

ReadRecoveryPoints reads data necessary to restore the status of a log stream replica - the first and last log entries and commit context. Incompatible between the boundary of log entries and commit context is okay; thus, it returns nil as err. However, if there is a fatal error, such as missing data in a log entry, it returns an error.

func (*Storage) Trim 

func (s *Storage) Trim(glsn types.GLSN) error

Trim deletes log entries whose GLSNs are less than or equal to the argument glsn. Internally, it removes records for both data and commits but does not remove the commit context. It returns the ErrNoLogEntry if there are no logs to delete.

type WriteBatch 

type WriteBatch struct {
	// contains filtered or unexported fields
}

WriteBatch is a batch of writes to storage.

func (*WriteBatch) Apply 

func (wb *WriteBatch) Apply() error

Apply applies the batch to the underlying storage.

func (*WriteBatch) Close 

func (wb *WriteBatch) Close() error

Close releases acquired resources.

func (*WriteBatch) Set 

func (wb *WriteBatch) Set(llsn types.LLSN, data []byte) error

Set writes the given LLSN and data to the batch.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.