storage

Documentation

Overview

Copyright 2016 The Transicator Authors

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright 2016 The Transicator Authors

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright 2016 The Transicator Authors

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

• Constants
• type BackupProgress
• type DB
• type Entry
• type SQL
  • func Open(baseFile string) (*SQL, error)
  • func (s *SQL) Backup(dest string) <-chan BackupProgress
  • func (s *SQL) Close()
  • func (s *SQL) Delete() error
  • func (s *SQL) Get(scope string, lsn uint64, index uint32) ([]byte, error)
  • func (s *SQL) GetBackup(w io.Writer) (err error)
  • func (s *SQL) Purge(oldest time.Time) (uint64, error)
  • func (s *SQL) Put(scope string, lsn uint64, index uint32, data []byte) error
  • func (s *SQL) PutBatch(entries []Entry) error
  • func (s *SQL) Scan(scopes []string, startLSN uint64, startIndex uint32, limit int, ...) (final [][]byte, firstSeq common.Sequence, lastSeq common.Sequence, err error)

Constants

const (
	EntryComparatorName    = "transicator-entries-v1"
	SequenceComparatorName = "transicator-sequence-v1"
)

Comparator names are persisted so should be consistent

Types

type BackupProgress

type BackupProgress struct {
	PagesRemaining int
	Done           bool
	Error          error
}

BackupProgress records are returned on a channel by the backup API.

type DB

type DB interface {
	// Close the database
	Close()

	// Delete everything from the database. Must be closed first.
	Delete() error

	// Insert a new entry.
	Put(scope string, lsn uint64, index uint32, data []byte) error

	// Insert a bunch of entries in one atomic batch
	PutBatch(entries []Entry) error

	// Retrieve a single entry
	Get(scope string, lsn uint64, index uint32) ([]byte, error)

	// Scan entries in order from startLSN and startIndex for all scopes
	// in the "scopes" array. If filter is non-nil, return only entries
	// for which it returns true.
	Scan(scopes []string,
		startLSN uint64, startIndex uint32,
		limit int,
		filter func([]byte) bool) ([][]byte, common.Sequence, common.Sequence, error)

	// Delete entries older than "oldest"
	Purge(oldest time.Time) (purgeCount uint64, err error)

	// Make a backup of the current database at the specified file name. The
	// function will return a channel that can be used to read the status of
	// the backup as it is being made.
	Backup(dest string) <-chan BackupProgress

	// GetBackup creates a backup of the current database in a temp filename,
	// and write the persistent db file to the specified writer.
	GetBackup(w io.Writer) error
}

A DB is a generic interface to the storage system. It maintains entries indexec by scope, lsn, and index within an LSN. It allows entries to be retrieved sequentially, or to be purged by time.

type Entry

type Entry struct {
	Scope string
	LSN   uint64
	Index uint32
	Data  []byte
}

An Entry represents a whole record in one go.

type SQL

type SQL struct {
	// contains filtered or unexported fields
}

An SQL is a handle to a database.

func Open

func Open(baseFile string) (*SQL, error)

Open opens a SQLite database and makes it available for reads and writes. Opened databases should be closed when done.

The "baseFile" parameter refers to the name of a directory where RocksDB can store its data. SQLite will create a few inside this directory. To create an empty database, make sure that it is empty.

func (*SQL) Backup

func (s *SQL) Backup(dest string) <-chan BackupProgress

Backup creates a backup of the current database in the specified file name, and sends results using the supplied channel.

func (*SQL) Close

func (s *SQL) Close()

Close closes the database cleanly.

func (*SQL) Delete

func (s *SQL) Delete() error

Delete deletes all the files used by the database.

func (*SQL) Get

func (s *SQL) Get(scope string, lsn uint64, index uint32) ([]byte, error)

Get returns what was written by PutEntry. It's mainly used for testing.

func (*SQL) GetBackup

func (s *SQL) GetBackup(w io.Writer) (err error)

GetBackup creates a backup of the current database in a temp filename, and write the persistent db file to the specified writer. It's the caller's responsibility to close the writer, if needed.

func (*SQL) Purge

func (s *SQL) Purge(oldest time.Time) (uint64, error)

Purge removes all entries older than the specified time.

func (*SQL) Put

func (s *SQL) Put(scope string, lsn uint64, index uint32, data []byte) error

Put writes an entry to the database indexed by scope, lsn, and index in order

func (*SQL) PutBatch

func (s *SQL) PutBatch(entries []Entry) error

PutBatch writes a whole bunch.

func (*SQL) Scan

func (s *SQL) Scan(
	scopes []string,
	startLSN uint64, startIndex uint32,
	limit int, filter func([]byte) bool) (final [][]byte, firstSeq common.Sequence, lastSeq common.Sequence, err error)

Scan returns entries in sequence number order a list of scopes. It also returns the sequences of the first and last records in the DB. The first entry returned will be the first entry that matches the specified startLSN and startIndex. No more than "limit" entries will be returned. To retrieve the very next entry after an entry, simply increment the index by 1. This method uses a snapshot to guarantee consistency even if data is being inserted to the database -- as long as the data is being inserted in LSN order! The array returned is the array of entries (again, in "sequence" order).