tstore

Documentation

Index

• Constants
• func DisableLog()
• func UseLogger(logger slog.Logger)
• type Tstore
  • func New(appDir, dataDir string, anp *chaincfg.Params, ...) (*Tstore, error)
  • func NewTestTstore(t *testing.T, dataDir string) *Tstore
  • func (t *Tstore) BlobSave(token []byte, be store.BlobEntry) error
  • func (t *Tstore) Blobs(token []byte, digests [][]byte) (map[string]store.BlobEntry, error)
  • func (t *Tstore) BlobsByDataDesc(token []byte, dataDesc []string) ([]store.BlobEntry, error)
  • func (t *Tstore) BlobsDel(token []byte, digests [][]byte) error
  • func (t *Tstore) Close()
  • func (t *Tstore) DigestsByDataDesc(token []byte, dataDesc []string) ([][]byte, error)
  • func (t *Tstore) Fsck()
  • func (t *Tstore) Inventory() ([][]byte, error)
  • func (t *Tstore) PluginHookPost(h plugins.HookT, payload string)
  • func (t *Tstore) PluginHookPre(h plugins.HookT, payload string) error
  • func (t *Tstore) PluginRead(token []byte, pluginID, cmd, payload string) (string, error)
  • func (t *Tstore) PluginRegister(b backend.Backend, p backend.Plugin) error
  • func (t *Tstore) PluginSetup(pluginID string) error
  • func (t *Tstore) PluginWrite(token []byte, pluginID, cmd, payload string) (string, error)
  • func (t *Tstore) Plugins() []backend.Plugin
  • func (t *Tstore) Record(token []byte, version uint32) (*backend.Record, error)
  • func (t *Tstore) RecordDel(token []byte) error
  • func (t *Tstore) RecordExists(token []byte) bool
  • func (t *Tstore) RecordFreeze(token []byte, rm backend.RecordMetadata, metadata []backend.MetadataStream, ...) error
  • func (t *Tstore) RecordLatest(token []byte) (*backend.Record, error)
  • func (t *Tstore) RecordNew() ([]byte, error)
  • func (t *Tstore) RecordPartial(token []byte, version uint32, filenames []string, omitAllFiles bool) (*backend.Record, error)
  • func (t *Tstore) RecordSave(token []byte, rm backend.RecordMetadata, metadata []backend.MetadataStream, ...) error
  • func (t *Tstore) RecordState(token []byte) (backend.StateT, error)
  • func (t *Tstore) RecordTimestamps(token []byte, version uint32) (*backend.RecordTimestamps, error)
  • func (t *Tstore) Setup() error
  • func (t *Tstore) Timestamp(token []byte, digest []byte) (*backend.Timestamp, error)

Constants

const (
	// DBTypeLevelDB is a config option that sets the backing key-value
	// store to a leveldb instance.
	DBTypeLevelDB = "leveldb"

	// DBTypeMySQL is a config option that sets the backing key-value
	// store to a MySQL instance.
	DBTypeMySQL = "mysql"
)

Functions

func DisableLog

func DisableLog()

DisableLog disables all library log output. Logging output is disabled by default until either UseLogger or SetLogWriter are called.

func UseLogger

func UseLogger(logger slog.Logger)

UseLogger uses a specified Logger to output package logging info. This should be used in preference to SetLogWriter if the caller is also using slog.

Types

type Tstore

type Tstore struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

Tstore is a data store that automatically timestamps all data saved to it onto the decred blockchain, making it possible to cryptographically prove that a piece of data existed at a specific block height. It combines a trillian log (tlog) and a key-value store. When data is saved to a tstore instance it is first saved to the key-value store then a digest of the data is appended onto the tlog tree. Tlog trees are episodically timestamped onto the decred blockchain. An inlcusion proof, i.e. the cryptographic proof that the data was included in the decred timestamp, can be retrieved for any individual piece of data saved to the tstore.

Saving only the digest of the data to tlog means that we separate the timestamp from the data itself. This allows us to remove content that is deemed undesirable from the key-value store without impacting the ability to retrieve inclusion proofs for any other pieces of data saved to tstore.

The tlog tree is append only and is treated as the source of truth. If any blobs make it into the key-value store but do not make it into the tlog tree they are considered to be orphaned and are simply ignored. We do not unwind failed calls.

func New

func New(appDir, dataDir string, anp *chaincfg.Params, tlogHost, tlogPass, dbType, dbHost, dbPass, dcrtimeHost, dcrtimeCert string) (*Tstore, error)

New returns a new tstore instance.

func NewTestTstore

func NewTestTstore(t *testing.T, dataDir string) *Tstore

NewTestTstore returns a tstore instance that is setup for testing.

func (*Tstore) BlobSave

func (t *Tstore) BlobSave(token []byte, be store.BlobEntry) error

BlobSave saves a BlobEntry to the tstore instance. The BlobEntry will be encrypted prior to being written to disk if the record is unvetted. The digest of the data, i.e. BlobEntry.Digest, can be thought of as the blob ID and can be used to get/del the blob from tstore.

This function satisfies the plugins TstoreClient interface.

func (*Tstore) Blobs

func (t *Tstore) Blobs(token []byte, digests [][]byte) (map[string]store.BlobEntry, error)

Blobs returns the blobs that correspond to the provided digests. If a blob does not exist it will not be included in the returned map. If a record is vetted, only vetted blobs will be returned.

This function satisfies the plugins TstoreClient interface.

func (*Tstore) BlobsByDataDesc

func (t *Tstore) BlobsByDataDesc(token []byte, dataDesc []string) ([]store.BlobEntry, error)

BlobsByDataDesc returns all blobs that match the provided data descriptors. The blobs will be ordered from oldest to newest. If a record is vetted then only vetted blobs will be returned.

This function satisfies the plugins TstoreClient interface.

func (*Tstore) BlobsDel

func (t *Tstore) BlobsDel(token []byte, digests [][]byte) error

BlobsDel deletes the blobs that correspond to the provided digests. Blobs can be deleted from both frozen and non-frozen records.

This function satisfies the plugins TstoreClient interface.

func (*Tstore) Close

func (t *Tstore) Close()

Close performs cleanup of the tstore.

func (*Tstore) DigestsByDataDesc

func (t *Tstore) DigestsByDataDesc(token []byte, dataDesc []string) ([][]byte, error)

DigestsByDataDesc returns the digests of all blobs that match the provided data descriptor. If a record is vetted then only vetted digests will be returned.

This function satisfies the plugins TstoreClient interface.

func (*Tstore) Fsck

func (t *Tstore) Fsck()

Fsck performs a filesystem check on the tstore.

func (*Tstore) Inventory

func (t *Tstore) Inventory() ([][]byte, error)

Inventory returns all record tokens that are in the tstore. Its possible for a token to be returned that does not correspond to an actual record. For example, if the tlog tree was created but saving the record to the tree failed due to an unexpected error then a empty tree with exist. This function does not filter those tokens out.

func (*Tstore) PluginHookPost

func (t *Tstore) PluginHookPost(h plugins.HookT, payload string)

PluginHookPre executes a tstore backend post hook. Post hooks are hooks that are executed after the tstore backend successfully writes data to disk. These hooks give plugins the opportunity to cache data from the write.

func (*Tstore) PluginHookPre

func (t *Tstore) PluginHookPre(h plugins.HookT, payload string) error

PluginHookPre executes a tstore backend pre hook. Pre hooks are hooks that are executed prior to the tstore backend writing data to disk. These hooks give plugins the opportunity to add plugin specific validation to record methods or plugin commands that write data.

func (*Tstore) PluginRead

func (t *Tstore) PluginRead(token []byte, pluginID, cmd, payload string) (string, error)

PluginRead executes a read-only plugin command.

func (*Tstore) PluginRegister

func (t *Tstore) PluginRegister(b backend.Backend, p backend.Plugin) error

PluginRegister registers a plugin. Plugin commands and hooks can be executed on the plugin once registered.

func (*Tstore) PluginSetup

func (t *Tstore) PluginSetup(pluginID string) error

PluginSetup performs any required setup for the specified plugin.

func (*Tstore) PluginWrite

func (t *Tstore) PluginWrite(token []byte, pluginID, cmd, payload string) (string, error)

PluginWrite executes a plugin command that writes data.

func (*Tstore) Plugins

func (t *Tstore) Plugins() []backend.Plugin

Plugins returns all registered plugins for the tstore instance.

func (*Tstore) Record

func (t *Tstore) Record(token []byte, version uint32) (*backend.Record, error)

Record returns the specified version of the record.

func (*Tstore) RecordDel

func (t *Tstore) RecordDel(token []byte) error

RecordDel walks the provided tree and deletes all blobs in the store that correspond to record files. This is done for all versions and all iterations of the record. Record metadata and metadata stream blobs are not deleted.

func (*Tstore) RecordExists

func (t *Tstore) RecordExists(token []byte) bool

RecordExists returns whether a record exists.

This method only returns whether a tree exists for the provided token. It's possible for a tree to exist that does not correspond to a record in the rare case that a tree was created but an unexpected error, such as a network error, was encoutered prior to the record being saved to the tree. We ignore this edge case because:

1. A user has no way to obtain this token unless the trillian instance has been opened to the public.

2. Even if they have the token they cannot do anything with it. Any attempt to read from the tree or write to the tree will return a RecordNotFound error.

Pulling the leaves from the tree to see if a record has been saved to the tree adds a large amount of overhead to this call, which should be a very light weight. Its for this reason that we rely on the tree exists call despite the edge case.

func (*Tstore) RecordFreeze

func (t *Tstore) RecordFreeze(token []byte, rm backend.RecordMetadata, metadata []backend.MetadataStream, files []backend.File) error

RecordFreeze updates the status of a record then freezes the trillian tree to prevent any additional updates.

A tree is considered to be frozen once the record index has been saved with its Frozen field set to true. The only thing that can be appended onto a frozen tree is one additional anchor record. Once a frozen tree has been anchored, the tstore fsck function will update the status of the tree to frozen in trillian, at which point trillian will prevent any changes to the tree.

func (*Tstore) RecordLatest

func (t *Tstore) RecordLatest(token []byte) (*backend.Record, error)

RecordLatest returns the latest version of a record.

func (*Tstore) RecordNew

func (t *Tstore) RecordNew() ([]byte, error)

RecordNew creates a new record in the tstore and returns the record token that serves as the unique identifier for the record. Creating a new record means creating a tlog tree for the record. Nothing is saved to the tree yet.

func (*Tstore) RecordPartial

func (t *Tstore) RecordPartial(token []byte, version uint32, filenames []string, omitAllFiles bool) (*backend.Record, error)

RecordPartial returns a partial record. This method gives the caller fine grained control over what version and what files are returned. The only required field is the token. All other fields are optional.

Version is used to request a specific version of a record. If no version is provided then the most recent version of the record will be returned.

Filenames can be used to request specific files. If filenames is not empty then the specified files will be the only files returned.

OmitAllFiles can be used to retrieve a record without any of the record files. This supersedes the filenames argument.

func (*Tstore) RecordSave

func (t *Tstore) RecordSave(token []byte, rm backend.RecordMetadata, metadata []backend.MetadataStream, files []backend.File) error

RecordSave saves the provided record to tstore. Once the record contents have been successfully saved to tstore, a recordIndex is created for this version of the record and saved to tstore as well. The record update is not considered to be valid until the record index has been successfully saved. If the record content makes it in but the record index does not, the record content blobs are orphaned and ignored.

func (*Tstore) RecordState

func (t *Tstore) RecordState(token []byte) (backend.StateT, error)

RecordState returns the state of a record. This call does not require retrieving any blobs from the kv store. The record state can be derived from only the tlog leaves.

func (*Tstore) RecordTimestamps

func (t *Tstore) RecordTimestamps(token []byte, version uint32) (*backend.RecordTimestamps, error)

RecordTimestamps returns the timestamps for the contents of a record. Timestamps for the record metadata, metadata streams, and files are all returned.

func (*Tstore) Setup

func (t *Tstore) Setup() error

Setup performs any required work to setup the tstore instance.

func (*Tstore) Timestamp

func (t *Tstore) Timestamp(token []byte, digest []byte) (*backend.Timestamp, error)

Timestamp returns the timestamp for the data blob that corresponds to the provided digest. If a record is vetted, only vetted timestamps will be returned.

This function satisfies the plugins TstoreClient interface.