storage

Documentation

Index

• type FieldKeyReader
  • func NewFieldKeyReader(qtid dax.QualifiedTableID, field dax.FieldName, writelog io.ReadCloser) *FieldKeyReader
  • func (r *FieldKeyReader) Close() error
  • func (r *FieldKeyReader) Read() (computer.FieldKeyMap, error)
• type Resource
  • func (m *Resource) Append(msg []byte) error
  • func (m *Resource) IncrementWLVersion() (bool, error)
  • func (m *Resource) IsLocked() bool
  • func (m *Resource) LoadLatestSnapshot() (data io.ReadCloser, err error)
  • func (m *Resource) LoadWriteLog() (data io.ReadCloser, err error)
  • func (m *Resource) Lock() error
  • func (m *Resource) Snapshot(rc io.ReadCloser) error
  • func (m *Resource) SnapshotTo(wt io.WriterTo) error
  • func (m *Resource) Unlock() error
• type ResourceManager
  • func NewResourceManager(s computer.SnapshotService, w computer.WritelogService, l logger.Logger) *ResourceManager
  • func (mm *ResourceManager) GetFieldKeyResource(qtid dax.QualifiedTableID, field dax.FieldName) *Resource
  • func (mm *ResourceManager) GetShardResource(qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum) *Resource
  • func (mm *ResourceManager) GetTableKeyResource(qtid dax.QualifiedTableID, partition dax.PartitionNum) *Resource
  • func (mm *ResourceManager) RemoveAll() error
  • func (mm *ResourceManager) RemoveFieldKeyResource(qtid dax.QualifiedTableID, field dax.FieldName)
  • func (mm *ResourceManager) RemoveShardResource(qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum)
  • func (mm *ResourceManager) RemoveTable(qtid dax.QualifiedTableID) error
  • func (mm *ResourceManager) RemoveTableKeyResource(qtid dax.QualifiedTableID, partition dax.PartitionNum)
• type ShardReader
  • func NewShardReader(qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, ...) *ShardReader
  • func (r *ShardReader) Close() error
  • func (r *ShardReader) Read() (computer.LogMessage, error)
• type TableKeyReader
  • func NewTableKeyReader(qtid dax.QualifiedTableID, partition dax.PartitionNum, writelog io.ReadCloser) *TableKeyReader
  • func (r *TableKeyReader) Close() error
  • func (r *TableKeyReader) Read() (computer.PartitionKeyMap, error)

Types

type FieldKeyReader

type FieldKeyReader struct {
	// contains filtered or unexported fields
}

func NewFieldKeyReader

func NewFieldKeyReader(qtid dax.QualifiedTableID, field dax.FieldName, writelog io.ReadCloser) *FieldKeyReader

func (*FieldKeyReader) Close

func (r *FieldKeyReader) Close() error

func (*FieldKeyReader) Read

func (r *FieldKeyReader) Read() (computer.FieldKeyMap, error)

type Resource

type Resource struct {
	// contains filtered or unexported fields
}

Resource wraps the snapshotter and writelogger to maintain messy state between calls. Resource is *not* threadsafe, care should be taken that concurrent calls are not made to Resource methods. The exception being that Snapshot and Append are safe to call concurrently.

func (*Resource) Append

func (m *Resource) Append(msg []byte) error

Append appends the msg to the write log. It will fail if we haven't properly loaded and gotten a lock for the resource we're writing to.

func (*Resource) IncrementWLVersion

func (m *Resource) IncrementWLVersion() (bool, error)

IncrementWLVersion should be called during snapshotting with a write Tx held on the local resource. This ensures that any writes which completed prior to the snapshot are in the prior WL and any that complete after the snapshot are in the incremented WL. If there have been no writes since the latest snapshot, this returns false and does nothing. In this case, Snapshot should *not* be called.

func (*Resource) IsLocked

func (m *Resource) IsLocked() bool

IsLocked checks to see if this particular instance of the resource believes it holds the lock. It does not look at the state of underlying storage to verify the lock.

func (*Resource) LoadLatestSnapshot

func (m *Resource) LoadLatestSnapshot() (data io.ReadCloser, err error)

LoadLatestSnapshot finds the most recent snapshot for this resource and returns a ReadCloser for that snapshot data. If there is no snapshot for this resource it returns nil, nil.

func (*Resource) LoadWriteLog

func (m *Resource) LoadWriteLog() (data io.ReadCloser, err error)

LoadWriteLog can be called after LoadLatestSnapshot. It loads any writelog data which has been written since the latest snapshot. Subsequent calls to LoadWriteLog will only return new data that hasn't previously been returned from LoadWriteLog. If there is no writelog, it returns nil, nil.

func (*Resource) Lock

func (m *Resource) Lock() error

Lock acquires an advisory lock for this resource which grants us exclusive access to write to it. The normal pattern is to call:

1. LoadLatestSnapshot 2. LoadWriteLog 3. Lock 4. LoadWriteLog

The second call to LoadWriteLog is necessary in case any writes occurred between the last load and acquiring the lock. Once the lock is acquired it should not be possible for any more writes to occur. Lock will error if (a) we fail to acquire the lock or (b) the state of the snapshot store for this resource is not identical to what is was before the lock was acquired. Case (b) means that quite a lot has happened in between LoadWriteLog and Lock, and we should probably just die and start over.

func (*Resource) Snapshot

func (m *Resource) Snapshot(rc io.ReadCloser) error

Snapshot takes a ReadCloser which has the contents of the resource being tracked at a particular point in time and writes them to the Snapshot Store. Upon a successful write it will truncate any write logs which are now incorporated into the snapshot. Do not call until after calling IncrementWLVersion, and only if that method returns "true".

func (*Resource) SnapshotTo

func (m *Resource) SnapshotTo(wt io.WriterTo) error

SnapshotTo is Snapshot's ugly stepsister supporting the weirdness of reading from translate stores who we're hoping to off in the next season.

func (*Resource) Unlock

func (m *Resource) Unlock() error

Unlock releases the lock. This should be called if control of the underlying resource is being transitioned to another node. Ideally it's also called if the process crashes (e.g. via a defer), but an implementation based on filesystem locks should have those removed by the operating system when the process exits anyway.

type ResourceManager

type ResourceManager struct {
	Snapshotter computer.SnapshotService
	Writelogger computer.WritelogService
	Logger      logger.Logger
	// contains filtered or unexported fields
}

ResourceManager holds all the various Resources each of which is specific to a particular shard, table key partition or field, but all of which use the same underlying snapshotter and writelogger.

func NewResourceManager

func NewResourceManager(s computer.SnapshotService, w computer.WritelogService, l logger.Logger) *ResourceManager

func (*ResourceManager) GetFieldKeyResource

func (mm *ResourceManager) GetFieldKeyResource(qtid dax.QualifiedTableID, field dax.FieldName) *Resource

func (*ResourceManager) GetShardResource

func (mm *ResourceManager) GetShardResource(qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum) *Resource

func (*ResourceManager) GetTableKeyResource

func (mm *ResourceManager) GetTableKeyResource(qtid dax.QualifiedTableID, partition dax.PartitionNum) *Resource

func (*ResourceManager) RemoveAll

func (mm *ResourceManager) RemoveAll() error

RemoveAll unlocks and deletes all resources held within this ResourceManager. To be clear, the underlying resources are *not* deleted, they are simply no longer under this resource manager's purview. If there's a problem removing any resource, we'll save the error, but keep going... this is only called on shutdown, so we do our best to unlock everything.

func (*ResourceManager) RemoveFieldKeyResource

func (mm *ResourceManager) RemoveFieldKeyResource(qtid dax.QualifiedTableID, field dax.FieldName)

func (*ResourceManager) RemoveShardResource

func (mm *ResourceManager) RemoveShardResource(qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum)

func (*ResourceManager) RemoveTable

func (mm *ResourceManager) RemoveTable(qtid dax.QualifiedTableID) error

RemoveTable, unlocks and removes all resources related to the given table from this ResourceManager. The underlying files are not deleted. (If the table is being dropped, deleting the files is under the purview of metadata services). In this case (in contrast to RemoveAll), we'll stop on any error and return it. Not sure it actually matters in either of these cases.

func (*ResourceManager) RemoveTableKeyResource

func (mm *ResourceManager) RemoveTableKeyResource(qtid dax.QualifiedTableID, partition dax.PartitionNum)

type ShardReader

type ShardReader struct {
	// contains filtered or unexported fields
}

func NewShardReader

func NewShardReader(qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, writelog io.ReadCloser) *ShardReader

func (*ShardReader) Close

func (r *ShardReader) Close() error

func (*ShardReader) Read

func (r *ShardReader) Read() (computer.LogMessage, error)

type TableKeyReader

type TableKeyReader struct {
	// contains filtered or unexported fields
}

func NewTableKeyReader

func NewTableKeyReader(qtid dax.QualifiedTableID, partition dax.PartitionNum, writelog io.ReadCloser) *TableKeyReader

func (*TableKeyReader) Close

func (r *TableKeyReader) Close() error

func (*TableKeyReader) Read

func (r *TableKeyReader) Read() (computer.PartitionKeyMap, error)