localstore

Documentation

Overview

Package localstore provides disk storage layer for Swarm Chunk persistence. It uses swarm/shed abstractions on top of github.com/syndtr/goleveldb LevelDB implementation.

The main type is DB which manages the storage by providing methods to access and add Chunks and to manage their status.

Modes are abstractions that do specific changes to Chunks. There are three mode types:

• ModeGet, for Chunk access
• ModePut, for adding Chunks to the database
• ModeSet, for changing Chunk statuses

Every mode type has a corresponding type (Getter, Putter and Setter) that provides adequate method to perform the opperation and that type should be injected into localstore consumers instead the whole DB. This provides more clear insight which operations consumer is performing on the database.

Getters, Putters and Setters accept different get, put and set modes to perform different actions. For example, ModeGet has two different variables ModeGetRequest and ModeGetSync and two different Getters can be constructed with them that are used when the chunk is requested or when the chunk is synced as this two events are differently changing the database.

Subscription methods are implemented for a specific purpose of continuous iterations over Chunks that should be provided to Push and Pull syncing.

DB implements an internal garbage collector that removes only synced Chunks from the database based on their most recent access time.

Internally, DB stores Chunk data and any required information, such as store and access timestamps in different shed indexes that can be iterated on by garbage collector or subscriptions.

Index

• Variables
• type ChunkDescriptor
  • func (c *ChunkDescriptor) String() string
• type DB
  • func New(path string, baseKey []byte, o *Options) (db *DB, err error)
  • func (db *DB) Close() (err error)
  • func (db *DB) LastPullSubscriptionChunk(bin uint8) (c *ChunkDescriptor, err error)
  • func (db *DB) NewGetter(mode ModeGet) *Getter
  • func (db *DB) NewPutter(mode ModePut) *Putter
  • func (db *DB) NewSetter(mode ModeSet) *Setter
  • func (db *DB) SubscribePull(ctx context.Context, bin uint8, since, until *ChunkDescriptor) (c <-chan ChunkDescriptor, stop func())
  • func (db *DB) SubscribePush(ctx context.Context) (c <-chan chunk.Chunk, stop func())
• type Getter
  • func (g *Getter) Get(addr chunk.Address) (ch chunk.Chunk, err error)
• type ModeGet
• type ModePut
• type ModeSet
• type Options
• type Putter
  • func (p *Putter) Put(ch chunk.Chunk) (err error)
• type Setter
  • func (s *Setter) Set(addr chunk.Address) (err error)

Variables

var (
	// ErrInvalidMode is retuned when an unknown Mode
	// is provided to the function.
	ErrInvalidMode = errors.New("invalid mode")
	// ErrAddressLockTimeout is returned when the same chunk
	// is updated in parallel and one of the updates
	// takes longer then the configured timeout duration.
	ErrAddressLockTimeout = errors.New("address lock timeout")
)

Types

type ChunkDescriptor

type ChunkDescriptor struct {
	Address        chunk.Address
	StoreTimestamp int64
}

ChunkDescriptor holds information required for Pull syncing. This struct is provided by subscribing to pull index.

func (*ChunkDescriptor) String

func (c *ChunkDescriptor) String() string

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is the local store implementation and holds database related objects.

func New

func New(path string, baseKey []byte, o *Options) (db *DB, err error)

New returns a new DB. All fields and indexes are initialized and possible conflicts with schema from existing database is checked. One goroutine for writing batches is created.

func (*DB) Close

func (db *DB) Close() (err error)

Close closes the underlying database.

func (*DB) LastPullSubscriptionChunk

func (db *DB) LastPullSubscriptionChunk(bin uint8) (c *ChunkDescriptor, err error)

LastPullSubscriptionChunk returns ChunkDescriptor of the latest Chunk in pull syncing index for a provided bin. If there are no chunks in that bin, chunk.ErrChunkNotFound is returned.

func (*DB) NewGetter

func (db *DB) NewGetter(mode ModeGet) *Getter

NewGetter returns a new Getter on database with a specific Mode.

func (*DB) NewPutter

func (db *DB) NewPutter(mode ModePut) *Putter

NewPutter returns a new Putter on database with a specific Mode.

func (*DB) NewSetter

func (db *DB) NewSetter(mode ModeSet) *Setter

NewSetter returns a new Setter on database with a specific Mode.

func (*DB) SubscribePull

func (db *DB) SubscribePull(ctx context.Context, bin uint8, since, until *ChunkDescriptor) (c <-chan ChunkDescriptor, stop func())

SubscribePull returns a channel that provides chunk addresses and stored times from pull syncing index. Pull syncing index can be only subscribed to a particular proximity order bin. If since is not nil, the iteration will start from the first item stored after that timestamp. If until is not nil, only chunks stored up to this timestamp will be send to the channel, and the returned channel will be closed. The since-until interval is open on the left and closed on the right (since,until]. Returned stop function will terminate current and further iterations without errors, and also close the returned channel. Make sure that you check the second returned parameter from the channel to stop iteration when its value is false.

func (*DB) SubscribePush

func (db *DB) SubscribePush(ctx context.Context) (c <-chan chunk.Chunk, stop func())

SubscribePush returns a channel that provides storage chunks with ordering from push syncing index. Returned stop function will terminate current and further iterations, and also it will close the returned channel without any errors. Make sure that you check the second returned parameter from the channel to stop iteration when its value is false.

type Getter

type Getter struct {
	// contains filtered or unexported fields
}

Getter provides Get method to retrieve Chunks from database.

func (*Getter) Get

func (g *Getter) Get(addr chunk.Address) (ch chunk.Chunk, err error)

Get returns a chunk from the database. If the chunk is not found chunk.ErrChunkNotFound will be returned. All required indexes will be updated required by the Getter Mode.

type ModeGet

type ModeGet int

ModeGet enumerates different Getter modes.

const (
	// ModeGetRequest: when accessed for retrieval
	ModeGetRequest ModeGet = iota
	// ModeGetSync: when accessed for syncing or proof of custody request
	ModeGetSync
)

Getter modes.

type ModePut

type ModePut int

ModePut enumerates different Putter modes.

const (
	// ModePutRequest: when a chunk is received as a result of retrieve request and delivery
	ModePutRequest ModePut = iota
	// ModePutSync: when a chunk is received via syncing
	ModePutSync
	// ModePutUpload: when a chunk is created by local upload
	ModePutUpload
)

Putter modes.

type ModeSet

type ModeSet int

ModeSet enumerates different Setter modes.

const (
	// ModeSetAccess: when an update request is received for a chunk or chunk is retrieved for delivery
	ModeSetAccess ModeSet = iota
	// ModeSetSync: when push sync receipt is received
	ModeSetSync
)

Setter modes.

type Options

type Options struct {
	// MockStore is a mock node store that is used to store
	// chunk data in a central store. It can be used to reduce
	// total storage space requirements in testing large number
	// of swarm nodes with chunk data deduplication provided by
	// the mock global store.
	MockStore *mock.NodeStore
	// Capacity is a limit that triggers garbage collection when
	// number of items in gcIndex equals or exceeds it.
	Capacity int64
	// MetricsPrefix defines a prefix for metrics names.
	MetricsPrefix string
}

Options struct holds optional parameters for configuring DB.

type Putter

type Putter struct {
	// contains filtered or unexported fields
}

Putter provides Put method to store Chunks to database.

func (*Putter) Put

func (p *Putter) Put(ch chunk.Chunk) (err error)

Put stores the Chunk to database and depending on the Putter mode, it updates required indexes.

type Setter

type Setter struct {
	// contains filtered or unexported fields
}

Setter sets the state of a particular Chunk in database by changing indexes.

func (*Setter) Set

func (s *Setter) Set(addr chunk.Address) (err error)

Set updates database indexes for a specific chunk represented by the address.