storage

package
v0.0.0-...-cdd27d5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 7, 2019 License: GPL-3.0 Imports: 15 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	C_READ = iota
	C_READBYTES
	C_WRITE
	C_WRITEBYTES
	C_DELETE
	NUM_C
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Backend 

type Backend interface {
	// Initialize the backend with the given configuration; this
	// is typically called only for real storage backends and not
	// interim ones (e.g. mapRunnerBackend, codecBackend)
	Init(config BackendConfiguration)

	// Flush is used to hint that currently is good time to
	// snapshot state, if any; storage is done with flushing its
	// current state so e.g. names and block hierarchies are most
	// consistent right now
	Flush()

	// Close the backend
	Close()

	// GetBytesAvailable returns number of bytes available.
	GetBytesAvailable() uint64

	// GetBytesUsed returns number of bytes used.
	GetBytesUsed() uint64

	// Supports can be used to query if a backend (chain) supports
	// a particular feature
	Supports(feature BackendFeature) bool

	BlockBackend
	NameBackend
}

Backend is the shadow behind the throne; it actually handles the low-level operations of blocks. It provides an API that returns results that are consistent with the previous calls. How it does this in practise is left as an exercise to the implementor. There is no guarantee it will not be called from multiple goroutines at once, and it is again the problem of the implementor to ensure that the results are consistent.

type BackendConfiguration 

type BackendConfiguration struct {
	// How much delay should there per asynchronous operation
	// (useful only for testing)
	DelayPerOp time.Duration

	// Directory to be used for storing backend data
	Directory string

	// ValueUpdateInterval describes how often cached values (e.g.
	// statfs stuff) are updated.
	ValueUpdateInterval time.Duration

	// Codec (if the backend actually handles the codec directly,
	// as opposed to using chaining codec like the rest)
	Codec codec.Codec

	// CacheSize (if any) in number of disk pages
	CacheSize int

	// Unsafe mode (if possible) ; non-sync writes mostly
	Unsafe bool
}

type BackendFeature 

type BackendFeature int
const (
	CodecFeature BackendFeature = iota
)

type Block 

type Block struct {
	BlockMetadata // contains RefCount, Status

	// Id contains identity of the block, derived from Data if not
	// set.
	Id string

	// Actually plaintext data (if available; GetData() should be
	// used to get it always when accessing from outside backends
	// as it may not be set at that point otherwise).
	Data util.ByteSliceAtomicPointer

	// Backend this is fetched from, if any
	Backend Backend

	// Stored version of the block metadata, if any. Set only if
	// something has changed locally. For fresh blocks, is nil.
	Stored *BlockMetadata
	// contains filtered or unexported fields
}

Block is abstraction used between Storage and its Backends.

func (*Block) GetData 

func (self *Block) GetData() []byte

func (*Block) String 

func (self *Block) String() string

type BlockBackend 

type BlockBackend interface {
	// GetBlockData retrieves lazily (if need be) block data
	GetBlockData(b *Block) []byte

	// GetBlockById returns block by id or nil.
	GetBlockById(id string) *Block

	// DeleteBlock removes block from storage, and it MUST exist.
	DeleteBlock(b *Block)

	// StoreBlock adds new block to  It MUST NOT exist.
	StoreBlock(b *Block)

	// UpdateBlock updates block metadata in  It MUST exist.
	UpdateBlock(b *Block) int
}

BlockBackend is subset of the storage Backend which deals with raw named + reference counted blocks.

type BlockIterateReferencesCallback 

type BlockIterateReferencesCallback func(string, []byte, BlockReferenceCallback)

type BlockMetadata 

type BlockMetadata struct {
	// RefCount is the non-negative number of references to a
	// block _on disk_ (or what should be on disk).
	RefCount int32 `zid:"0"`

	// Status describes the desired behavior of sub-references and
	// availability of data of a block.
	Status BlockStatus `zid:"1"`
}

func (*BlockMetadata) DecodeMsg 

func (z *BlockMetadata) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable We treat empty fields as if we read a Nil from the wire.

func (BlockMetadata) EncodeMsg 

func (z BlockMetadata) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (BlockMetadata) MarshalMsg 

func (z BlockMetadata) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (BlockMetadata) Msgsize 

func (z BlockMetadata) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*BlockMetadata) UnmarshalMsg 

func (z *BlockMetadata) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

func (*BlockMetadata) UnmarshalMsgWithCfg 

func (z *BlockMetadata) UnmarshalMsgWithCfg(bts []byte, cfg *msgp.RuntimeConfig) (o []byte, err error)

type BlockPointerFuture 

type BlockPointerFuture struct {
	ValueCallback BlockPointerFutureValueCallback
	// contains filtered or unexported fields
}

BlockPointerFuture is a value-based Future object. Threadsafe, and multiple Gets do work as expected unlike some other implementations. There are two ways to set this up: provide SetCallback, or call Set by hand at some point.

func (*BlockPointerFuture) Get 

func (self *BlockPointerFuture) Get() *Block

func (*BlockPointerFuture) Set 

func (self *BlockPointerFuture) Set(v *Block)

func (*BlockPointerFuture) SetCallback 

func (self *BlockPointerFuture) SetCallback(cb BlockPointerFutureValueCallback) *BlockPointerFuture

type BlockPointerFutureValueCallback 

type BlockPointerFutureValueCallback func() *Block

type BlockReferenceCallback 

type BlockReferenceCallback func(string)

type BlockStatus 

type BlockStatus byte
const (
	BS_UNSET BlockStatus = iota

	// Has references on based on data, data present
	BS_NORMAL

	// Has references on based on data, data gone
	BS_MISSING

	// No references on based on data, no data
	BS_WANT_NORMAL

	// No references on based on data, data present
	BS_WEAK

	// No references on based on data, no data
	BS_WANT_WEAK
)

func (*BlockStatus) DecodeMsg 

func (z *BlockStatus) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable We treat empty fields as if we read a Nil from the wire.

func (BlockStatus) EncodeMsg 

func (z BlockStatus) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (BlockStatus) MarshalMsg 

func (z BlockStatus) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (BlockStatus) Msgsize 

func (z BlockStatus) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*BlockStatus) UnmarshalMsg 

func (z *BlockStatus) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

func (*BlockStatus) UnmarshalMsgWithCfg 

func (z *BlockStatus) UnmarshalMsgWithCfg(bts []byte, cfg *msgp.RuntimeConfig) (o []byte, err error)

type DirectoryBackendBase 

type DirectoryBackendBase struct {
	BackendConfiguration
	// contains filtered or unexported fields
}

func (*DirectoryBackendBase) GetBytesAvailable 

func (self *DirectoryBackendBase) GetBytesAvailable() uint64

func (*DirectoryBackendBase) GetBytesUsed 

func (self *DirectoryBackendBase) GetBytesUsed() uint64

func (*DirectoryBackendBase) Init 

func (self *DirectoryBackendBase) Init(config BackendConfiguration)

type NameBackend 

type NameBackend interface {

	// GetBlockIdByName returns block id mapped to particular name.
	GetBlockIdByName(name string) string

	// SetBlockIdName sets the logical name to map to particular block id.
	SetNameToBlockId(name, block_id string)
}

NameBackend is subset of storage Backend which deals with names.

type NameInBlockBackend 

type NameInBlockBackend struct {
	// contains filtered or unexported fields
}

NameInBlockBackend saves names in a single block. Unfortunately that design is somewhat vulnerable (as there is a time during which there are 0 name-blocks in db), so probably better design would be to use name-prefix and generation count. TBD, watch this space.

func (*NameInBlockBackend) GetBlockIdByName 

func (self *NameInBlockBackend) GetBlockIdByName(name string) string

func (*NameInBlockBackend) Init 

func (self *NameInBlockBackend) Init(mapName string, bb BlockBackend)

func (*NameInBlockBackend) SetNameToBlockId 

func (self *NameInBlockBackend) SetNameToBlockId(name, block_id string)

type NameMapBlock 

type NameMapBlock struct {
	NameToBlockId map[string]string `zid:"0"`
}

func (*NameMapBlock) DecodeMsg 

func (z *NameMapBlock) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable We treat empty fields as if we read a Nil from the wire.

func (*NameMapBlock) EncodeMsg 

func (z *NameMapBlock) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*NameMapBlock) MarshalMsg 

func (z *NameMapBlock) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*NameMapBlock) Msgsize 

func (z *NameMapBlock) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*NameMapBlock) UnmarshalMsg 

func (z *NameMapBlock) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

func (*NameMapBlock) UnmarshalMsgWithCfg 

func (z *NameMapBlock) UnmarshalMsgWithCfg(bts []byte, cfg *msgp.RuntimeConfig) (o []byte, err error)

type Storage 

type Storage struct {
	// Backend specifies the backend to use.
	Backend Backend

	// IterateReferencesCallback is used to find block references
	// inside block data.
	IterateReferencesCallback BlockIterateReferencesCallback

	// QueueLength specifies how long channel queue is there for
	// storage operations. Zero means unbuffered (which is fancy
	// for debugging but crap performance-wise).
	QueueLength int

	// Codec (if set) specifies the codec used to encode the data
	// before it is stored in backend, or to decode it when
	// fetching it from backend
	Codec codec.Codec
	// contains filtered or unexported fields
}

func (*Storage) Close 

func (self *Storage) Close()

func (*Storage) Flush 

func (self *Storage) Flush()

func (*Storage) GetBlockById 

func (self *Storage) GetBlockById(id string) *StorageBlock

func (*Storage) GetBlockIdByName 

func (self *Storage) GetBlockIdByName(name string) string

func (Storage) Init 

func (self Storage) Init() *Storage

Init sets up the default values to be usable

func (*Storage) ReferBlockId 

func (self *Storage) ReferBlockId(id string)

func (*Storage) ReferOrStoreBlock 

func (self *Storage) ReferOrStoreBlock(id string, status BlockStatus, data []byte) *StorageBlock

func (*Storage) ReferOrStoreBlock0 

func (self *Storage) ReferOrStoreBlock0(id string, status BlockStatus, data []byte, deps *util.StringList) *StorageBlock

func (*Storage) ReferOrStoreBlockBytes0 

func (self *Storage) ReferOrStoreBlockBytes0(status BlockStatus, b []byte, deps *util.StringList) *StorageBlock

func (*Storage) ReferStorageBlockId 

func (self *Storage) ReferStorageBlockId(id string)

func (*Storage) ReleaseBlockId 

func (self *Storage) ReleaseBlockId(id string)

func (*Storage) ReleaseStorageBlockId 

func (self *Storage) ReleaseStorageBlockId(id string)

func (*Storage) SetNameToBlockId 

func (self *Storage) SetNameToBlockId(name, block_id string)

func (*Storage) StoreBlock 

func (self *Storage) StoreBlock(id string, status BlockStatus, data []byte) *StorageBlock

func (*Storage) StoreBlock0 

func (self *Storage) StoreBlock0(id string, status BlockStatus, data []byte) *StorageBlock

func (*Storage) TransientCount 

func (self *Storage) TransientCount() int

type StorageBlock 

type StorageBlock struct {
	// contains filtered or unexported fields
}

StorageBlock is the public read-only view of a block to the outside world. All provided methods are synchronous, and actually cause changes to be propagated to Storage (eventually) if need be.

func (*StorageBlock) Close 

func (self *StorageBlock) Close()

func (*StorageBlock) Data 

func (self *StorageBlock) Data() []byte

func (*StorageBlock) Id 

func (self *StorageBlock) Id() string

func (*StorageBlock) IterateReferences 

func (self *StorageBlock) IterateReferences(cb func(id string))

func (*StorageBlock) Open 

func (self *StorageBlock) Open() *StorageBlock

func (*StorageBlock) SetStatus 

func (self *StorageBlock) SetStatus(status BlockStatus) bool

func (*StorageBlock) Status 

func (self *StorageBlock) Status() BlockStatus

func (*StorageBlock) String 

func (self *StorageBlock) String() string

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.