engine

package
v1.22.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 29, 2022 License: GPL-3.0 Imports: 28 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrEndOfListing = shard.ErrEndOfListing

ErrEndOfListing is returned from an object listing with cursor when the storage can't return any more objects after the provided cursor. Use nil cursor object to start listing again.

Functions

func ContainerSize added in v0.15.0 

func ContainerSize(e *StorageEngine, id cid.ID) (uint64, error)

ContainerSize calls ContainerSize method on engine to calculate sum of estimation container sizes among all shards.

func Get 

func Get(storage *StorageEngine, addr oid.Address) (*objectSDK.Object, error)

Get reads object from local storage by provided address.

func GetRange 

func GetRange(storage *StorageEngine, addr oid.Address, rng *objectSDK.Range) ([]byte, error)

GetRange reads object payload range from local storage by provided address. 

func Head(storage *StorageEngine, addr oid.Address) (*objectSDK.Object, error)

Head reads object header from local storage by provided address.

func HeadRaw 

func HeadRaw(storage *StorageEngine, addr oid.Address, raw bool) (*objectSDK.Object, error)

HeadRaw reads object header from local storage by provided address and raw flag.

func List 

func List(storage *StorageEngine, limit uint64) ([]oid.Address, error)

List returns `limit` available physically storage object addresses in engine. If limit is zero, then returns all available object addresses.

func ListContainers added in v0.15.0 

func ListContainers(e *StorageEngine) ([]cid.ID, error)

ListContainers calls ListContainers method on engine to get a unique container IDs presented in the engine objects.

func Put 

func Put(storage *StorageEngine, obj *objectSDK.Object) error

Put writes provided object to local storage.

func Select 

func Select(storage *StorageEngine, cnr cid.ID, fs object.SearchFilters) ([]oid.Address, error)

Select selects objects from local storage using provided filters.

Types

type ContainerSizePrm added in v0.15.0 

type ContainerSizePrm struct {
	// contains filtered or unexported fields
}

ContainerSizePrm groups parameters of ContainerSize operation.

func (*ContainerSizePrm) SetContainerID added in v0.27.0 

func (p *ContainerSizePrm) SetContainerID(cnr cid.ID)

SetContainerID sets the identifier of the container to estimate the size.

type ContainerSizeRes added in v0.15.0 

type ContainerSizeRes struct {
	// contains filtered or unexported fields
}

ContainerSizeRes resulting values of ContainerSize operation.

func (ContainerSizeRes) Size added in v0.15.0 

func (r ContainerSizeRes) Size() uint64

Size returns calculated estimation of the container size.

type Cursor added in v0.27.0 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor is a type for continuous object listing.

type DeletePrm 

type DeletePrm struct {
	// contains filtered or unexported fields
}

DeletePrm groups the parameters of Delete operation.

func (*DeletePrm) WithAddress added in v0.33.0 

func (p *DeletePrm) WithAddress(addr oid.Address)

WithAddress is a Delete option to set the addresses of the objects to delete.

Option is required.

func (*DeletePrm) WithForceRemoval added in v0.30.0 

func (p *DeletePrm) WithForceRemoval()

WithForceRemoval is a Delete option to remove an object despite any restrictions imposed on deleting that object. Expected to be used only in control service.

type DeleteRes 

type DeleteRes struct{}

DeleteRes groups the resulting values of Delete operation.

type EvacuateShardPrm added in v0.33.0 

type EvacuateShardPrm struct {
	// contains filtered or unexported fields
}

EvacuateShardPrm represents parameters for the EvacuateShard operation.

func (*EvacuateShardPrm) WithFaultHandler added in v0.33.0 

func (p *EvacuateShardPrm) WithFaultHandler(f func(oid.Address, *objectSDK.Object) error)

WithFaultHandler sets handler to call for objects which cannot be saved on other shards.

func (*EvacuateShardPrm) WithIgnoreErrors added in v0.33.0 

func (p *EvacuateShardPrm) WithIgnoreErrors(ignore bool)

WithIgnoreErrors sets flag to ignore errors.

func (*EvacuateShardPrm) WithShardIDList added in v0.33.0 

func (p *EvacuateShardPrm) WithShardIDList(id []*shard.ID)

WithShardIDList sets shard ID.

type EvacuateShardRes added in v0.33.0 

type EvacuateShardRes struct {
	// contains filtered or unexported fields
}

EvacuateShardRes represents result of the EvacuateShard operation.

func (EvacuateShardRes) Count added in v0.33.0 

func (p EvacuateShardRes) Count() int

Count returns amount of evacuated objects. Objects for which handler returned no error are also assumed evacuated.

type FlushWriteCachePrm added in v0.33.0 

type FlushWriteCachePrm struct {
	// contains filtered or unexported fields
}

FlushWriteCachePrm groups the parameters of FlushWriteCache operation.

func (*FlushWriteCachePrm) SetIgnoreErrors added in v0.33.0 

func (p *FlushWriteCachePrm) SetIgnoreErrors(ignore bool)

SetIgnoreErrors sets errors ignore flag..

func (*FlushWriteCachePrm) SetShardID added in v0.33.0 

func (p *FlushWriteCachePrm) SetShardID(id *shard.ID)

SetShardID is an option to set shard ID.

Option is required.

type FlushWriteCacheRes added in v0.33.0 

type FlushWriteCacheRes struct{}

FlushWriteCacheRes groups the resulting values of FlushWriteCache operation.

type GetPrm 

type GetPrm struct {
	// contains filtered or unexported fields
}

GetPrm groups the parameters of Get operation.

func (*GetPrm) WithAddress 

func (p *GetPrm) WithAddress(addr oid.Address)

WithAddress is a Get option to set the address of the requested object.

Option is required.

type GetRes 

type GetRes struct {
	// contains filtered or unexported fields
}

GetRes groups the resulting values of Get operation.

func (GetRes) Object 

func (r GetRes) Object() *objectSDK.Object

Object returns the requested object.

type HeadPrm 

type HeadPrm struct {
	// contains filtered or unexported fields
}

HeadPrm groups the parameters of Head operation.

func (*HeadPrm) WithAddress 

func (p *HeadPrm) WithAddress(addr oid.Address)

WithAddress is a Head option to set the address of the requested object.

Option is required.

func (*HeadPrm) WithRaw 

func (p *HeadPrm) WithRaw(raw bool)

WithRaw is a Head option to set raw flag value. If flag is unset, then Head returns the header of the virtual object, otherwise it returns SplitInfo of the virtual object.

type HeadRes 

type HeadRes struct {
	// contains filtered or unexported fields
}

HeadRes groups the resulting values of Head operation.

func (HeadRes) Header 

func (r HeadRes) Header() *objectSDK.Object

Header returns the requested object header.

Instance has empty payload.

type Info 

type Info struct {
	Shards []shard.Info
}

Info groups the information about StorageEngine.

type InhumePrm 

type InhumePrm struct {
	// contains filtered or unexported fields
}

InhumePrm encapsulates parameters for inhume operation.

func (*InhumePrm) MarkAsGarbage added in v0.16.0 

func (p *InhumePrm) MarkAsGarbage(addrs ...oid.Address)

MarkAsGarbage marks an object to be physically removed from local storage.

Should not be called along with WithTarget.

func (*InhumePrm) WithForceRemoval added in v0.30.0 

func (p *InhumePrm) WithForceRemoval()

WithForceRemoval inhumes objects specified via MarkAsGarbage with GC mark without any object restrictions checks.

func (*InhumePrm) WithTarget 

func (p *InhumePrm) WithTarget(tombstone oid.Address, addrs ...oid.Address)

WithTarget sets a list of objects that should be inhumed and tombstone address as the reason for inhume operation.

tombstone should not be nil, addr should not be empty. Should not be called along with MarkAsGarbage.

type InhumeRes 

type InhumeRes struct{}

InhumeRes encapsulates results of inhume operation.

type ListContainersPrm added in v0.15.0 

type ListContainersPrm struct{}

ListContainersPrm groups parameters of ListContainers operation.

type ListContainersRes added in v0.15.0 

type ListContainersRes struct {
	// contains filtered or unexported fields
}

ListContainersRes groups the resulting values of ListContainers operation.

func (ListContainersRes) Containers added in v0.15.0 

func (r ListContainersRes) Containers() []cid.ID

Containers returns a list of identifiers of the containers in which local objects are stored.

type ListWithCursorPrm added in v0.27.0 

type ListWithCursorPrm struct {
	// contains filtered or unexported fields
}

ListWithCursorPrm contains parameters for ListWithCursor operation.

func (*ListWithCursorPrm) WithCount added in v0.27.0 

func (p *ListWithCursorPrm) WithCount(count uint32)

WithCount sets the maximum amount of addresses that ListWithCursor should return.

func (*ListWithCursorPrm) WithCursor added in v0.27.0 

func (p *ListWithCursorPrm) WithCursor(cursor *Cursor)

WithCursor sets a cursor for ListWithCursor operation. For initial request ignore this param or use nil value. For consecutive requests, use value from ListWithCursorRes.

type ListWithCursorRes added in v0.27.0 

type ListWithCursorRes struct {
	// contains filtered or unexported fields
}

ListWithCursorRes contains values returned from ListWithCursor operation.

func (ListWithCursorRes) AddressList added in v0.27.0 

func (l ListWithCursorRes) AddressList() []objectcore.AddressWithType

AddressList returns addresses selected by ListWithCursor operation.

func (ListWithCursorRes) Cursor added in v0.27.0 

func (l ListWithCursorRes) Cursor() *Cursor

Cursor returns cursor for consecutive listing requests.

type MetricRegister added in v0.17.0 

type MetricRegister interface {
	AddListContainersDuration(d time.Duration)
	AddEstimateContainerSizeDuration(d time.Duration)
	AddDeleteDuration(d time.Duration)
	AddExistsDuration(d time.Duration)
	AddGetDuration(d time.Duration)
	AddHeadDuration(d time.Duration)
	AddInhumeDuration(d time.Duration)
	AddPutDuration(d time.Duration)
	AddRangeDuration(d time.Duration)
	AddSearchDuration(d time.Duration)
	AddListObjectsDuration(d time.Duration)

	SetObjectCounter(shardID, objectType string, v uint64)
	AddToObjectCounter(shardID, objectType string, delta int)
}

type Option 

type Option func(*cfg)

Option represents StorageEngine's constructor option.

func WithErrorThreshold added in v0.28.0 

func WithErrorThreshold(sz uint32) Option

WithErrorThreshold returns an option to specify size amount of errors after which shard is moved to read-only mode.

func WithLogger 

func WithLogger(l *logger.Logger) Option

WithLogger returns option to set StorageEngine's logger.

func WithMetrics added in v0.17.0 

func WithMetrics(v MetricRegister) Option

func WithShardPoolSize added in v0.26.0 

func WithShardPoolSize(sz uint32) Option

WithShardPoolSize returns option to specify size of worker pool for each shard.

type PutPrm 

type PutPrm struct {
	// contains filtered or unexported fields
}

PutPrm groups the parameters of Put operation.

func (*PutPrm) WithObject 

func (p *PutPrm) WithObject(obj *objectSDK.Object)

WithObject is a Put option to set object to save.

Option is required.

type PutRes 

type PutRes struct{}

PutRes groups the resulting values of Put operation.

type ReConfiguration added in v0.33.0 

type ReConfiguration struct {
	// contains filtered or unexported fields
}

func (*ReConfiguration) AddShard added in v0.33.0 

func (rCfg *ReConfiguration) AddShard(id string, opts []shard.Option)

AddShard adds a shard for the reconfiguration. Shard identifier is calculated from paths used in blobstor.

func (*ReConfiguration) SetErrorsThreshold added in v0.33.0 

func (rCfg *ReConfiguration) SetErrorsThreshold(errorsThreshold uint32)

SetErrorsThreshold sets a size amount of errors after which shard is moved to read-only mode.

func (*ReConfiguration) SetShardPoolSize added in v0.33.0 

func (rCfg *ReConfiguration) SetShardPoolSize(shardPoolSize uint32)

SetShardPoolSize sets a size of worker pool for each shard.

type RngPrm 

type RngPrm struct {
	// contains filtered or unexported fields
}

RngPrm groups the parameters of GetRange operation.

func (*RngPrm) WithAddress 

func (p *RngPrm) WithAddress(addr oid.Address)

WithAddress is a GetRng option to set the address of the requested object.

Option is required.

func (*RngPrm) WithPayloadRange 

func (p *RngPrm) WithPayloadRange(rng *objectSDK.Range)

WithPayloadRange is a GetRange option to set range of requested payload data.

Missing an option or calling with zero length is equivalent to getting the full payload range.

type RngRes 

type RngRes struct {
	// contains filtered or unexported fields
}

RngRes groups the resulting values of GetRange operation.

func (RngRes) Object 

func (r RngRes) Object() *objectSDK.Object

Object returns the requested object part.

Instance payload contains the requested range of the original object.

type SelectPrm 

type SelectPrm struct {
	// contains filtered or unexported fields
}

SelectPrm groups the parameters of Select operation.

func (*SelectPrm) WithContainerID 

func (p *SelectPrm) WithContainerID(cnr cid.ID)

WithContainerID is a Select option to set the container id to search in.

func (*SelectPrm) WithFilters 

func (p *SelectPrm) WithFilters(fs object.SearchFilters)

WithFilters is a Select option to set the object filters.

type SelectRes 

type SelectRes struct {
	// contains filtered or unexported fields
}

SelectRes groups the resulting values of Select operation.

func (SelectRes) AddressList 

func (r SelectRes) AddressList() []oid.Address

AddressList returns list of addresses of the selected objects.

type StorageEngine 

type StorageEngine struct {
	// contains filtered or unexported fields
}

StorageEngine represents NeoFS local storage engine.

func New 

func New(opts ...Option) *StorageEngine

New creates, initializes and returns new StorageEngine instance.

func (*StorageEngine) AddShard 

func (e *StorageEngine) AddShard(opts ...shard.Option) (*shard.ID, error)

AddShard adds a new shard to the storage engine.

Returns any error encountered that did not allow adding a shard. Otherwise returns the ID of the added shard.

func (*StorageEngine) BlockExecution added in v0.27.0 

func (e *StorageEngine) BlockExecution(err error) error

BlockExecution blocks the execution of any data-related operation. All blocked ops will return err. To resume the execution, use ResumeExecution method.

Сan be called regardless of the fact of the previous blocking. If execution wasn't blocked, releases all resources similar to Close. Can be called concurrently with Close and any data related method (waits for all executions to complete). Returns error if any Close has been called before.

Must not be called concurrently with either Open or Init.

Note: technically passing nil error will resume the execution, otherwise, it is recommended to call ResumeExecution for this.

func (*StorageEngine) Close 

func (e *StorageEngine) Close() error

Close releases all StorageEngine's components. Waits for all data-related operations to complete. After the call, all the next ones will fail.

The method MUST only be called when the application exits.

func (*StorageEngine) ContainerSize added in v0.15.0 

func (e *StorageEngine) ContainerSize(prm ContainerSizePrm) (res ContainerSizeRes, err error)

ContainerSize returns the sum of estimation container sizes among all shards.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Delete 

func (e *StorageEngine) Delete(prm DeletePrm) (res DeleteRes, err error)

Delete marks the objects to be removed.

Returns an error if executions are blocked (see BlockExecution).

Returns apistatus.ObjectLocked if at least one object is locked. In this case no object from the list is marked to be deleted.

NOTE: Marks any object to be deleted (despite any prohibitions on operations with that object) if WithForceRemoval option has been provided.

func (*StorageEngine) DumpInfo 

func (e *StorageEngine) DumpInfo() (i Info)

DumpInfo returns information about the StorageEngine.

func (*StorageEngine) DumpShard added in v0.27.5 

func (e *StorageEngine) DumpShard(id *shard.ID, prm shard.DumpPrm) error

DumpShard dumps objects from the shard with provided identifier.

Returns an error if shard is not read-only.

func (*StorageEngine) Evacuate added in v0.33.0 

func (e *StorageEngine) Evacuate(prm EvacuateShardPrm) (EvacuateShardRes, error)

Evacuate moves data from one shard to the others. The shard being moved must be in read-only mode.

func (*StorageEngine) FlushWriteCache added in v0.33.0 

func (e *StorageEngine) FlushWriteCache(p FlushWriteCachePrm) (FlushWriteCacheRes, error)

FlushWriteCache flushes write-cache on a single shard.

func (*StorageEngine) Get 

func (e *StorageEngine) Get(prm GetPrm) (res GetRes, err error)

Get reads an object from local storage.

Returns any error encountered that did not allow to completely read the object part.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the object has been marked as removed.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) GetRange 

func (e *StorageEngine) GetRange(prm RngPrm) (res RngRes, err error)

GetRange reads part of an object from local storage.

Returns any error encountered that did not allow to completely read the object part.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the requested object is inhumed. Returns ErrRangeOutOfBounds if the requested object range is out of bounds.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) HandleNewEpoch added in v0.33.0 

func (e *StorageEngine) HandleNewEpoch(epoch uint64)

HandleNewEpoch notifies every shard about NewEpoch event.

func (*StorageEngine) Head 

func (e *StorageEngine) Head(prm HeadPrm) (res HeadRes, err error)

Head reads object header from local storage.

Returns any error encountered that did not allow to completely read the object header.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the requested object was inhumed.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Inhume 

func (e *StorageEngine) Inhume(prm InhumePrm) (res InhumeRes, err error)

Inhume calls metabase. Inhume method to mark an object as removed. It won't be removed physically from the shard until `Delete` operation.

Allows inhuming non-locked objects only. Returns apistatus.ObjectLocked if at least one object is locked.

NOTE: Marks any object as removed (despite any prohibitions on operations with that object) if WithForceRemoval option has been provided.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Init 

func (e *StorageEngine) Init() error

Init initializes all StorageEngine's components.

func (*StorageEngine) List 

func (e *StorageEngine) List(limit uint64) (res SelectRes, err error)

List returns `limit` available physically storage object addresses in engine. If limit is zero, then returns all available object addresses.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) ListContainers added in v0.15.0 

func (e *StorageEngine) ListContainers(_ ListContainersPrm) (res ListContainersRes, err error)

ListContainers returns a unique container IDs presented in the engine objects.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) ListWithCursor added in v0.27.0 

func (e *StorageEngine) ListWithCursor(prm ListWithCursorPrm) (ListWithCursorRes, error)

ListWithCursor lists physical objects available in the engine starting from the cursor. It includes regular, tombstone and storage group objects. Does not include inhumed objects. Use cursor value from the response for consecutive requests.

Returns ErrEndOfListing if there are no more objects to return or count parameter set to zero.

func (*StorageEngine) Lock added in v0.28.0 

func (e *StorageEngine) Lock(idCnr cid.ID, locker oid.ID, locked []oid.ID) error

Lock marks objects as locked with another object. All objects from the specified container.

Allows locking regular objects only (otherwise returns apistatus.LockNonRegularObject).

Locked list should be unique. Panics if it is empty.

func (*StorageEngine) Open 

func (e *StorageEngine) Open() error

Open opens all StorageEngine's components.

func (*StorageEngine) Put 

func (e *StorageEngine) Put(prm PutPrm) (res PutRes, err error)

Put saves the object to local storage.

Returns any error encountered that did not allow to completely save the object.

Returns an error if executions are blocked (see BlockExecution).

Returns an error of type apistatus.ObjectAlreadyRemoved if the object has been marked as removed.

func (*StorageEngine) Reload added in v0.33.0 

func (e *StorageEngine) Reload(rcfg ReConfiguration) error

Reload reloads StorageEngine's configuration in runtime.

func (*StorageEngine) RestoreShard added in v0.27.5 

func (e *StorageEngine) RestoreShard(id *shard.ID, prm shard.RestorePrm) error

RestoreShard restores objects from dump to the shard with provided identifier.

Returns an error if shard is not read-only.

func (*StorageEngine) ResumeExecution added in v0.27.0 

func (e *StorageEngine) ResumeExecution() error

ResumeExecution resumes the execution of any data-related operation. To block the execution, use BlockExecution method.

Сan be called regardless of the fact of the previous blocking. If execution was blocked, prepares all resources similar to Open. Can be called concurrently with Close and any data related method (waits for all executions to complete). Returns error if any Close has been called before.

Must not be called concurrently with either Open or Init.

func (*StorageEngine) Select 

func (e *StorageEngine) Select(prm SelectPrm) (res SelectRes, err error)

Select selects the objects from local storage that match select parameters.

Returns any error encountered that did not allow to completely select the objects.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) SetShardMode 

func (e *StorageEngine) SetShardMode(id *shard.ID, m mode.Mode, resetErrorCounter bool) error

SetShardMode sets mode of the shard with provided identifier.

Returns an error if shard mode was not set, or shard was not found in storage engine.

func (*StorageEngine) TreeAddByPath added in v0.30.0 

func (e *StorageEngine) TreeAddByPath(d pilorama.CIDDescriptor, treeID string, attr string, path []string, m []pilorama.KeyValue) ([]pilorama.LogMove, error)

TreeAddByPath implements the pilorama.Forest interface.

func (*StorageEngine) TreeApply added in v0.30.0 

func (e *StorageEngine) TreeApply(d pilorama.CIDDescriptor, treeID string, m *pilorama.Move) error

TreeApply implements the pilorama.Forest interface.

func (*StorageEngine) TreeDrop added in v0.32.0 

func (e *StorageEngine) TreeDrop(cid cidSDK.ID, treeID string) error

TreeDrop implements the pilorama.Forest interface.

func (*StorageEngine) TreeExists added in v0.36.0 

func (e *StorageEngine) TreeExists(cid cidSDK.ID, treeID string) (bool, error)

TreeExists implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetByPath added in v0.30.0 

func (e *StorageEngine) TreeGetByPath(cid cidSDK.ID, treeID string, attr string, path []string, latest bool) ([]pilorama.Node, error)

TreeGetByPath implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetChildren added in v0.30.0 

func (e *StorageEngine) TreeGetChildren(cid cidSDK.ID, treeID string, nodeID pilorama.Node) ([]uint64, error)

TreeGetChildren implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetMeta added in v0.30.0 

func (e *StorageEngine) TreeGetMeta(cid cidSDK.ID, treeID string, nodeID pilorama.Node) (pilorama.Meta, uint64, error)

TreeGetMeta implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetOpLog added in v0.30.0 

func (e *StorageEngine) TreeGetOpLog(cid cidSDK.ID, treeID string, height uint64) (pilorama.Move, error)

TreeGetOpLog implements the pilorama.Forest interface.

func (*StorageEngine) TreeList added in v0.34.0 

func (e *StorageEngine) TreeList(cid cidSDK.ID) ([]string, error)

TreeList implements the pilorama.Forest interface.

func (*StorageEngine) TreeMove added in v0.30.0 

func (e *StorageEngine) TreeMove(d pilorama.CIDDescriptor, treeID string, m *pilorama.Move) (*pilorama.LogMove, error)

TreeMove implements the pilorama.Forest interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.