engine

package
v0.28.0-rc.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 8, 2022 License: GPL-3.0 Imports: 22 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrEndOfListing = shard.ErrEndOfListing

ErrEndOfListing is returned from object listing with cursor when storage can't return any more objects after provided cursor. Use nil cursor object to start listing again.

Functions

func ContainerSize added in v0.15.0 

func ContainerSize(e *StorageEngine, id *cid.ID) (uint64, error)

ContainerSize calls ContainerSize method on engine to calculate sum of estimation container sizes among all shards.

func Get 

func Get(storage *StorageEngine, addr *addressSDK.Address) (*objectSDK.Object, error)

Get reads object from local storage by provided address.

func GetRange 

func GetRange(storage *StorageEngine, addr *addressSDK.Address, rng *objectSDK.Range) ([]byte, error)

GetRange reads object payload range from local storage by provided address. 

func Head(storage *StorageEngine, addr *addressSDK.Address) (*objectSDK.Object, error)

Head reads object header from local storage by provided address.

func HeadRaw 

func HeadRaw(storage *StorageEngine, addr *addressSDK.Address, raw bool) (*objectSDK.Object, error)

HeadRaw reads object header from local storage by provided address and raw flag.

func List 

func List(storage *StorageEngine, limit uint64) ([]*addressSDK.Address, error)

List returns `limit` available physically storage object addresses in engine. If limit is zero, then returns all available object addresses.

func ListContainers added in v0.15.0 

func ListContainers(e *StorageEngine) ([]*cid.ID, error)

ListContainers calls ListContainers method on engine to get unique container IDs presented in the engine objects.

func Put 

func Put(storage *StorageEngine, obj *objectSDK.Object) error

Put writes provided object to local storage.

func Select 

func Select(storage *StorageEngine, cid *cid.ID, fs object.SearchFilters) ([]*addressSDK.Address, error)

Select selects objects from local storage using provided filters.

Types

type ContainerSizePrm added in v0.15.0 

type ContainerSizePrm struct {
	// contains filtered or unexported fields
}

ContainerSizePrm groups parameters of ContainerSize operation.

func (*ContainerSizePrm) SetContainerID added in v0.27.0 

func (p *ContainerSizePrm) SetContainerID(cid *cid.ID)

SetContainerID sets identifier of the container to estimate the size.

type ContainerSizeRes added in v0.15.0 

type ContainerSizeRes struct {
	// contains filtered or unexported fields
}

ContainerSizeRes resulting values of ContainerSize operation.

func (ContainerSizeRes) Size added in v0.15.0 

func (r ContainerSizeRes) Size() uint64

Size returns calculated estimation of the container size.

type Cursor added in v0.27.0 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor is a type for continuous object listing.

type DeletePrm 

type DeletePrm struct {
	// contains filtered or unexported fields
}

DeletePrm groups the parameters of Delete operation.

func (*DeletePrm) WithAddresses 

func (p *DeletePrm) WithAddresses(addr ...*addressSDK.Address) *DeletePrm

WithAddresses is a Delete option to set the addresses of the objects to delete.

Option is required.

type DeleteRes 

type DeleteRes struct{}

DeleteRes groups resulting values of Delete operation.

type GetPrm 

type GetPrm struct {
	// contains filtered or unexported fields
}

GetPrm groups the parameters of Get operation.

func (*GetPrm) WithAddress 

func (p *GetPrm) WithAddress(addr *addressSDK.Address) *GetPrm

WithAddress is a Get option to set the address of the requested object.

Option is required.

type GetRes 

type GetRes struct {
	// contains filtered or unexported fields
}

GetRes groups resulting values of Get operation.

func (*GetRes) Object 

func (r *GetRes) Object() *objectSDK.Object

Object returns the requested object.

type HeadPrm 

type HeadPrm struct {
	// contains filtered or unexported fields
}

HeadPrm groups the parameters of Head operation.

func (*HeadPrm) WithAddress 

func (p *HeadPrm) WithAddress(addr *addressSDK.Address) *HeadPrm

WithAddress is a Head option to set the address of the requested object.

Option is required.

func (*HeadPrm) WithRaw 

func (p *HeadPrm) WithRaw(raw bool) *HeadPrm

WithRaw is a Head option to set raw flag value. If flag is unset, then Head returns header of virtual object, otherwise it returns SplitInfo of virtual object.

type HeadRes 

type HeadRes struct {
	// contains filtered or unexported fields
}

HeadRes groups resulting values of Head operation.

func (*HeadRes) Header 

func (r *HeadRes) Header() *objectSDK.Object

Header returns the requested object header.

Instance has empty payload.

type Info 

type Info struct {
	Shards []shard.Info
}

Info groups the information about StorageEngine.

type InhumePrm 

type InhumePrm struct {
	// contains filtered or unexported fields
}

InhumePrm encapsulates parameters for inhume operation.

func (*InhumePrm) MarkAsGarbage added in v0.16.0 

func (p *InhumePrm) MarkAsGarbage(addrs ...*addressSDK.Address) *InhumePrm

MarkAsGarbage marks object to be physically removed from local storage.

Should not be called along with WithTarget.

func (*InhumePrm) WithTarget 

func (p *InhumePrm) WithTarget(tombstone *addressSDK.Address, addrs ...*addressSDK.Address) *InhumePrm

WithTarget sets list of objects that should be inhumed and tombstone address as the reason for inhume operation.

tombstone should not be nil, addr should not be empty. Should not be called along with MarkAsGarbage.

type InhumeRes 

type InhumeRes struct{}

InhumeRes encapsulates results of inhume operation.

type ListContainersPrm added in v0.15.0 

type ListContainersPrm struct{}

ListContainersPrm groups parameters of ListContainers operation.

type ListContainersRes added in v0.15.0 

type ListContainersRes struct {
	// contains filtered or unexported fields
}

ListContainersRes groups resulting values of ListContainers operation.

func (ListContainersRes) Containers added in v0.15.0 

func (r ListContainersRes) Containers() []*cid.ID

Containers returns list of identifiers of the containers in which local objects are stored.

type ListWithCursorPrm added in v0.27.0 

type ListWithCursorPrm struct {
	// contains filtered or unexported fields
}

ListWithCursorPrm contains parameters for ListWithCursor operation.

func (*ListWithCursorPrm) WithCount added in v0.27.0 

func (p *ListWithCursorPrm) WithCount(count uint32) *ListWithCursorPrm

WithCount sets maximum amount of addresses that ListWithCursor should return.

func (*ListWithCursorPrm) WithCursor added in v0.27.0 

func (p *ListWithCursorPrm) WithCursor(cursor *Cursor) *ListWithCursorPrm

WithCursor sets cursor for ListWithCursor operation. For initial request ignore this param or use nil value. For consecutive requests, use value from ListWithCursorRes.

type ListWithCursorRes added in v0.27.0 

type ListWithCursorRes struct {
	// contains filtered or unexported fields
}

ListWithCursorRes contains values returned from ListWithCursor operation.

func (ListWithCursorRes) AddressList added in v0.27.0 

func (l ListWithCursorRes) AddressList() []*addressSDK.Address

AddressList returns addresses selected by ListWithCursor operation.

func (ListWithCursorRes) Cursor added in v0.27.0 

func (l ListWithCursorRes) Cursor() *Cursor

Cursor returns cursor for consecutive listing requests.

type MetricRegister added in v0.17.0 

type MetricRegister interface {
	AddListContainersDuration(d time.Duration)
	AddEstimateContainerSizeDuration(d time.Duration)
	AddDeleteDuration(d time.Duration)
	AddExistsDuration(d time.Duration)
	AddGetDuration(d time.Duration)
	AddHeadDuration(d time.Duration)
	AddInhumeDuration(d time.Duration)
	AddPutDuration(d time.Duration)
	AddRangeDuration(d time.Duration)
	AddSearchDuration(d time.Duration)
	AddListObjectsDuration(d time.Duration)
}

type Option 

type Option func(*cfg)

Option represents StorageEngine's constructor option.

func WithErrorThreshold added in v0.28.0 

func WithErrorThreshold(sz uint32) Option

WithErrorThreshold returns an option to specify size amount of errors after which shard is moved to read-only mode.

func WithLogger 

func WithLogger(l *logger.Logger) Option

WithLogger returns option to set StorageEngine's logger.

func WithMetrics added in v0.17.0 

func WithMetrics(v MetricRegister) Option

func WithShardPoolSize added in v0.26.0 

func WithShardPoolSize(sz uint32) Option

WithShardPoolSize returns option to specify size of worker pool for each shard.

type PutPrm 

type PutPrm struct {
	// contains filtered or unexported fields
}

PutPrm groups the parameters of Put operation.

func (*PutPrm) WithObject 

func (p *PutPrm) WithObject(obj *objectSDK.Object) *PutPrm

WithObject is a Put option to set object to save.

Option is required.

type PutRes 

type PutRes struct{}

PutRes groups resulting values of Put operation.

type RngPrm 

type RngPrm struct {
	// contains filtered or unexported fields
}

RngPrm groups the parameters of GetRange operation.

func (*RngPrm) WithAddress 

func (p *RngPrm) WithAddress(addr *addressSDK.Address) *RngPrm

WithAddress is a GetRng option to set the address of the requested object.

Option is required.

func (*RngPrm) WithPayloadRange 

func (p *RngPrm) WithPayloadRange(rng *objectSDK.Range) *RngPrm

WithPayloadRange is a GetRange option to set range of requested payload data.

Missing an option or calling with zero length is equivalent to getting the full payload range.

type RngRes 

type RngRes struct {
	// contains filtered or unexported fields
}

RngRes groups resulting values of GetRange operation.

func (*RngRes) Object 

func (r *RngRes) Object() *objectSDK.Object

Object returns the requested object part.

Instance payload contains the requested range of the original object.

type SelectPrm 

type SelectPrm struct {
	// contains filtered or unexported fields
}

SelectPrm groups the parameters of Select operation.

func (*SelectPrm) WithContainerID 

func (p *SelectPrm) WithContainerID(cid *cid.ID) *SelectPrm

WithContainerID is a Select option to set the container id to search in.

func (*SelectPrm) WithFilters 

func (p *SelectPrm) WithFilters(fs object.SearchFilters) *SelectPrm

WithFilters is a Select option to set the object filters.

type SelectRes 

type SelectRes struct {
	// contains filtered or unexported fields
}

SelectRes groups resulting values of Select operation.

func (*SelectRes) AddressList 

func (r *SelectRes) AddressList() []*addressSDK.Address

AddressList returns list of addresses of the selected objects.

type StorageEngine 

type StorageEngine struct {
	// contains filtered or unexported fields
}

StorageEngine represents NeoFS local storage engine.

func New 

func New(opts ...Option) *StorageEngine

New creates, initializes and returns new StorageEngine instance.

func (*StorageEngine) AddShard 

func (e *StorageEngine) AddShard(opts ...shard.Option) (*shard.ID, error)

AddShard adds a new shard to the storage engine.

Returns any error encountered that did not allow adding a shard. Otherwise returns the ID of the added shard.

func (*StorageEngine) BlockExecution added in v0.27.0 

func (e *StorageEngine) BlockExecution(err error) error

BlockExecution blocks the execution of any data-related operation. All blocked ops will return err. To resume the execution, use ResumeExecution method.

Сan be called regardless of the fact of the previous blocking. If execution wasn't blocked, releases all resources similar to Close. Can be called concurrently with Close and any data related method (waits for all executions to complete). Returns error if any Close has been called before.

Must not be called concurrently with either Open or Init.

Note: technically passing nil error will resume the execution, otherwise, it is recommended to call ResumeExecution for this.

func (*StorageEngine) Close 

func (e *StorageEngine) Close() error

Close releases all StorageEngine's components. Waits for all data-related operations to complete. After the call, all the next ones will fail.

The method is supposed to be called when the application exits.

func (*StorageEngine) ContainerSize added in v0.15.0 

func (e *StorageEngine) ContainerSize(prm ContainerSizePrm) (res *ContainerSizeRes, err error)

ContainerSize returns sum of estimation container sizes among all shards.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Delete 

func (e *StorageEngine) Delete(prm *DeletePrm) (res *DeleteRes, err error)

Delete marks the objects to be removed.

Returns an error if executions are blocked (see BlockExecution).

Returns apistatus.ObjectLocked if at least one object is locked. In this case no object from the list is marked to be deleted.

func (*StorageEngine) DumpInfo 

func (e *StorageEngine) DumpInfo() (i Info)

DumpInfo returns information about the StorageEngine.

func (*StorageEngine) DumpShard added in v0.27.5 

func (e *StorageEngine) DumpShard(id *shard.ID, prm *shard.DumpPrm) error

DumpShard dumps objects from the shard with provided identifier.

Returns an error if shard is not read-only.

func (*StorageEngine) Get 

func (e *StorageEngine) Get(prm *GetPrm) (res *GetRes, err error)

Get reads an object from local storage.

Returns any error encountered that did not allow to completely read the object part.

Returns an error of type apistatus.ObjectNotFound if requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if object has been marked as removed.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) GetRange 

func (e *StorageEngine) GetRange(prm *RngPrm) (res *RngRes, err error)

GetRange reads part of an object from local storage.

Returns any error encountered that did not allow to completely read the object part.

Returns an error of type apistatus.ObjectNotFound if requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if requested object is inhumed. Returns ErrRangeOutOfBounds if requested object range is out of bounds.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Head 

func (e *StorageEngine) Head(prm *HeadPrm) (res *HeadRes, err error)

Head reads object header from local storage.

Returns any error encountered that did not allow to completely read the object header.

Returns an error of type apistatus.ObjectNotFound if requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if requested object was inhumed.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Inhume 

func (e *StorageEngine) Inhume(prm *InhumePrm) (res *InhumeRes, err error)

Inhume calls metabase. Inhume method to mark object as removed. It won't be removed physically from shard until `Delete` operation.

Allows inhuming non-locked objects only. Returns apistatus.ObjectLocked if at least one object is locked.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Init 

func (e *StorageEngine) Init() error

Init initializes all StorageEngine's components.

func (*StorageEngine) List 

func (e *StorageEngine) List(limit uint64) (res *SelectRes, err error)

List returns `limit` available physically storage object addresses in engine. If limit is zero, then returns all available object addresses.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) ListContainers added in v0.15.0 

func (e *StorageEngine) ListContainers(_ ListContainersPrm) (res *ListContainersRes, err error)

ListContainers returns unique container IDs presented in the engine objects.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) ListWithCursor added in v0.27.0 

func (e *StorageEngine) ListWithCursor(prm *ListWithCursorPrm) (*ListWithCursorRes, error)

ListWithCursor lists physical objects available in engine starting from cursor. Includes regular,tombstone and storage group objects. Does not include inhumed objects. Use cursor value from response for consecutive requests.

Returns ErrEndOfListing if there are no more objects to return or count parameter set to zero.

func (*StorageEngine) Lock added in v0.28.0 

func (e *StorageEngine) Lock(idCnr cid.ID, locker oid.ID, locked []oid.ID) error

Lock marks objects as locked with another object. All objects from the specified container.

Allows locking regular objects only (otherwise returns apistatus.LockNonRegularObject).

Locked list should be unique. Panics if it is empty.

func (*StorageEngine) Open 

func (e *StorageEngine) Open() error

Open opens all StorageEngine's components.

func (*StorageEngine) Put 

func (e *StorageEngine) Put(prm *PutPrm) (res *PutRes, err error)

Put saves the object to local storage.

Returns any error encountered that did not allow to completely save the object.

Returns an error if executions are blocked (see BlockExecution).

Returns an error of type apistatus.ObjectAlreadyRemoved if object has been marked as removed.

func (*StorageEngine) RestoreShard added in v0.27.5 

func (e *StorageEngine) RestoreShard(id *shard.ID, prm *shard.RestorePrm) error

RestoreShard restores objects from dump to the shard with provided identifier.

Returns an error if shard is not read-only.

func (*StorageEngine) ResumeExecution added in v0.27.0 

func (e *StorageEngine) ResumeExecution() error

ResumeExecution resumes the execution of any data-related operation. To block the execution, use BlockExecution method.

Сan be called regardless of the fact of the previous blocking. If execution was blocked, prepares all resources similar to Open. Can be called concurrently with Close and any data related method (waits for all executions to complete). Returns error if any Close has been called before.

Must not be called concurrently with either Open or Init.

func (*StorageEngine) Select 

func (e *StorageEngine) Select(prm *SelectPrm) (res *SelectRes, err error)

Select selects the objects from local storage that match select parameters.

Returns any error encountered that did not allow to completely select the objects.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) SetShardMode 

func (e *StorageEngine) SetShardMode(id *shard.ID, m shard.Mode, resetErrorCounter bool) error

SetShardMode sets mode of the shard with provided identifier.

Returns an error if shard mode was not set, or shard was not found in storage engine.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.