engine

package
v0.44.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 28, 2024 License: GPL-3.0 Imports: 30 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrEndOfListing = shard.ErrEndOfListing

ErrEndOfListing is returned from an object listing with cursor when the storage can't return any more objects after the provided cursor. Use nil cursor object to start listing again.

Functions

This section is empty.

Types

type Cursor added in v0.27.0 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor is a type for continuous object listing. It's returned from StorageEngine.ListWithCursor and can be reused as a parameter for it for subsequent requests.

type Info 

type Info struct {
	Shards []shard.Info
}

Info groups the information about StorageEngine.

type MetricRegister added in v0.17.0 

type MetricRegister interface {
	AddListContainersDuration(d time.Duration)
	AddEstimateContainerSizeDuration(d time.Duration)
	AddDeleteDuration(d time.Duration)
	AddExistsDuration(d time.Duration)
	AddGetDuration(d time.Duration)
	AddHeadDuration(d time.Duration)
	AddInhumeDuration(d time.Duration)
	AddPutDuration(d time.Duration)
	AddRangeDuration(d time.Duration)
	AddSearchDuration(d time.Duration)
	AddListObjectsDuration(d time.Duration)

	SetObjectCounter(shardID, objectType string, v uint64)
	AddToObjectCounter(shardID, objectType string, delta int)

	SetReadonly(shardID string, readonly bool)

	AddToContainerSize(cnrID string, size int64)
	AddToPayloadCounter(shardID string, size int64)
}

type ObjectShardStatus added in v0.39.0 

type ObjectShardStatus struct {
	ID    string
	Shard shard.ObjectStatus
}

ObjectShardStatus contains the status of the object in the Shard and Shard ID.

type ObjectStatus added in v0.39.0 

type ObjectStatus struct {
	Shards []ObjectShardStatus
}

ObjectStatus represents the status of the object in the StorageEngine.

type Option 

type Option func(*cfg)

Option represents StorageEngine's constructor option.

func WithContainersSource added in v0.40.0 

func WithContainersSource(cs container.Source) Option

WithContainersSource returns an option to specify container source.

func WithErrorThreshold added in v0.28.0 

func WithErrorThreshold(sz uint32) Option

WithErrorThreshold returns an option to specify size amount of errors after which shard is moved to read-only mode.

func WithIgnoreUninitedShards added in v0.44.0 

func WithIgnoreUninitedShards(flag bool) Option

WithIgnoreUninitedShards return an option to specify whether uninited shards should be ignored.

func WithLogger 

func WithLogger(l *zap.Logger) Option

WithLogger returns option to set StorageEngine's logger.

func WithMetrics added in v0.17.0 

func WithMetrics(v MetricRegister) Option

func WithObjectPutRetryTimeout added in v0.44.0 

func WithObjectPutRetryTimeout(t time.Duration) Option

WithObjectPutRetryTimeout return an option to specify time for object PUT operation. It does not stop any disk operation, only affects retryes policy. Zero value is acceptable and means no retry on any shard.

func WithShardPoolSize added in v0.26.0 

func WithShardPoolSize(sz uint32) Option

WithShardPoolSize returns option to specify size of worker pool for each shard.

type ReConfiguration added in v0.33.0 

type ReConfiguration struct {
	// contains filtered or unexported fields
}

func (*ReConfiguration) AddShard added in v0.33.0 

func (rCfg *ReConfiguration) AddShard(id string, opts []shard.Option)

AddShard adds a shard for the reconfiguration. Shard identifier is calculated from paths used in blobstor.

func (*ReConfiguration) SetErrorsThreshold added in v0.33.0 

func (rCfg *ReConfiguration) SetErrorsThreshold(errorsThreshold uint32)

SetErrorsThreshold sets a size amount of errors after which shard is moved to read-only mode.

func (*ReConfiguration) SetShardPoolSize added in v0.33.0 

func (rCfg *ReConfiguration) SetShardPoolSize(shardPoolSize uint32)

SetShardPoolSize sets a size of worker pool for each shard.

type ReviveShardStatus added in v0.44.0 

type ReviveShardStatus struct {
	ID     string
	Status meta.ReviveStatus
}

ReviveShardStatus contains the Status of the object's revival in the Shard and Shard ID.

type ReviveStatus added in v0.44.0 

type ReviveStatus struct {
	Shards []ReviveShardStatus
}

ReviveStatus represents the status of the object's revival in the StorageEngine.

type StorageEngine 

type StorageEngine struct {
	// contains filtered or unexported fields
}

StorageEngine represents NeoFS local storage engine. If WithContainersSource is used, StorageEngine deletes all stored containers outside [ContainersSource.GetContainers] set.

func New 

func New(opts ...Option) *StorageEngine

New creates, initializes and returns new StorageEngine instance.

func (*StorageEngine) AddShard 

func (e *StorageEngine) AddShard(opts ...shard.Option) (*shard.ID, error)

AddShard adds a new shard to the storage engine.

Returns any error encountered that did not allow adding a shard. Otherwise returns the ID of the added shard.

func (*StorageEngine) BlockExecution added in v0.27.0 

func (e *StorageEngine) BlockExecution(err error) error

BlockExecution blocks the execution of any data-related operation. All blocked ops will return err. To resume the execution, use ResumeExecution method.

Сan be called regardless of the fact of the previous blocking. If execution wasn't blocked, releases all resources similar to Close. Can be called concurrently with Close and any data related method (waits for all executions to complete). Returns error if any Close has been called before.

Must not be called concurrently with either Open or Init.

Note: technically passing nil error will resume the execution, otherwise, it is recommended to call ResumeExecution for this.

func (*StorageEngine) Close 

func (e *StorageEngine) Close() error

Close releases all StorageEngine's components. Waits for all data-related operations to complete. After the call, all the next ones will fail.

The method MUST only be called when the application exits.

func (*StorageEngine) ContainerSize added in v0.15.0 

func (e *StorageEngine) ContainerSize(cnr cid.ID) (uint64, error)

ContainerSize returns the sum of estimation container sizes among all shards.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Delete 

func (e *StorageEngine) Delete(addr oid.Address) error

Delete marks the objects to be removed.

Returns an error if executions are blocked (see BlockExecution).

NOTE: This is a forced removal, marks any object to be deleted (despite any prohibitions on operations with that object).

func (*StorageEngine) DeleteContainer added in v0.40.0 

func (e *StorageEngine) DeleteContainer(ctx context.Context, cID cid.ID) error

DeleteContainer deletes container's objects that engine stores.

func (*StorageEngine) DumpInfo 

func (e *StorageEngine) DumpInfo() (i Info)

DumpInfo returns information about the StorageEngine.

func (*StorageEngine) DumpShard added in v0.27.5 

func (e *StorageEngine) DumpShard(id *shard.ID, w io.Writer, ignoreErrors bool) error

DumpShard dumps objects from the shard with provided identifier.

Returns an error if shard is not read-only.

func (*StorageEngine) Evacuate added in v0.33.0 

func (e *StorageEngine) Evacuate(shardIDs []*shard.ID, ignoreErrors bool, faultHandler func(oid.Address, *objectSDK.Object) error) (int, error)

Evacuate moves data from a set of given shards to other shards available to this engine (so at least one shard must be available unless faultHandler is given). Shards being moved must be in read-only mode. Will return an error if unable to get an object unless ignoreErrors is set to true. If unable to put an object into any of the provided shards invokes faultHandler (if provided, fails otherwise) which can return its own error to abort evacuation (or nil to continue). Returns the number of evacuated objects (which can be non-zero even in case of error).

func (*StorageEngine) FlushWriteCache added in v0.33.0 

func (e *StorageEngine) FlushWriteCache(id *shard.ID) error

FlushWriteCache flushes write-cache on a single shard with the given ID.

func (*StorageEngine) Get 

func (e *StorageEngine) Get(addr oid.Address) (*objectSDK.Object, error)

Get reads an object from local storage.

Returns any error encountered that did not allow to completely read the object part.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the object has been marked as removed.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) GetBytes added in v0.41.0 

func (e *StorageEngine) GetBytes(addr oid.Address) ([]byte, error)

GetBytes reads object from the StorageEngine by address into memory buffer in a canonical NeoFS binary format. Returns apistatus.ObjectNotFound if object is missing.

func (*StorageEngine) GetRange 

func (e *StorageEngine) GetRange(addr oid.Address, offset uint64, length uint64) ([]byte, error)

GetRange reads a part of an object from local storage. Zero length is interpreted as requiring full object length independent of the offset.

Returns any error encountered that did not allow to completely read the object part.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the requested object is inhumed. Returns ErrRangeOutOfBounds if the requested object range is out of bounds.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) HandleNewEpoch added in v0.33.0 

func (e *StorageEngine) HandleNewEpoch(epoch uint64)

HandleNewEpoch notifies every shard about NewEpoch event.

func (*StorageEngine) Head 

func (e *StorageEngine) Head(addr oid.Address, raw bool) (*objectSDK.Object, error)

Head reads object header from local storage. If raw is true returns SplitInfo of the virtual object instead of the virtual object header.

Returns any error encountered that did not allow to completely read the object header.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the requested object was inhumed.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Inhume 

func (e *StorageEngine) Inhume(tombstone oid.Address, tombExpiration uint64, addrs ...oid.Address) error

Inhume calls [metabase.Inhume] method to mark an object as removed following tombstone data. It won't be removed physically from the shard until GC cycle does it.

Allows inhuming non-locked objects only. Returns apistatus.ObjectLocked if at least one object is locked.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) InhumeContainer added in v0.40.0 

func (e *StorageEngine) InhumeContainer(cID cid.ID) error

InhumeContainer marks every object in a container as removed. Any further StorageEngine.Get calls will return apistatus.ObjectNotFound errors. There is no any LOCKs, forced GC marks and any relations checks, every object that belongs to a provided container will be marked as a removed one.

func (*StorageEngine) Init 

func (e *StorageEngine) Init() error

Init initializes all StorageEngine's components.

func (*StorageEngine) IsLocked added in v0.38.0 

func (e *StorageEngine) IsLocked(addr oid.Address) (bool, error)

IsLocked checks whether an object is locked according to StorageEngine's state.

func (*StorageEngine) List 

func (e *StorageEngine) List(limit uint64) ([]oid.Address, error)

List returns `limit` available physically storage object addresses in engine. If limit is zero, then returns all available object addresses.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) ListContainers added in v0.15.0 

func (e *StorageEngine) ListContainers() ([]cid.ID, error)

ListContainers returns a unique container IDs presented in the engine objects.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) ListWithCursor added in v0.27.0 

func (e *StorageEngine) ListWithCursor(count uint32, cursor *Cursor) ([]objectcore.AddressWithType, *Cursor, error)

ListWithCursor lists physical objects available in the engine starting from the cursor. It includes regular, tombstone and storage group objects. Does not include inhumed objects. Use cursor value from the response for consecutive requests (it's nil when iteration is over).

Returns ErrEndOfListing if there are no more objects to return or count parameter set to zero.

func (*StorageEngine) Lock added in v0.28.0 

func (e *StorageEngine) Lock(idCnr cid.ID, locker oid.ID, locked []oid.ID) error

Lock marks objects as locked with another object. All objects from the specified container.

Allows locking regular objects only (otherwise returns apistatus.LockNonRegularObject).

Locked list should be unique. Panics if it is empty.

func (*StorageEngine) ObjectStatus added in v0.39.0 

func (e *StorageEngine) ObjectStatus(address oid.Address) (ObjectStatus, error)

ObjectStatus returns the status of the object in the StorageEngine. It contains status of the object in all shards.

func (*StorageEngine) Open 

func (e *StorageEngine) Open() error

Open opens all StorageEngine's components.

func (*StorageEngine) Put 

func (e *StorageEngine) Put(obj *objectSDK.Object, objBin []byte, hdrLen int) error

Put saves an object to local storage. objBin and hdrLen parameters are optional and used to optimize out object marshaling, when used both must be valid.

Returns any error encountered that did not allow to completely save the object.

Returns an error if executions are blocked (see BlockExecution).

Returns an error of type apistatus.ObjectAlreadyRemoved if the object has been marked as removed.

func (*StorageEngine) Reload added in v0.33.0 

func (e *StorageEngine) Reload(rcfg ReConfiguration) error

Reload reloads StorageEngine's configuration in runtime.

func (*StorageEngine) RestoreShard added in v0.27.5 

func (e *StorageEngine) RestoreShard(id *shard.ID, r io.Reader, ignoreErrors bool) error

RestoreShard restores objects from dump to the shard with provided identifier.

Returns an error if shard is not read-only.

func (*StorageEngine) ResumeExecution added in v0.27.0 

func (e *StorageEngine) ResumeExecution() error

ResumeExecution resumes the execution of any data-related operation. To block the execution, use BlockExecution method.

Сan be called regardless of the fact of the previous blocking. If execution was blocked, prepares all resources similar to Open. Can be called concurrently with Close and any data related method (waits for all executions to complete). Returns error if any Close has been called before.

Must not be called concurrently with either Open or Init.

func (*StorageEngine) ReviveObject added in v0.44.0 

func (e *StorageEngine) ReviveObject(address oid.Address) (ReviveStatus, error)

ReviveObject forcefully revives object by oid.Address in the StorageEngine. Iterate over all shards despite errors and purge all removal marks from all metabases.

func (*StorageEngine) Select 

func (e *StorageEngine) Select(cnr cid.ID, filters object.SearchFilters) ([]oid.Address, error)

Select selects the objects from local storage that match select parameters.

Returns any error encountered that did not allow to completely select the objects.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) SetShardMode 

func (e *StorageEngine) SetShardMode(id *shard.ID, m mode.Mode, resetErrorCounter bool) error

SetShardMode sets mode of the shard with provided identifier.

Returns an error if shard mode was not set, or shard was not found in storage engine.

func (*StorageEngine) TreeAddByPath added in v0.30.0 

func (e *StorageEngine) TreeAddByPath(d pilorama.CIDDescriptor, treeID string, attr string, path []string, m []pilorama.KeyValue) ([]pilorama.LogMove, error)

TreeAddByPath implements the pilorama.Forest interface.

func (*StorageEngine) TreeApply added in v0.30.0 

func (e *StorageEngine) TreeApply(d pilorama.CIDDescriptor, treeID string, m *pilorama.Move, backgroundSync bool) error

TreeApply implements the pilorama.Forest interface.

func (*StorageEngine) TreeDrop added in v0.32.0 

func (e *StorageEngine) TreeDrop(cid cidSDK.ID, treeID string) error

TreeDrop implements the pilorama.Forest interface.

func (*StorageEngine) TreeExists added in v0.36.0 

func (e *StorageEngine) TreeExists(cid cidSDK.ID, treeID string) (bool, error)

TreeExists implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetByPath added in v0.30.0 

func (e *StorageEngine) TreeGetByPath(cid cidSDK.ID, treeID string, attr string, path []string, latest bool) ([]pilorama.Node, error)

TreeGetByPath implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetChildren added in v0.30.0 

func (e *StorageEngine) TreeGetChildren(cid cidSDK.ID, treeID string, nodeID pilorama.Node) ([]uint64, error)

TreeGetChildren implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetMeta added in v0.30.0 

func (e *StorageEngine) TreeGetMeta(cid cidSDK.ID, treeID string, nodeID pilorama.Node) (pilorama.Meta, uint64, error)

TreeGetMeta implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetOpLog added in v0.30.0 

func (e *StorageEngine) TreeGetOpLog(cid cidSDK.ID, treeID string, height uint64) (pilorama.Move, error)

TreeGetOpLog implements the pilorama.Forest interface.

func (*StorageEngine) TreeList added in v0.34.0 

func (e *StorageEngine) TreeList(cid cidSDK.ID) ([]string, error)

TreeList implements the pilorama.Forest interface.

func (*StorageEngine) TreeMove added in v0.30.0 

func (e *StorageEngine) TreeMove(d pilorama.CIDDescriptor, treeID string, m *pilorama.Move) (*pilorama.LogMove, error)

TreeMove implements the pilorama.Forest interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.