engine

package
v0.44.0-rc.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 31, 2024 License: GPL-3.0 Imports: 42 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrEndOfListing = shard.ErrEndOfListing

ErrEndOfListing is returned from an object listing with cursor when the storage can't return any more objects after the provided cursor. Use nil cursor object to start listing again.

View Source 
var (
	ErrMustBeReadOnly = logicerr.New("shard must be in read-only mode")
)

Functions

func ContainerSize 

func ContainerSize(e *StorageEngine, id cid.ID) (uint64, error)

ContainerSize calls ContainerSize method on engine to calculate sum of estimation container sizes among all shards.

func Get 

func Get(ctx context.Context, storage *StorageEngine, addr oid.Address) (*objectSDK.Object, error)

Get reads object from local storage by provided address.

func GetRange 

func GetRange(ctx context.Context, storage *StorageEngine, addr oid.Address, rng *objectSDK.Range) ([]byte, error)

GetRange reads object payload range from local storage by provided address. 

func Head(ctx context.Context, storage *StorageEngine, addr oid.Address) (*objectSDK.Object, error)

Head reads object header from local storage by provided address.

func List 

func List(ctx context.Context, storage *StorageEngine, limit uint64) ([]oid.Address, error)

List returns `limit` available physically storage object addresses in engine. If limit is zero, then returns all available object addresses.

func ListContainers 

func ListContainers(ctx context.Context, e *StorageEngine) ([]cid.ID, error)

ListContainers calls ListContainers method on engine to get a unique container IDs presented in the engine objects.

func Put 

func Put(ctx context.Context, storage *StorageEngine, obj *objectSDK.Object, indexedContainer bool) error

Put writes provided object to local storage.

func Select 

func Select(ctx context.Context, storage *StorageEngine, cnr cid.ID, isIndexedContainer bool, fs objectSDK.SearchFilters) ([]oid.Address, error)

Select selects objects from local storage using provided filters.

Types

type ContainerSizePrm 

type ContainerSizePrm struct {
	// contains filtered or unexported fields
}

ContainerSizePrm groups parameters of ContainerSize operation.

func (*ContainerSizePrm) SetContainerID 

func (p *ContainerSizePrm) SetContainerID(cnr cid.ID)

SetContainerID sets the identifier of the container to estimate the size.

type ContainerSizeRes 

type ContainerSizeRes struct {
	// contains filtered or unexported fields
}

ContainerSizeRes resulting values of ContainerSize operation.

func (ContainerSizeRes) Size 

func (r ContainerSizeRes) Size() uint64

Size returns calculated estimation of the container size.

type Cursor 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor is a type for continuous object listing. Cursor contains shard IDs to read and shard cursors that contain state from previous read.

type DeletePrm 

type DeletePrm struct {
	// contains filtered or unexported fields
}

DeletePrm groups the parameters of Delete operation.

func (*DeletePrm) WithAddress 

func (p *DeletePrm) WithAddress(addr oid.Address)

WithAddress is a Delete option to set the addresses of the objects to delete.

Option is required.

func (*DeletePrm) WithForceRemoval 

func (p *DeletePrm) WithForceRemoval()

WithForceRemoval is a Delete option to remove an object despite any restrictions imposed on deleting that object. Expected to be used only in control service.

type DeleteRes 

type DeleteRes struct{}

DeleteRes groups the resulting values of Delete operation.

type EvacuateProcessState added in v0.37.0 

type EvacuateProcessState int
const (
	EvacuateProcessStateUndefined EvacuateProcessState = iota
	EvacuateProcessStateRunning
	EvacuateProcessStateCompleted
)

type EvacuateScope added in v0.38.0 

type EvacuateScope uint32

EvacuateScope is an evacuation scope. Keep in sync with pkg/services/control/service.proto. 

var (
	EvacuateScopeObjects EvacuateScope = 1
	EvacuateScopeTrees   EvacuateScope = 2
)

func (EvacuateScope) String added in v0.38.0 

func (s EvacuateScope) String() string

func (EvacuateScope) TreesOnly added in v0.38.0 

func (s EvacuateScope) TreesOnly() bool

func (EvacuateScope) WithObjects added in v0.38.0 

func (s EvacuateScope) WithObjects() bool

func (EvacuateScope) WithTrees added in v0.38.0 

func (s EvacuateScope) WithTrees() bool

type EvacuateShardPrm 

type EvacuateShardPrm struct {
	ShardID        []*shard.ID
	ObjectsHandler func(context.Context, oid.Address, *objectSDK.Object) (bool, error)
	TreeHandler    func(context.Context, cid.ID, string, pilorama.Forest) (bool, string, error)
	IgnoreErrors   bool
	Async          bool
	Scope          EvacuateScope
	RepOneOnly     bool

	ContainerWorkerCount uint32
	ObjectWorkerCount    uint32
}

EvacuateShardPrm represents parameters for the EvacuateShard operation.

type EvacuateShardRes 

type EvacuateShardRes struct {
	// contains filtered or unexported fields
}

EvacuateShardRes represents result of the EvacuateShard operation.

func NewEvacuateShardRes added in v0.37.0 

func NewEvacuateShardRes() *EvacuateShardRes

NewEvacuateShardRes creates new EvacuateShardRes instance.

func (*EvacuateShardRes) DeepCopy added in v0.37.0 

func (p *EvacuateShardRes) DeepCopy() *EvacuateShardRes

DeepCopy returns deep copy of result instance.

func (*EvacuateShardRes) ObjectsEvacuated added in v0.38.0 

func (p *EvacuateShardRes) ObjectsEvacuated() uint64

ObjectsEvacuated returns amount of evacuated objects. Objects for which handler returned no error are also assumed evacuated.

func (*EvacuateShardRes) ObjectsFailed added in v0.38.0 

func (p *EvacuateShardRes) ObjectsFailed() uint64

ObjectsFailed returns count of failed objects to evacuate.

func (*EvacuateShardRes) ObjectsSkipped added in v0.38.0 

func (p *EvacuateShardRes) ObjectsSkipped() uint64

ObjectsSkipped returns count of skipped objects.

func (*EvacuateShardRes) ObjectsTotal added in v0.38.0 

func (p *EvacuateShardRes) ObjectsTotal() uint64

ObjectsTotal returns total count objects to evacuate.

func (*EvacuateShardRes) TreesEvacuated added in v0.38.0 

func (p *EvacuateShardRes) TreesEvacuated() uint64

TreesEvacuated returns amount of evacuated trees.

func (*EvacuateShardRes) TreesFailed added in v0.38.0 

func (p *EvacuateShardRes) TreesFailed() uint64

TreesFailed returns count of failed trees to evacuate.

func (*EvacuateShardRes) TreesTotal added in v0.38.0 

func (p *EvacuateShardRes) TreesTotal() uint64

TreesTotal returns total count trees to evacuate.

type EvacuationState added in v0.37.0 

type EvacuationState struct {
	// contains filtered or unexported fields
}

func (*EvacuationState) DeepCopy added in v0.37.0 

func (s *EvacuationState) DeepCopy() *EvacuationState

func (*EvacuationState) ErrorMessage added in v0.37.0 

func (s *EvacuationState) ErrorMessage() string

func (*EvacuationState) FinishedAt added in v0.37.0 

func (s *EvacuationState) FinishedAt() *time.Time

func (*EvacuationState) ObjectsEvacuated added in v0.38.0 

func (s *EvacuationState) ObjectsEvacuated() uint64

func (*EvacuationState) ObjectsFailed added in v0.38.0 

func (s *EvacuationState) ObjectsFailed() uint64

func (*EvacuationState) ObjectsSkipped added in v0.38.0 

func (s *EvacuationState) ObjectsSkipped() uint64

func (*EvacuationState) ObjectsTotal added in v0.38.0 

func (s *EvacuationState) ObjectsTotal() uint64

func (*EvacuationState) ProcessingStatus added in v0.37.0 

func (s *EvacuationState) ProcessingStatus() EvacuateProcessState

func (*EvacuationState) ShardIDs added in v0.37.0 

func (s *EvacuationState) ShardIDs() []string

func (*EvacuationState) StartedAt added in v0.37.0 

func (s *EvacuationState) StartedAt() *time.Time

func (*EvacuationState) TreesEvacuated added in v0.38.0 

func (s *EvacuationState) TreesEvacuated() uint64

func (*EvacuationState) TreesFailed added in v0.38.0 

func (s *EvacuationState) TreesFailed() uint64

func (*EvacuationState) TreesTotal added in v0.38.0 

func (s *EvacuationState) TreesTotal() uint64

type FlushWriteCachePrm 

type FlushWriteCachePrm struct {
	// contains filtered or unexported fields
}

FlushWriteCachePrm groups the parameters of FlushWriteCache operation.

func (*FlushWriteCachePrm) SetIgnoreErrors 

func (p *FlushWriteCachePrm) SetIgnoreErrors(ignore bool)

SetIgnoreErrors sets errors ignore flag.

func (*FlushWriteCachePrm) SetSeal added in v0.38.0 

func (p *FlushWriteCachePrm) SetSeal(v bool)

SetSeal sets seal flag.

func (*FlushWriteCachePrm) SetShardID 

func (p *FlushWriteCachePrm) SetShardID(id *shard.ID)

SetShardID is an option to set shard ID.

Option is required.

type FlushWriteCacheRes 

type FlushWriteCacheRes struct{}

FlushWriteCacheRes groups the resulting values of FlushWriteCache operation.

type GetPrm 

type GetPrm struct {
	// contains filtered or unexported fields
}

GetPrm groups the parameters of Get operation.

func (*GetPrm) WithAddress 

func (p *GetPrm) WithAddress(addr oid.Address)

WithAddress is a Get option to set the address of the requested object.

Option is required.

type GetRes 

type GetRes struct {
	// contains filtered or unexported fields
}

GetRes groups the resulting values of Get operation.

func (GetRes) Object 

func (r GetRes) Object() *objectSDK.Object

Object returns the requested object.

type HeadPrm 

type HeadPrm struct {
	// contains filtered or unexported fields
}

HeadPrm groups the parameters of Head operation.

func (*HeadPrm) WithAddress 

func (p *HeadPrm) WithAddress(addr oid.Address)

WithAddress is a Head option to set the address of the requested object.

Option is required.

func (*HeadPrm) WithRaw 

func (p *HeadPrm) WithRaw(raw bool)

WithRaw is a Head option to set raw flag value. If flag is unset, then Head returns the header of the virtual object, otherwise it returns SplitInfo of the virtual object.

type HeadRes 

type HeadRes struct {
	// contains filtered or unexported fields
}

HeadRes groups the resulting values of Head operation.

func (HeadRes) Header 

func (r HeadRes) Header() *objectSDK.Object

Header returns the requested object header.

Instance has empty payload.

type Info 

type Info struct {
	Shards []shard.Info
}

Info groups the information about StorageEngine.

type InhumePrm 

type InhumePrm struct {
	// contains filtered or unexported fields
}

InhumePrm encapsulates parameters for inhume operation.

func (*InhumePrm) MarkAsGarbage 

func (p *InhumePrm) MarkAsGarbage(addrs ...oid.Address)

MarkAsGarbage marks an object to be physically removed from local storage.

Should not be called along with WithTarget.

func (*InhumePrm) WithForceRemoval 

func (p *InhumePrm) WithForceRemoval()

WithForceRemoval inhumes objects specified via MarkAsGarbage with GC mark without any object restrictions checks.

func (*InhumePrm) WithTarget 

func (p *InhumePrm) WithTarget(tombstone oid.Address, addrs ...oid.Address)

WithTarget sets a list of objects that should be inhumed and tombstone address as the reason for inhume operation.

tombstone should not be nil, addr should not be empty. Should not be called along with MarkAsGarbage.

type InhumeRes 

type InhumeRes struct{}

InhumeRes encapsulates results of inhume operation.

type ListContainersPrm 

type ListContainersPrm struct{}

ListContainersPrm groups parameters of ListContainers operation.

type ListContainersRes 

type ListContainersRes struct {
	// contains filtered or unexported fields
}

ListContainersRes groups the resulting values of ListContainers operation.

func (ListContainersRes) Containers 

func (r ListContainersRes) Containers() []cid.ID

Containers returns a list of identifiers of the containers in which local objects are stored.

type ListWithCursorPrm 

type ListWithCursorPrm struct {
	// contains filtered or unexported fields
}

ListWithCursorPrm contains parameters for ListWithCursor operation.

func (*ListWithCursorPrm) WithCount 

func (p *ListWithCursorPrm) WithCount(count uint32)

WithCount sets the maximum amount of addresses that ListWithCursor should return.

func (*ListWithCursorPrm) WithCursor 

func (p *ListWithCursorPrm) WithCursor(cursor *Cursor)

WithCursor sets a cursor for ListWithCursor operation. For initial request ignore this param or use nil value. For consecutive requests, use value from ListWithCursorRes.

type ListWithCursorRes 

type ListWithCursorRes struct {
	// contains filtered or unexported fields
}

ListWithCursorRes contains values returned from ListWithCursor operation.

func (ListWithCursorRes) AddressList 

func (l ListWithCursorRes) AddressList() []objectcore.Info

AddressList returns addresses selected by ListWithCursor operation.

func (ListWithCursorRes) Cursor 

func (l ListWithCursorRes) Cursor() *Cursor

Cursor returns cursor for consecutive listing requests.

type MetricRegister 

type MetricRegister interface {
	AddMethodDuration(method string, d time.Duration)

	SetObjectCounter(shardID, objectType string, v uint64)
	AddToObjectCounter(shardID, objectType string, delta int)

	SetMode(shardID string, mode mode.Mode)

	AddToContainerSize(cnrID string, size int64)
	DeleteContainerSize(cnrID string)
	DeleteContainerCount(cnrID string)
	AddToPayloadCounter(shardID string, size int64)
	IncErrorCounter(shardID string)
	ClearErrorCounter(shardID string)
	DeleteShardMetrics(shardID string)

	SetContainerObjectCounter(shardID, contID, objectType string, v uint64)
	IncContainerObjectCounter(shardID, contID, objectType string)
	SubContainerObjectCounter(shardID, contID, objectType string, v uint64)

	IncRefillObjectsCount(shardID, path string, size int, success bool)
	SetRefillPercent(shardID, path string, percent uint32)
	SetRefillStatus(shardID, path, status string)
	SetEvacuationInProgress(shardID string, value bool)

	WriteCache() metrics.WriteCacheMetrics
	GC() metrics.GCMetrics
}

type Option 

type Option func(*cfg)

Option represents StorageEngine's constructor option.

func WithErrorThreshold 

func WithErrorThreshold(sz uint32) Option

WithErrorThreshold returns an option to specify size amount of errors after which shard is moved to read-only mode.

func WithLogger 

func WithLogger(l *logger.Logger) Option

WithLogger returns option to set StorageEngine's logger.

func WithLowMemoryConsumption added in v0.37.0 

func WithLowMemoryConsumption(lowMemCons bool) Option

WithLowMemoryConsumption returns an option to set the flag to reduce memory consumption by reducing performance.

func WithMetrics 

func WithMetrics(v MetricRegister) Option

func WithShardPoolSize 

func WithShardPoolSize(sz uint32) Option

WithShardPoolSize returns option to specify size of worker pool for each shard.

type PutPrm 

type PutPrm struct {
	Object             *objectSDK.Object
	IsIndexedContainer bool
}

PutPrm groups the parameters of Put operation.

type ReConfiguration 

type ReConfiguration struct {
	// contains filtered or unexported fields
}

func (*ReConfiguration) AddShard 

func (rCfg *ReConfiguration) AddShard(id string, opts []shard.Option)

AddShard adds a shard for the reconfiguration. Shard identifier is calculated from paths used in blobstor.

type RebuildPrm added in v0.43.0 

type RebuildPrm struct {
	ShardIDs          []*shard.ID
	ConcurrencyLimit  uint32
	TargetFillPercent uint32
}

type RebuildRes added in v0.43.0 

type RebuildRes struct {
	ShardResults []ShardRebuildResult
}

type RemoveDuplicatesPrm 

type RemoveDuplicatesPrm struct {
	Concurrency int
}

type RngPrm 

type RngPrm struct {
	// contains filtered or unexported fields
}

RngPrm groups the parameters of GetRange operation.

func (*RngPrm) WithAddress 

func (p *RngPrm) WithAddress(addr oid.Address)

WithAddress is a GetRng option to set the address of the requested object.

Option is required.

func (*RngPrm) WithPayloadRange 

func (p *RngPrm) WithPayloadRange(rng *objectSDK.Range)

WithPayloadRange is a GetRange option to set range of requested payload data.

Missing an option or calling with zero length is equivalent to getting the full payload range.

type RngRes 

type RngRes struct {
	// contains filtered or unexported fields
}

RngRes groups the resulting values of GetRange operation.

func (RngRes) Object 

func (r RngRes) Object() *objectSDK.Object

Object returns the requested object part.

Instance payload contains the requested range of the original object.

type SealWriteCachePrm added in v0.38.0 

type SealWriteCachePrm struct {
	ShardIDs     []*shard.ID
	IgnoreErrors bool
	Async        bool
	RestoreMode  bool
	Shrink       bool
}

type SealWriteCacheRes added in v0.38.0 

type SealWriteCacheRes struct {
	ShardResults []ShardSealResult
}

type SelectPrm 

type SelectPrm struct {
	// contains filtered or unexported fields
}

SelectPrm groups the parameters of Select operation.

func (*SelectPrm) WithContainerID 

func (p *SelectPrm) WithContainerID(cnr cid.ID, indexedContainer bool)

WithContainerID is a Select option to set the container id to search in.

func (*SelectPrm) WithFilters 

func (p *SelectPrm) WithFilters(fs objectSDK.SearchFilters)

WithFilters is a Select option to set the object filters.

type SelectRes 

type SelectRes struct {
	// contains filtered or unexported fields
}

SelectRes groups the resulting values of Select operation.

func (SelectRes) AddressList 

func (r SelectRes) AddressList() []oid.Address

AddressList returns list of addresses of the selected objects.

type ShardRebuildResult added in v0.43.0 

type ShardRebuildResult struct {
	ShardID  *shard.ID
	Success  bool
	ErrorMsg string
}

type ShardSealResult added in v0.38.0 

type ShardSealResult struct {
	ShardID  *shard.ID
	Success  bool
	ErrorMsg string
}

type StorageEngine 

type StorageEngine struct {
	// contains filtered or unexported fields
}

StorageEngine represents FrostFS local storage engine.

func New 

func New(opts ...Option) *StorageEngine

New creates, initializes and returns new StorageEngine instance.

func (*StorageEngine) AddShard 

func (e *StorageEngine) AddShard(ctx context.Context, opts ...shard.Option) (*shard.ID, error)

AddShard adds a new shard to the storage engine.

Returns any error encountered that did not allow adding a shard. Otherwise returns the ID of the added shard.

func (*StorageEngine) BlockExecution 

func (e *StorageEngine) BlockExecution(err error) error

BlockExecution blocks the execution of any data-related operation. All blocked ops will return err. To resume the execution, use ResumeExecution method.

Сan be called regardless of the fact of the previous blocking. If execution wasn't blocked, releases all resources similar to Close. Can be called concurrently with Close and any data related method (waits for all executions to complete). Returns error if any Close has been called before.

Must not be called concurrently with either Open or Init.

Note: technically passing nil error will resume the execution, otherwise, it is recommended to call ResumeExecution for this.

func (*StorageEngine) Close 

func (e *StorageEngine) Close(ctx context.Context) error

Close releases all StorageEngine's components. Waits for all data-related operations to complete. After the call, all the next ones will fail.

The method MUST only be called when the application exits.

func (*StorageEngine) ContainerSize 

func (e *StorageEngine) ContainerSize(prm ContainerSizePrm) (res ContainerSizeRes, err error)

ContainerSize returns the sum of estimation container sizes among all shards.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Delete 

func (e *StorageEngine) Delete(ctx context.Context, prm DeletePrm) (res DeleteRes, err error)

Delete marks the objects to be removed.

Returns an error if executions are blocked (see BlockExecution).

Returns apistatus.ObjectLocked if at least one object is locked. In this case no object from the list is marked to be deleted.

NOTE: Marks any object to be deleted (despite any prohibitions on operations with that object) if WithForceRemoval option has been provided.

func (*StorageEngine) DetachShards added in v0.38.0 

func (e *StorageEngine) DetachShards(ids []*shard.ID) error

func (*StorageEngine) DumpInfo 

func (e *StorageEngine) DumpInfo() (i Info)

DumpInfo returns information about the StorageEngine.

func (*StorageEngine) EnqueRunningEvacuationStop added in v0.37.0 

func (e *StorageEngine) EnqueRunningEvacuationStop(ctx context.Context) error

func (*StorageEngine) Evacuate 

func (e *StorageEngine) Evacuate(ctx context.Context, prm EvacuateShardPrm) (*EvacuateShardRes, error)

Evacuate moves data from one shard to the others. The shard being moved must be in read-only mode.

func (*StorageEngine) FlushWriteCache 

func (e *StorageEngine) FlushWriteCache(ctx context.Context, p FlushWriteCachePrm) (FlushWriteCacheRes, error)

FlushWriteCache flushes write-cache on a single shard.

func (*StorageEngine) Get 

func (e *StorageEngine) Get(ctx context.Context, prm GetPrm) (res GetRes, err error)

Get reads an object from local storage.

Returns any error encountered that did not allow to completely read the object part.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the object has been marked as removed.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) GetEvacuationState added in v0.37.0 

func (e *StorageEngine) GetEvacuationState(ctx context.Context) (*EvacuationState, error)

func (*StorageEngine) GetLocked added in v0.40.0 

func (e *StorageEngine) GetLocked(ctx context.Context, addr oid.Address) ([]oid.ID, error)

GetLocked return lock id's if object is locked according to StorageEngine's state.

func (*StorageEngine) GetRange 

func (e *StorageEngine) GetRange(ctx context.Context, prm RngPrm) (res RngRes, err error)

GetRange reads part of an object from local storage.

Returns any error encountered that did not allow to completely read the object part.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the requested object is inhumed. Returns ErrRangeOutOfBounds if the requested object range is out of bounds.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) HandleNewEpoch 

func (e *StorageEngine) HandleNewEpoch(ctx context.Context, epoch uint64)

HandleNewEpoch notifies every shard about NewEpoch event.

func (*StorageEngine) Head 

func (e *StorageEngine) Head(ctx context.Context, prm HeadPrm) (res HeadRes, err error)

Head reads object header from local storage.

Returns any error encountered that did not allow to completely read the object header.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the requested object was inhumed.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Inhume 

func (e *StorageEngine) Inhume(ctx context.Context, prm InhumePrm) (res InhumeRes, err error)

Inhume calls metabase. Inhume method to mark an object as removed. It won't be removed physically from the shard until `Delete` operation.

Allows inhuming non-locked objects only. Returns apistatus.ObjectLocked if at least one object is locked.

NOTE: Marks any object as removed (despite any prohibitions on operations with that object) if WithForceRemoval option has been provided.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Init 

func (e *StorageEngine) Init(ctx context.Context) error

Init initializes all StorageEngine's components.

func (*StorageEngine) IsLocked 

func (e *StorageEngine) IsLocked(ctx context.Context, addr oid.Address) (bool, error)

IsLocked checks whether an object is locked according to StorageEngine's state.

func (*StorageEngine) List 

func (e *StorageEngine) List(ctx context.Context, limit uint64) (res SelectRes, err error)

List returns `limit` available physically storage object addresses in engine. If limit is zero, then returns all available object addresses.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) ListContainers 

func (e *StorageEngine) ListContainers(ctx context.Context, _ ListContainersPrm) (res ListContainersRes, err error)

ListContainers returns a unique container IDs presented in the engine objects.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) ListWithCursor 

func (e *StorageEngine) ListWithCursor(ctx context.Context, prm ListWithCursorPrm) (ListWithCursorRes, error)

ListWithCursor lists physical objects available in the engine starting from the cursor. It includes regular, tombstone and storage group objects. Does not include inhumed objects. Use cursor value from the response for consecutive requests.

If count param is big enough, then the method reads objects from different shards by portions. In this case shards are chosen randomly, if they're not read out yet.

Adding a shard between ListWithCursor does not invalidate the cursor but new shard won't be listed. Removing a shard between ListWithCursor leads to the undefined behavior (e.g. usage of the objects from the removed shard).

Returns ErrEndOfListing if there are no more objects to return or count parameter set to zero.

func (*StorageEngine) Lock 

func (e *StorageEngine) Lock(ctx context.Context, idCnr cid.ID, locker oid.ID, locked []oid.ID) error

Lock marks objects as locked with another object. All objects from the specified container.

Allows locking regular objects only (otherwise returns apistatus.LockNonRegularObject).

Locked list should be unique. Panics if it is empty.

func (*StorageEngine) Open 

func (e *StorageEngine) Open(ctx context.Context) error

Open opens all StorageEngine's components.

func (*StorageEngine) Put 

func (e *StorageEngine) Put(ctx context.Context, prm PutPrm) (err error)

Put saves the object to local storage.

Returns any error encountered that did not allow to completely save the object.

Returns an error if executions are blocked (see BlockExecution).

Returns an error of type apistatus.ObjectAlreadyRemoved if the object has been marked as removed.

func (*StorageEngine) Rebuild added in v0.43.0 

func (e *StorageEngine) Rebuild(ctx context.Context, prm RebuildPrm) (RebuildRes, error)

func (*StorageEngine) Reload 

func (e *StorageEngine) Reload(ctx context.Context, rcfg ReConfiguration) error

Reload reloads StorageEngine's configuration in runtime.

func (*StorageEngine) RemoveDuplicates 

func (e *StorageEngine) RemoveDuplicates(ctx context.Context, prm RemoveDuplicatesPrm) error

RemoveDuplicates iterates over all objects and removes duplicate object copies from shards which are worse as defined by HRW sort. Safety:

  1. Concurrent execution is prohibited, thus 1 object copy should always be left.
  2. If we delete an object from another thread, this is not a problem. Currently, we have 2 thread that can remove "valid" (non-expired and logically non-removed) objects: policer and rebalance. For rebalance see (1). If policer removes something, we do not care if both copies are removed or one of them is left, as the remaining copy will be removed during the next policer iteration.

func (*StorageEngine) ResetEvacuationStatus added in v0.38.0 

func (e *StorageEngine) ResetEvacuationStatus(ctx context.Context) error

func (*StorageEngine) ResetEvacuationStatusForShards added in v0.43.0 

func (e *StorageEngine) ResetEvacuationStatusForShards()

func (*StorageEngine) ResumeExecution 

func (e *StorageEngine) ResumeExecution() error

ResumeExecution resumes the execution of any data-related operation. To block the execution, use BlockExecution method.

Сan be called regardless of the fact of the previous blocking. If execution was blocked, prepares all resources similar to Open. Can be called concurrently with Close and any data related method (waits for all executions to complete). Returns error if any Close has been called before.

Must not be called concurrently with either Open or Init.

func (*StorageEngine) SealWriteCache added in v0.38.0 

func (e *StorageEngine) SealWriteCache(ctx context.Context, prm SealWriteCachePrm) (SealWriteCacheRes, error)

SealWriteCache flushed all data to blobstore and moves write-cache to degraded read only mode.

func (*StorageEngine) Select 

func (e *StorageEngine) Select(ctx context.Context, prm SelectPrm) (res SelectRes, err error)

Select selects the objects from local storage that match select parameters.

Returns any error encountered that did not allow to completely select the objects.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) SetContainerSource added in v0.38.0 

func (e *StorageEngine) SetContainerSource(cs container.Source)

SetContainerSource sets container source.

func (*StorageEngine) SetShardMode 

func (e *StorageEngine) SetShardMode(id *shard.ID, m mode.Mode, resetErrorCounter bool) error

SetShardMode sets mode of the shard with provided identifier.

Returns an error if shard mode was not set, or shard was not found in storage engine.

func (*StorageEngine) TreeAddByPath 

func (e *StorageEngine) TreeAddByPath(ctx context.Context, d pilorama.CIDDescriptor, treeID string, attr string, path []string, m []pilorama.KeyValue) ([]pilorama.Move, error)

TreeAddByPath implements the pilorama.Forest interface.

func (*StorageEngine) TreeApply 

func (e *StorageEngine) TreeApply(ctx context.Context, cnr cidSDK.ID, treeID string, m *pilorama.Move, backgroundSync bool) error

TreeApply implements the pilorama.Forest interface.

func (*StorageEngine) TreeDrop 

func (e *StorageEngine) TreeDrop(ctx context.Context, cid cidSDK.ID, treeID string) error

TreeDrop implements the pilorama.Forest interface.

func (*StorageEngine) TreeExists 

func (e *StorageEngine) TreeExists(ctx context.Context, cid cidSDK.ID, treeID string) (bool, error)

TreeExists implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetByPath 

func (e *StorageEngine) TreeGetByPath(ctx context.Context, cid cidSDK.ID, treeID string, attr string, path []string, latest bool) ([]pilorama.Node, error)

TreeGetByPath implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetChildren 

func (e *StorageEngine) TreeGetChildren(ctx context.Context, cid cidSDK.ID, treeID string, nodeID pilorama.Node) ([]pilorama.NodeInfo, error)

TreeGetChildren implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetMeta 

func (e *StorageEngine) TreeGetMeta(ctx context.Context, cid cidSDK.ID, treeID string, nodeID pilorama.Node) (pilorama.Meta, uint64, error)

TreeGetMeta implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetOpLog 

func (e *StorageEngine) TreeGetOpLog(ctx context.Context, cid cidSDK.ID, treeID string, height uint64) (pilorama.Move, error)

TreeGetOpLog implements the pilorama.Forest interface.

func (*StorageEngine) TreeHeight 

func (e *StorageEngine) TreeHeight(ctx context.Context, cid cidSDK.ID, treeID string) (uint64, error)

func (*StorageEngine) TreeLastSyncHeight 

func (e *StorageEngine) TreeLastSyncHeight(ctx context.Context, cid cidSDK.ID, treeID string) (uint64, error)

TreeLastSyncHeight implements the pilorama.Forest interface.

func (*StorageEngine) TreeList 

func (e *StorageEngine) TreeList(ctx context.Context, cid cidSDK.ID) ([]string, error)

TreeList implements the pilorama.Forest interface.

func (*StorageEngine) TreeMove 

func (e *StorageEngine) TreeMove(ctx context.Context, d pilorama.CIDDescriptor, treeID string, m *pilorama.Move) (*pilorama.Move, error)

TreeMove implements the pilorama.Forest interface.

func (*StorageEngine) TreeSortedByFilename added in v0.38.2 

func (e *StorageEngine) TreeSortedByFilename(ctx context.Context, cid cidSDK.ID, treeID string, nodeID pilorama.MultiNode, last *string, count int) ([]pilorama.MultiNodeInfo, *string, error)

TreeSortedByFilename implements the pilorama.Forest interface.

func (*StorageEngine) TreeUpdateLastSyncHeight 

func (e *StorageEngine) TreeUpdateLastSyncHeight(ctx context.Context, cid cidSDK.ID, treeID string, height uint64) error

TreeUpdateLastSyncHeight implements the pilorama.Forest interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.