meta

package
v0.44.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 28, 2024 License: GPL-3.0 Imports: 30 Imported by: 1

Documentation

Index

Constants

View Source 
const (
	// ReviveStatusGraveyard is the type of revival status of an object from a graveyard.
	ReviveStatusGraveyard reviveStatusType = iota
	// ReviveStatusGarbage is the type of revival status of an object from the garbage bucket.
	ReviveStatusGarbage
	// ReviveStatusError is the type of status when an error occurs during revive.
	ReviveStatusError
)

Variables

View Source 
var (
	ErrUnknownObjectType        = errors.New("unknown object type")
	ErrIncorrectSplitInfoUpdate = errors.New("updating split info on object without it")
	ErrIncorrectRootObject      = errors.New("invalid root object")
)
View Source 
var ErrDegradedMode = logicerr.New("metabase is in a degraded mode")

ErrDegradedMode is returned when metabase is in a degraded mode.

View Source 
var ErrEndOfListing = logicerr.New("end of object listing")

ErrEndOfListing is returned from object listing with cursor when storage can't return any more objects after provided cursor. Use nil cursor object to start listing again.

View Source 
var ErrInterruptIterator = logicerr.New("iterator is interrupted")

ErrInterruptIterator is returned by iteration handlers as a "break" keyword.

View Source 
var ErrLackSplitInfo = logicerr.New("no split info on parent object")
View Source 
var ErrLockObjectRemoval = logicerr.New("lock object removal")

ErrLockObjectRemoval is returned when inhume operation is being performed on lock object, and it is not a forced object removal.

View Source 
var ErrObjectIsExpired = logicerr.New("object is expired")

ErrObjectIsExpired is returned when the requested object's epoch is less than the current one. Such objects are considered as removed and should not be returned from the Storage Engine.

View Source 
var ErrObjectWasNotRemoved = logicerr.New("object neither in the graveyard nor was marked with GC mark")

ErrObjectWasNotRemoved is returned when object neither in the graveyard nor was marked with GC mark.

View Source 
var ErrOutdatedVersion = logicerr.New("invalid version, resynchronization is required")

ErrOutdatedVersion is returned on initializing an existing metabase that is not compatible with the current code version.

View Source 
var ErrReadOnlyMode = logicerr.New("metabase is in a read-only mode")

ErrReadOnlyMode is returned when metabase is in a read-only mode.

View Source 
var ErrReviveFromContainerGarbage = logicerr.New("revive from container marked with GC mark")

ErrReviveFromContainerGarbage is returned when the object is in the container that marked with GC mark.

Functions

func IsErrRemoved added in v0.28.0 

func IsErrRemoved(err error) bool

IsErrRemoved checks if error returned by Shard Exists/Get/Put method corresponds to removed object.

Types

type BucketValue added in v0.43.0 

type BucketValue struct {
	BucketIndex int
	Value       []byte
}

BucketValue pairs a bucket index and a value that relates an object.

type Cursor added in v0.27.0 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor is a type for continuous object listing.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB represents local metabase of storage node.

func New added in v0.13.0 

func New(opts ...Option) *DB

New creates and returns new Metabase instance.

func (*DB) Close 

func (db *DB) Close() error

Close closes boltDB instance.

func (*DB) ContainerSize added in v0.15.0 

func (db *DB) ContainerSize(id cid.ID) (size uint64, err error)

func (*DB) Containers added in v0.13.0 

func (db *DB) Containers() (list []cid.ID, err error)

func (*DB) Delete 

func (db *DB) Delete(addrs []oid.Address) (DeleteRes, error)

Delete removes object records from metabase indexes. Does not stop on an error if there are more objects to handle requested; returns the first error appeared with a number of deleted objects wrapped.

func (*DB) DeleteContainer added in v0.40.0 

func (db *DB) DeleteContainer(cID cid.ID) error

DeleteContainer removes any information that the metabase has associated with the provided container (its objects) except the graveyard-related one.

func (*DB) DoNotMove added in v0.13.0 

func (db *DB) DoNotMove(addr oid.Address) error

DoNotMove removes `MoveIt` mark from the object.

func (*DB) DropExpiredTSMarks added in v0.44.0 

func (db *DB) DropExpiredTSMarks(epoch uint64) (int, error)

DropExpiredTSMarks run through the graveyard and drops tombstone marks with tombstones whose expiration is _less_ than provided epoch. Returns number of marks dropped.

func (*DB) DumpInfo added in v0.13.0 

func (db *DB) DumpInfo() Info

DumpInfo returns information about the DB.

func (*DB) Exists added in v0.13.0 

func (db *DB) Exists(addr oid.Address, ignoreExpiration bool) (bool, error)

Exists returns ErrAlreadyRemoved if addr was marked as removed. Otherwise it returns true if addr is in primary index or false if it is not.

Returns an error of type apistatus.ObjectAlreadyRemoved if object has been placed in graveyard. Returns the object.ErrObjectIsExpired if the object is presented but already expired.

func (*DB) FilterExpired added in v0.38.0 

func (db *DB) FilterExpired(addresses []oid.Address) ([]oid.Address, error)

FilterExpired filters expired object from `addresses` and return them. Uses internal epoch state provided via the WithEpochState option.

func (*DB) FreeLockedBy added in v0.28.0 

func (db *DB) FreeLockedBy(lockers []oid.Address) ([]oid.Address, error)

FreeLockedBy unlocks all objects in DB which are locked by lockers. Returns unlocked objects if any.

func (*DB) Get 

func (db *DB) Get(addr oid.Address, raw bool) (*objectSDK.Object, error)

Get returns object header for specified address.

"raw" flag controls virtual object processing, when false (default) a proper object header is returned, when true only SplitInfo of virtual object is returned.

Returns an error of type apistatus.ObjectNotFound if object is missing in DB. Returns an error of type apistatus.ObjectAlreadyRemoved if object has been placed in graveyard. Returns the object.ErrObjectIsExpired if the object is presented but already expired.

func (*DB) GetGarbage added in v0.40.0 

func (db *DB) GetGarbage(limit int) ([]oid.Address, []cid.ID, error)

GetGarbage returns garbage according to the metabase state. Garbage includes objects marked with GC mark (expired, tombstoned but not deleted from disk, extra replicated, etc.) and removed containers. The first return value describes garbage objects. These objects should be removed. The second return value describes garbage containers whose _all_ garbage objects were included in the first return value and, therefore, these containers can be deleted (if their objects are handled and deleted too).

func (*DB) Inhume added in v0.13.0 

func (db *DB) Inhume(tombstone oid.Address, tombExpiration uint64, handleLocks bool, addrs ...oid.Address) (uint64, []oid.Address, error)

Inhume marks objects as removed but not removes it from metabase.

Allows inhuming non-locked objects only. Returns apistatus.ObjectLocked if at least one object is locked. Returns ErrLockObjectRemoval if inhuming is being performed on lock (not locked) object.

Returns the number of available objects that were inhumed and a list of deleted LOCK objects (if handleLocks parameter is set).

func (*DB) InhumeContainer added in v0.40.0 

func (db *DB) InhumeContainer(cID cid.ID) (uint64, error)

InhumeContainer marks every object in a container as removed. Any further DB.Get calls will return apistatus.ObjectNotFound errors. Returns number of available objects marked with GC. There is no any LOCKs, forced GC marks and any relations checks, every object that belongs to a provided container will be marked as a removed one.

func (*DB) Init added in v0.13.0 

func (db *DB) Init() error

Init initializes metabase. It creates static (CID-independent) buckets in underlying BoltDB instance.

Returns ErrOutdatedVersion if a database at the provided path is outdated.

Does nothing if metabase has already been initialized and filled. To roll back the database to its initial state, use Reset.

func (*DB) IsLocked added in v0.36.0 

func (db *DB) IsLocked(addr oid.Address) (bool, error)

IsLocked checks is the provided object is locked by any `LOCK`. Not found object is considered as non-locked.

Returns only non-logical errors related to underlying database.

func (*DB) IterateCoveredByTombstones added in v0.16.0 

func (db *DB) IterateCoveredByTombstones(tss map[string]oid.Address, h func(oid.Address) error) error

IterateCoveredByTombstones iterates over all objects in DB which are covered by tombstone with string address from tss. Locked objects are not included (do not confuse with objects of type LOCK).

If h returns ErrInterruptIterator, nil returns immediately. Returns other errors of h directly.

Does not modify tss.

func (*DB) IterateExpired added in v0.16.0 

func (db *DB) IterateExpired(epoch uint64, h ExpiredObjectHandler) error

IterateExpired iterates over all objects in DB which are out of date relative to epoch. Locked objects are not included (do not confuse with objects of type LOCK).

If h returns ErrInterruptIterator, nil returns immediately. Returns other errors of h directly.

func (*DB) IterateOverGarbage added in v0.28.0 

func (db *DB) IterateOverGarbage(h GarbageHandler, offset *oid.Address) error

IterateOverGarbage iterates over all objects marked with GC mark.

The handler will be applied to the next after the specified offset if any are left.

Note: if offset is not found in db, iteration starts from the element that WOULD BE the following after the offset if offset was presented. That means that it is safe to delete offset element and pass if to the iteration once again: iteration would start from the next element.

Nil offset means start an integration from the beginning.

If h returns ErrInterruptIterator, nil returns immediately. Returns other errors of h directly.

func (*DB) IterateOverGraveyard added in v0.16.0 

func (db *DB) IterateOverGraveyard(h TombstonedHandler, offset *oid.Address) error

IterateOverGraveyard iterates over all graves in DB.

The handler will be applied to the next after the specified offset if any are left.

Note: if offset is not found in db, iteration starts from the element that WOULD BE the following after the offset if offset was presented. That means that it is safe to delete offset element and pass if to the iteration once again: iteration would start from the next element.

Nil offset means start an integration from the beginning.

If h returns ErrInterruptIterator, nil returns immediately. Returns other errors of h directly.

func (*DB) ListContainerObjects added in v0.40.0 

func (db *DB) ListContainerObjects(cID cid.ID, limit int) ([]oid.ID, error)

ListContainerObjects returns objects stored in the metabase that belong to the provided container. No more than limit objects per call. Negative limit values make the result empty. Note: metabase can store information about a locked object, but it will not be included to the result if the object is not stored in the metabase (in other words, no information in the regular objects index).

func (*DB) ListWithCursor added in v0.27.0 

func (db *DB) ListWithCursor(count int, cursor *Cursor) ([]objectcore.AddressWithType, *Cursor, error)

ListWithCursor lists physical objects available in metabase starting from cursor. Includes objects of all types. Does not include inhumed objects. Use cursor value from response for consecutive requests.

Returns ErrEndOfListing if there are no more objects to return or count parameter set to zero.

func (*DB) Lock added in v0.28.0 

func (db *DB) Lock(cnr cid.ID, locker oid.ID, locked []oid.ID) error

Lock marks objects as locked with another object. All objects are from the specified container.

Allows locking regular objects only (otherwise returns apistatus.LockNonRegularObject).

Locked list should be unique. Panics if it is empty.

func (*DB) MarkGarbage added in v0.44.0 

func (db *DB) MarkGarbage(force bool, handleLocks bool, addrs ...oid.Address) (uint64, []oid.Address, error)

MarkGarbage marks objects to be physically removed from shard. force flag allows to override any restrictions imposed on object deletion (to be used by control service and other manual intervention cases). Otherwise similar to DB.Inhume, but doesn't need a tombstone.

func (*DB) Movable added in v0.13.0 

func (db *DB) Movable() ([]oid.Address, error)

Movable returns list of marked objects to move into other shard.

func (*DB) ObjectCounters added in v0.32.0 

func (db *DB) ObjectCounters() (ObjectCounters, error)

ObjectCounters returns object counters that metabase has tracked since it was opened and initialized.

Returns only the errors that do not allow reading counter in Bolt database.

func (*DB) ObjectStatus added in v0.39.0 

func (db *DB) ObjectStatus(address oid.Address) (ObjectStatus, error)

ObjectStatus returns the status of the object in the Metabase. It contains state, path, storageID and indexed information about an object.

func (*DB) Open added in v0.13.0 

func (db *DB) Open(readOnly bool) error

Open boltDB instance for metabase.

func (*DB) Put 

func (db *DB) Put(obj *objectSDK.Object, storageID []byte, binHeader []byte) error

Put saves object header in metabase. Object payload is expected to be cut.

binHeader parameter is optional and allows to provide an already encoded object header in DB format. If provided, the encoding step is skipped. It's the caller's responsibility to ensure that the data matches the object structure being processed.

Returns an error of type apistatus.ObjectAlreadyRemoved if object has been placed in graveyard. Returns the object.ErrObjectIsExpired if the object is presented but already expired.

func (*DB) ReadLastResyncEpoch added in v0.44.0 

func (db *DB) ReadLastResyncEpoch() (epoch uint64, err error)

ReadLastResyncEpoch reads from db last epoch when metabase was resynchronized. If id is missing, returns 0, nil.

func (*DB) ReadShardID added in v0.27.7 

func (db *DB) ReadShardID() ([]byte, error)

ReadShardID reads shard id from db. If id is missing, returns nil, nil.

func (*DB) Reload added in v0.34.0 

func (db *DB) Reload(opts ...Option) (bool, error)

Reload reloads part of the configuration. It returns true iff database was reopened. If a config option is invalid, it logs an error and returns nil. If there was a problem with applying new configuration, an error is returned.

If a metabase was couldn't be reopened because of an error, ErrDegradedMode is returned.

func (*DB) Reset added in v0.25.0 

func (db *DB) Reset() error

Reset resets metabase. Works similar to Init but cleans up all static buckets and removes all dynamic (CID-dependent) ones in non-blank BoltDB instances.

func (*DB) ReviveObject added in v0.44.0 

func (db *DB) ReviveObject(addr oid.Address) (res ReviveStatus, err error)

ReviveObject revives object by oid.Address. Removes GCMark/Tombstone records in the corresponding buckets and restore metrics.

func (*DB) Select 

func (db *DB) Select(cnr cid.ID, filters object.SearchFilters) ([]oid.Address, error)

Select returns list of addresses of objects that match search filters.

Only creation epoch, payload size, user attributes and unknown system ones are allowed with numeric operators. Values of numeric filters must be base-10 integers.

Returns [object.ErrInvalidSearchQuery] if specified query is invalid.

func (*DB) SetLogger added in v0.30.0 

func (db *DB) SetLogger(l *zap.Logger)

SetLogger sets logger. It is used after the shard ID was generated to use it in logs.

func (*DB) SetMode added in v0.30.0 

func (db *DB) SetMode(m mode.Mode) error

SetMode sets the metabase mode of operation. If the mode assumes no operation metabase, the database is closed.

func (*DB) StorageID added in v0.32.0 

func (db *DB) StorageID(addr oid.Address) ([]byte, error)

StorageID returns storage descriptor for objects from the blobstor. It is put together with the object can makes get/delete operation faster.

func (*DB) SyncCounters added in v0.32.0 

func (db *DB) SyncCounters() error

SyncCounters forces to synchronize the object counters.

func (*DB) ToMoveIt added in v0.13.0 

func (db *DB) ToMoveIt(addr oid.Address) error

ToMoveIt marks objects to move it into another shard. This useful for faster HRW fetching.

func (*DB) UpdateStorageID added in v0.36.0 

func (db *DB) UpdateStorageID(addr oid.Address, newID []byte) error

UpdateStorageID updates storage descriptor for objects from the blobstor.

func (*DB) WriteShardID added in v0.27.7 

func (db *DB) WriteShardID(id []byte) error

WriteShardID writes shard it to db.

type DeleteRes added in v0.13.0 

type DeleteRes struct {
	// RawRemoved contains the number of removed raw objects.
	RawRemoved uint64
	// AvailableRemoved contains the number of removed available objects.
	AvailableRemoved uint64
	// Sizes contains the sizes of removed objects.
	// The order of the sizes is the same as in addresses'
	// slice that was provided in the [DB.Delete] address list,
	// meaning that i-th size equals the number of freed up bytes
	// after removing an object by i-th address. A zero size is
	// allowed, it claims a missing object.
	Sizes []uint64
}

DeleteRes groups the resulting values of Delete operation.

type EpochState added in v0.31.0 

type EpochState interface {
	// CurrentEpoch must return current epoch height.
	CurrentEpoch() uint64
}

EpochState is an interface that provides access to the current epoch number.

type ExpiredObject added in v0.16.0 

type ExpiredObject struct {
	// contains filtered or unexported fields
}

ExpiredObject is a descriptor of expired object from DB.

func (*ExpiredObject) Address added in v0.16.0 

func (e *ExpiredObject) Address() oid.Address

Address returns address of the expired object.

func (*ExpiredObject) Type added in v0.16.0 

func (e *ExpiredObject) Type() object.Type

Type returns type of the expired object.

type ExpiredObjectHandler added in v0.16.0 

type ExpiredObjectHandler func(*ExpiredObject) error

ExpiredObjectHandler is an ExpiredObject handling function.

type GarbageHandler added in v0.28.0 

type GarbageHandler func(GarbageObject) error

GarbageHandler is a GarbageObject handling function.

type GarbageObject added in v0.28.0 

type GarbageObject struct {
	// contains filtered or unexported fields
}

GarbageObject represents descriptor of the object that has been marked with GC.

func (GarbageObject) Address added in v0.28.0 

func (g GarbageObject) Address() oid.Address

Address returns garbage object address.

type Info added in v0.13.0 

type Info struct {
	// Full path to the metabase.
	Path string

	// Permission of database file.
	Permission fs.FileMode

	// LastResyncEpoch is a last epoch when metabase was resynchronized.
	LastResyncEpoch uint64
}

Info groups the information about DB.

type ObjectCounters added in v0.32.0 

type ObjectCounters struct {
	// contains filtered or unexported fields
}

ObjectCounters groups object counter according to metabase state.

func (ObjectCounters) Logic added in v0.32.0 

func (o ObjectCounters) Logic() uint64

Logic returns logical object counter.

func (ObjectCounters) Phy added in v0.32.0 

func (o ObjectCounters) Phy() uint64

Phy returns physical object counter.

type ObjectStatus added in v0.39.0 

type ObjectStatus struct {
	Version   uint64
	Buckets   []BucketValue
	State     []string
	Path      string
	StorageID string
	Error     error
}

ObjectStatus represents the status of the object in the Metabase.

type Option 

type Option func(*cfg)

Option is an option of DB constructor.

func WithBoltDBOptions added in v0.13.0 

func WithBoltDBOptions(opts *bbolt.Options) Option

WithBoltDBOptions returns option to specify BoltDB options.

func WithEpochState added in v0.31.0 

func WithEpochState(s EpochState) Option

WithEpochState return option to specify a source of current epoch height.

func WithLogger 

func WithLogger(l *zap.Logger) Option

WithLogger returns option to set logger of DB.

func WithMaxBatchDelay added in v0.29.0 

func WithMaxBatchDelay(d time.Duration) Option

WithMaxBatchDelay returns option to specify maximum time to wait before the batch of concurrent transactions is processed. This option is missing from `bbolt.Options` but is set right after DB is open.

func WithMaxBatchSize added in v0.29.0 

func WithMaxBatchSize(s int) Option

WithMaxBatchSize returns option to specify maximum concurrent operations to be processed in a single transactions. This option is missing from `bbolt.Options` but is set right after DB is open.

func WithPath added in v0.13.0 

func WithPath(path string) Option

WithPath returns option to set system path to Metabase.

func WithPermissions added in v0.13.0 

func WithPermissions(perm fs.FileMode) Option

WithPermissions returns option to specify permission bits of Metabase system path.

type ReviveStatus added in v0.44.0 

type ReviveStatus struct {
	// contains filtered or unexported fields
}

ReviveStatus groups the resulting values of ReviveObject operation. Contains the type of revival status and message for details.

func (*ReviveStatus) Message added in v0.44.0 

func (s *ReviveStatus) Message() string

Message returns message of status.

func (*ReviveStatus) StatusType added in v0.44.0 

func (s *ReviveStatus) StatusType() reviveStatusType

StatusType returns the type of revival status.

type TombstonedHandler added in v0.28.0 

type TombstonedHandler func(object TombstonedObject) error

TombstonedHandler is a TombstonedObject handling function.

type TombstonedObject added in v0.28.0 

type TombstonedObject struct {
	// contains filtered or unexported fields
}

TombstonedObject represents descriptor of the object that has been covered with tombstone.

func (TombstonedObject) Address added in v0.28.0 

func (g TombstonedObject) Address() oid.Address

Address returns tombstoned object address.

func (TombstonedObject) Tombstone added in v0.28.0 

func (g TombstonedObject) Tombstone() oid.Address

Tombstone returns address of a tombstone that covers object.

func (TombstonedObject) TombstoneExpiration added in v0.44.0 

func (g TombstonedObject) TombstoneExpiration() uint64

TombstoneExpiration returns tombstone's expiration. It can be zero if metabase version does not support expiration indexing or if TS does not expire.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.