blobstor

package
v0.44.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 6, 2024 License: GPL-3.0 Imports: 26 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrInitBlobovniczas = errors.New("failure on blobovnicza initialization stage")

ErrInitBlobovniczas is returned when blobovnicza initialization fails.

View Source 
var ErrNoPlaceFound = logicerr.New("couldn't find a place to store an object")

ErrNoPlaceFound is returned when object can't be saved to any sub-storage component because of the policy.

Functions

func IterateBinaryObjects 

func IterateBinaryObjects(ctx context.Context, blz *BlobStor, f func(addr oid.Address, data []byte, descriptor []byte) error) error

IterateBinaryObjects is a helper function which iterates over BlobStor and passes binary objects to f. Errors related to object reading and unmarshaling are logged and skipped.

Types

type BlobStor 

type BlobStor struct {
	// contains filtered or unexported fields
}

BlobStor represents FrostFS local BLOB storage.

func New 

func New(opts ...Option) *BlobStor

New creates, initializes and returns new BlobStor instance.

func (*BlobStor) Close 

func (b *BlobStor) Close(ctx context.Context) error

Close releases all internal resources of BlobStor.

func (*BlobStor) Compressor added in v0.37.0 

func (b *BlobStor) Compressor() *compression.Config

func (*BlobStor) Delete 

func (b *BlobStor) Delete(ctx context.Context, prm common.DeletePrm) (common.DeleteRes, error)

func (*BlobStor) DumpInfo 

func (b *BlobStor) DumpInfo() Info

DumpInfo returns information about blob stor.

func (*BlobStor) Exists 

func (b *BlobStor) Exists(ctx context.Context, prm common.ExistsPrm) (common.ExistsRes, error)

Exists checks if the object is presented in BLOB storage.

Returns any error encountered that did not allow to completely check object existence.

func (*BlobStor) Get 

func (b *BlobStor) Get(ctx context.Context, prm common.GetPrm) (res common.GetRes, err error)

Get reads the object from b. If the descriptor is present, only one sub-storage is tried, Otherwise, each sub-storage is tried in order.

func (*BlobStor) GetRange 

func (b *BlobStor) GetRange(ctx context.Context, prm common.GetRangePrm) (res common.GetRangeRes, err error)

GetRange reads object payload data from b. If the descriptor is present, only one sub-storage is tried, Otherwise, each sub-storage is tried in order.

func (*BlobStor) Init 

func (b *BlobStor) Init(ctx context.Context) error

Init initializes internal data structures and system resources.

If BlobStor is already initialized, no action is taken.

Returns wrapped ErrInitBlobovniczas on blobovnicza tree's initializaiton failure.

func (*BlobStor) Iterate 

func (b *BlobStor) Iterate(ctx context.Context, prm common.IteratePrm) (common.IterateRes, error)

Iterate traverses the storage over the stored objects and calls the handler on each element.

Returns any error encountered that did not allow to completely iterate over the storage.

If handler returns an error, method wraps and returns it immediately.

func (*BlobStor) ObjectsCount added in v0.39.0 

func (b *BlobStor) ObjectsCount(ctx context.Context) (uint64, error)

ObjectsCount returns Blobstore's total objects count.

func (*BlobStor) Open 

func (b *BlobStor) Open(ctx context.Context, mode mode.Mode) error

Open opens BlobStor.

func (*BlobStor) Put 

func (b *BlobStor) Put(ctx context.Context, prm common.PutPrm) (common.PutRes, error)

Put saves the object in BLOB storage.

If object is "big", BlobStor saves the object in shallow dir. Otherwise, BlobStor saves the object in blobonicza. In this case the identifier of blobovnicza is returned.

Returns any error encountered that did not allow to completely save the object.

func (*BlobStor) Rebuild added in v0.38.0 

func (b *BlobStor) Rebuild(ctx context.Context, upd StorageIDUpdate, limiter ConcurrentWorkersLimiter, fillPercent int) error

func (*BlobStor) SetLogger 

func (b *BlobStor) SetLogger(l *logger.Logger)

SetLogger sets logger. It is used after the shard ID was generated to use it in logs.

func (*BlobStor) SetMode 

func (b *BlobStor) SetMode(ctx context.Context, m mode.Mode) error

SetMode sets the blobstor mode of operation.

func (*BlobStor) SetParentID added in v0.37.0 

func (b *BlobStor) SetParentID(parentID string)

func (*BlobStor) SetReportErrorFunc 

func (b *BlobStor) SetReportErrorFunc(f func(context.Context, string, error))

SetReportErrorFunc allows to provide a function to be called on disk errors. This function MUST be called before Open.

type ConcurrentWorkersLimiter added in v0.38.0 

type ConcurrentWorkersLimiter interface {
	AcquireWorkSlot(ctx context.Context) error
	ReleaseWorkSlot()
}

type Info 

type Info struct {
	SubStorages []SubStorageInfo
}

Info contains information about blobstor.

type Metrics added in v0.37.0 

type Metrics interface {
	SetParentID(parentID string)
	SetMode(readOnly bool)
	Close()

	Delete(d time.Duration, success, withStorageID bool)
	Exists(d time.Duration, success, withStorageID bool)
	GetRange(d time.Duration, size int, success, withStorageID bool)
	Get(d time.Duration, size int, success, withStorageID bool)
	Iterate(d time.Duration, success bool)
	Put(d time.Duration, size int, success bool)
	ObjectsCount(d time.Duration, success bool)
}

type Option 

type Option func(*cfg)

Option represents BlobStor's constructor option.

func WithCompressObjects 

func WithCompressObjects(comp bool) Option

WithCompressObjects returns option to toggle compression of the stored objects.

If true, Zstandard algorithm is used for data compression.

If compressor (decompressor) creation failed, the uncompressed option will be used, and the error is recorded in the provided log.

func WithCompressibilityEstimate added in v0.38.0 

func WithCompressibilityEstimate(v bool) Option

WithCompressibilityEstimate returns an option to use normilized compressibility estimate to decide compress data or not.

See https://github.com/klauspost/compress/blob/v1.17.2/compressible.go#L5

func WithCompressibilityEstimateThreshold added in v0.38.0 

func WithCompressibilityEstimateThreshold(threshold float64) Option

WithCompressibilityEstimateThreshold returns an option to set normilized compressibility estimate threshold.

See https://github.com/klauspost/compress/blob/v1.17.2/compressible.go#L5

func WithLogger 

func WithLogger(l *logger.Logger) Option

WithLogger returns option to specify BlobStor's logger.

func WithMetrics added in v0.37.0 

func WithMetrics(m Metrics) Option

func WithStorages 

func WithStorages(st []SubStorage) Option

WithStorages provides sub-blobstors.

func WithUncompressableContentTypes 

func WithUncompressableContentTypes(values []string) Option

WithUncompressableContentTypes returns option to disable decompression for specific content types as seen by object.AttributeContentType attribute.

type StorageIDUpdate added in v0.38.0 

type StorageIDUpdate interface {
	UpdateStorageID(ctx context.Context, addr oid.Address, storageID []byte) error
}

type SubStorage 

type SubStorage struct {
	Storage common.Storage
	Policy  func(*objectSDK.Object, []byte) bool
}

SubStorage represents single storage component with some storage policy.

type SubStorageInfo 

type SubStorageInfo struct {
	Type string
	Path string
}

SubStorageInfo contains information about blobstor storage component.

Source Files

View all Source files

Directories

Path Synopsis
internal
blobstortest
Package memstore implements a memory-backed common.Storage for testing purposes.
 Package memstore implements a memory-backed common.Storage for testing purposes.
Package teststore provides a common.Storage implementation for testing/mocking purposes.
 Package teststore provides a common.Storage implementation for testing/mocking purposes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.