blobstor

Documentation

Index

• Variables
• func IterateBinaryObjects(blz *BlobStor, f func(addr oid.Address, data []byte, descriptor []byte) error) error
• type BlobStor
  • func New(opts ...Option) *BlobStor
  • func (b *BlobStor) Close() error
  • func (b *BlobStor) Delete(prm common.DeletePrm) (common.DeleteRes, error)
  • func (b *BlobStor) DumpInfo() Info
  • func (b *BlobStor) Exists(prm common.ExistsPrm) (common.ExistsRes, error)
  • func (b *BlobStor) Get(prm common.GetPrm) (common.GetRes, error)
  • func (b *BlobStor) GetBytes(addr oid.Address, subStorageID []byte) ([]byte, error)
  • func (b *BlobStor) GetRange(prm common.GetRangePrm) (common.GetRangeRes, error)
  • func (b *BlobStor) Init() error
  • func (b *BlobStor) Iterate(prm common.IteratePrm) (common.IterateRes, error)
  • func (b *BlobStor) NeedsCompression(obj *objectSDK.Object) bool
  • func (b *BlobStor) ObjectStatus(address oid.Address) (ObjectStatus, error)
  • func (b *BlobStor) Open(readOnly bool) error
  • func (b *BlobStor) Put(prm common.PutPrm) (common.PutRes, error)
  • func (b *BlobStor) SetLogger(l *zap.Logger)
  • func (b *BlobStor) SetMode(m mode.Mode) error
  • func (b *BlobStor) SetReportErrorFunc(f func(string, error))
• type Info
• type ObjectStatus
• type ObjectSubstorageStatus
• type Option
  • func WithCompressObjects(comp bool) Option
  • func WithLogger(l *zap.Logger) Option
  • func WithStorages(st []SubStorage) Option
  • func WithUncompressableContentTypes(values []string) Option
• type SubStorage
• type SubStorageInfo

Variables

var ErrNoPlaceFound = logicerr.New("couldn't find a place to store an object")

ErrNoPlaceFound is returned when object can't be saved to any sub-storage component because of the policy.

Functions

func IterateBinaryObjects

func IterateBinaryObjects(blz *BlobStor, f func(addr oid.Address, data []byte, descriptor []byte) error) error

IterateBinaryObjects is a helper function which iterates over BlobStor and passes binary objects to f. Errors related to object reading and unmarshaling are logged and skipped.

Types

type BlobStor

type BlobStor struct {
	// contains filtered or unexported fields
}

BlobStor represents NeoFS local BLOB storage.

func New

func New(opts ...Option) *BlobStor

New creates, initializes and returns new BlobStor instance.

func (*BlobStor) Close

func (b *BlobStor) Close() error

Close releases all internal resources of BlobStor.

func (*BlobStor) Delete

func (b *BlobStor) Delete(prm common.DeletePrm) (common.DeleteRes, error)

func (*BlobStor) DumpInfo

func (b *BlobStor) DumpInfo() Info

DumpInfo returns information about blob stor.

func (*BlobStor) Exists

func (b *BlobStor) Exists(prm common.ExistsPrm) (common.ExistsRes, error)

Exists checks if the object is presented in BLOB storage.

Returns any error encountered that did not allow to completely check object existence.

func (*BlobStor) Get

func (b *BlobStor) Get(prm common.GetPrm) (common.GetRes, error)

Get reads the object from b. If the descriptor is present, only one sub-storage is tried, Otherwise, each sub-storage is tried in order.

func (*BlobStor) GetBytes

func (b *BlobStor) GetBytes(addr oid.Address, subStorageID []byte) ([]byte, error)

GetBytes reads object from the BlobStor by address into memory buffer in a canonical NeoFS binary format. Returns apistatus.ObjectNotFound if object is missing.

func (*BlobStor) GetRange

func (b *BlobStor) GetRange(prm common.GetRangePrm) (common.GetRangeRes, error)

GetRange reads object payload data from b. If the descriptor is present, only one sub-storage is tried, Otherwise, each sub-storage is tried in order.

func (*BlobStor) Init

func (b *BlobStor) Init() error

Init initializes internal data structures and system resources.

If BlobStor is already initialized, no action is taken.

func (*BlobStor) Iterate

func (b *BlobStor) Iterate(prm common.IteratePrm) (common.IterateRes, error)

Iterate traverses the storage over the stored objects and calls the handler on each element.

Returns any error encountered that did not allow to completely iterate over the storage.

If handler returns an error, method wraps and returns it immediately.

func (*BlobStor) NeedsCompression

func (b *BlobStor) NeedsCompression(obj *objectSDK.Object) bool

NeedsCompression returns true if the object should be compressed. For an object to be compressed 2 conditions must hold: 1. Compression is enabled in settings. 2. Object MIME Content-Type is allowed for compression.

func (*BlobStor) ObjectStatus

func (b *BlobStor) ObjectStatus(address oid.Address) (ObjectStatus, error)

ObjectStatus returns the status of the object in the Blob storage. It contains status of the object in all blob substorages.

func (*BlobStor) Open

func (b *BlobStor) Open(readOnly bool) error

Open opens BlobStor.

func (*BlobStor) Put

func (b *BlobStor) Put(prm common.PutPrm) (common.PutRes, error)

Put saves the object in BLOB storage.

If object is "big", BlobStor saves the object in shallow dir. Otherwise, BlobStor saves the object in peapod.

Returns any error encountered that did not allow to completely save the object.

func (*BlobStor) SetLogger

func (b *BlobStor) SetLogger(l *zap.Logger)

SetLogger sets logger. It is used after the shard ID was generated to use it in logs.

func (*BlobStor) SetMode

func (b *BlobStor) SetMode(m mode.Mode) error

SetMode sets the blobstor mode of operation.

func (*BlobStor) SetReportErrorFunc

func (b *BlobStor) SetReportErrorFunc(f func(string, error))

SetReportErrorFunc allows to provide a function to be called on disk errors. This function MUST be called before Open.

type Info

type Info struct {
	SubStorages []SubStorageInfo
}

Info contains information about blobstor.

type ObjectStatus

type ObjectStatus struct {
	Substorages []ObjectSubstorageStatus
}

ObjectStatus represents the status of the object in the Blob storage.

type ObjectSubstorageStatus

type ObjectSubstorageStatus struct {
	Type  string
	Path  string
	Error error
}

ObjectSubstorageStatus represents the status of the object in the Blob storage, containing the type and path of the storage and an error if it occurred.

type Option

type Option func(*cfg)

Option represents BlobStor's constructor option.

func WithCompressObjects

func WithCompressObjects(comp bool) Option

WithCompressObjects returns option to toggle compression of the stored objects.

If true, Zstandard algorithm is used for data compression.

If compressor (decompressor) creation failed, the uncompressed option will be used, and the error is recorded in the provided log.

func WithLogger

func WithLogger(l *zap.Logger) Option

WithLogger returns option to specify BlobStor's logger.

func WithStorages

func WithStorages(st []SubStorage) Option

WithStorages provides sub-blobstors.

func WithUncompressableContentTypes

func WithUncompressableContentTypes(values []string) Option

WithUncompressableContentTypes returns option to disable decompression for specific content types as seen by object.AttributeContentType attribute.

type SubStorage

type SubStorage struct {
	Storage common.Storage
	Policy  func(*objectSDK.Object, []byte) bool
}

SubStorage represents single storage component with some storage policy.

type SubStorageInfo

type SubStorageInfo struct {
	Type string
	Path string
}

SubStorageInfo contains information about blobstor storage component.