blobstor

package
v0.30.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 25, 2022 License: GPL-3.0 Imports: 23 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrInitBlobovniczas = errors.New("failure on blobovnicza initialization stage")

ErrInitBlobovniczas is returned when blobovnicza initialization fails.

Functions

func IterateBinaryObjects added in v0.25.0 

func IterateBinaryObjects(blz *BlobStor, f func(addr oid.Address, data []byte, blzID *blobovnicza.ID) error) error

IterateBinaryObjects is a helper function which iterates over BlobStor and passes binary objects to f. Errors related to object reading and unmarshaling are logged and skipped.

Types

type BlobStor 

type BlobStor struct {
	// contains filtered or unexported fields
}

BlobStor represents NeoFS local BLOB storage.

func New 

func New(opts ...Option) *BlobStor

New creates, initializes and returns new BlobStor instance.

func (*BlobStor) Close 

func (b *BlobStor) Close() error

Close releases all internal resources of BlobStor.

func (*BlobStor) DeleteBig 

func (b *BlobStor) DeleteBig(prm DeleteBigPrm) (DeleteBigRes, error)

DeleteBig removes an object from shallow dir of BLOB storage.

Returns any error encountered that did not allow to completely remove the object.

Returns an error of type apistatus.ObjectNotFound if there is no object to delete.

func (*BlobStor) DeleteSmall 

func (b *BlobStor) DeleteSmall(prm DeleteSmallPrm) (DeleteSmallRes, error)

DeleteSmall removes an object from blobovnicza of BLOB storage.

If blobovnicza ID is not set or set to nil, BlobStor tries to find and remove object from any blobovnicza.

Returns any error encountered that did not allow to completely remove the object.

Returns an error of type apistatus.ObjectNotFound if there is no object to delete.

func (*BlobStor) DumpInfo 

func (b *BlobStor) DumpInfo() fstree.Info

DumpInfo returns information about blob stor.

func (*BlobStor) Exists 

func (b *BlobStor) Exists(prm ExistsPrm) (ExistsRes, error)

Exists checks if the object is presented in BLOB storage.

Returns any error encountered that did not allow to completely check object existence.

func (*BlobStor) GetBig 

func (b *BlobStor) GetBig(prm GetBigPrm) (GetBigRes, error)

GetBig reads the object from shallow dir of BLOB storage by address.

Returns any error encountered that did not allow to completely read the object.

Returns an error of type apistatus.ObjectNotFound if the requested object is not presented in shallow dir.

func (*BlobStor) GetRangeBig 

func (b *BlobStor) GetRangeBig(prm GetRangeBigPrm) (GetRangeBigRes, error)

GetRangeBig reads data of object payload range from shallow dir of BLOB storage.

Returns any error encountered that did not allow to completely read the object payload range.

Returns ErrRangeOutOfBounds if the requested object range is out of bounds. Returns an error of type apistatus.ObjectNotFound if object is missing.

func (*BlobStor) GetRangeSmall 

func (b *BlobStor) GetRangeSmall(prm GetRangeSmallPrm) (GetRangeSmallRes, error)

GetRangeSmall reads data of object payload range from blobovnicza of BLOB storage.

If blobovnicza ID is not set or set to nil, BlobStor tries to get payload range from any blobovnicza.

Returns any error encountered that did not allow to completely read the object payload range.

Returns ErrRangeOutOfBounds if the requested object range is out of bounds. Returns an error of type apistatus.ObjectNotFound if the requested object is missing in blobovnicza(s).

func (*BlobStor) GetSmall 

func (b *BlobStor) GetSmall(prm GetSmallPrm) (GetSmallRes, error)

GetSmall reads the object from blobovnicza of BLOB storage by address.

If blobovnicza ID is not set or set to nil, BlobStor tries to get the object from any blobovnicza.

Returns any error encountered that did not allow to completely read the object.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in blobovnicza(s).

func (*BlobStor) Init 

func (b *BlobStor) Init() error

Init initializes internal data structures and system resources.

If BlobStor is already initialized, no action is taken.

Returns wrapped ErrInitBlobovniczas on blobovnicza tree's initializaiton failure.

func (*BlobStor) Iterate 

func (b *BlobStor) Iterate(prm IteratePrm) (IterateRes, error)

Iterate traverses the storage over the stored objects and calls the handler on each element.

Returns any error encountered that did not allow to completely iterate over the storage.

If handler returns an error, method wraps and returns it immediately.

func (*BlobStor) NeedsCompression added in v0.27.4 

func (b *BlobStor) NeedsCompression(obj *objectSDK.Object) bool

NeedsCompression returns true if the object should be compressed. For an object to be compressed 2 conditions must hold: 1. Compression is enabled in settings. 2. Object MIME Content-Type is allowed for compression.

func (*BlobStor) Open 

func (b *BlobStor) Open(readOnly bool) error

Open opens BlobStor.

func (*BlobStor) Put 

func (b *BlobStor) Put(prm PutPrm) (PutRes, error)

Put saves the object in BLOB storage.

If object is "big", BlobStor saves the object in shallow dir. Otherwise, BlobStor saves the object in blobonicza. In this case the identifier of blobovnicza is returned.

Returns any error encountered that did not allow to completely save the object.

func (*BlobStor) PutRaw added in v0.19.0 

func (b *BlobStor) PutRaw(addr oid.Address, data []byte, compress bool) (PutRes, error)

PutRaw saves an already marshaled object in BLOB storage.

func (*BlobStor) SetLogger added in v0.30.0 

func (b *BlobStor) SetLogger(l *zap.Logger)

SetLogger sets logger. It is used after the shard ID was generated to use it in logs.

func (*BlobStor) SetMode added in v0.30.0 

func (b *BlobStor) SetMode(m mode.Mode) error

SetMode sets the blobstor mode of operation.

type DeleteBigPrm 

type DeleteBigPrm struct {
	// contains filtered or unexported fields
}

DeleteBigPrm groups the parameters of DeleteBig operation.

func (*DeleteBigPrm) SetAddress 

func (a *DeleteBigPrm) SetAddress(addr oid.Address)

SetAddress sets the address of the requested object.

type DeleteBigRes 

type DeleteBigRes struct{}

DeleteBigRes groups the resulting values of DeleteBig operation.

type DeleteSmallPrm 

type DeleteSmallPrm struct {
	// contains filtered or unexported fields
}

DeleteSmallPrm groups the parameters of DeleteSmall operation.

func (*DeleteSmallPrm) SetAddress 

func (a *DeleteSmallPrm) SetAddress(addr oid.Address)

SetAddress sets the address of the requested object.

func (*DeleteSmallPrm) SetBlobovniczaID 

func (v *DeleteSmallPrm) SetBlobovniczaID(id *blobovnicza.ID)

SetBlobovniczaID sets blobovnicza ID.

type DeleteSmallRes 

type DeleteSmallRes struct{}

DeleteSmallRes groups the resulting values of DeleteSmall operation.

type ExistsPrm 

type ExistsPrm struct {
	// contains filtered or unexported fields
}

ExistsPrm groups the parameters of Exists operation.

func (*ExistsPrm) SetAddress 

func (a *ExistsPrm) SetAddress(addr oid.Address)

SetAddress sets the address of the requested object.

type ExistsRes 

type ExistsRes struct {
	// contains filtered or unexported fields
}

ExistsRes groups the resulting values of Exists operation.

func (ExistsRes) Exists 

func (r ExistsRes) Exists() bool

Exists returns the fact that the object is in BLOB storage.

type GetBigPrm 

type GetBigPrm struct {
	// contains filtered or unexported fields
}

GetBigPrm groups the parameters of GetBig operation.

func (*GetBigPrm) SetAddress 

func (a *GetBigPrm) SetAddress(addr oid.Address)

SetAddress sets the address of the requested object.

type GetBigRes 

type GetBigRes struct {
	// contains filtered or unexported fields
}

GetBigRes groups the resulting values of GetBig operation.

func (GetBigRes) Object 

func (o GetBigRes) Object() *object.Object

Object returns the object.

type GetRangeBigPrm 

type GetRangeBigPrm struct {
	// contains filtered or unexported fields
}

GetRangeBigPrm groups the parameters of GetRangeBig operation.

func (*GetRangeBigPrm) SetAddress 

func (a *GetRangeBigPrm) SetAddress(addr oid.Address)

SetAddress sets the address of the requested object.

func (*GetRangeBigPrm) SetRange 

func (r *GetRangeBigPrm) SetRange(rng *object.Range)

SetRange sets range of the object payload.

type GetRangeBigRes 

type GetRangeBigRes struct {
	// contains filtered or unexported fields
}

GetRangeBigRes groups the resulting values of GetRangeBig operation.

func (GetRangeBigRes) RangeData 

func (d GetRangeBigRes) RangeData() []byte

RangeData returns data of the requested payload range.

type GetRangeSmallPrm 

type GetRangeSmallPrm struct {
	// contains filtered or unexported fields
}

GetRangeSmallPrm groups the parameters of GetRangeSmall operation.

func (*GetRangeSmallPrm) SetAddress 

func (a *GetRangeSmallPrm) SetAddress(addr oid.Address)

SetAddress sets the address of the requested object.

func (*GetRangeSmallPrm) SetBlobovniczaID 

func (v *GetRangeSmallPrm) SetBlobovniczaID(id *blobovnicza.ID)

SetBlobovniczaID sets blobovnicza ID.

func (*GetRangeSmallPrm) SetRange 

func (r *GetRangeSmallPrm) SetRange(rng *object.Range)

SetRange sets range of the object payload.

type GetRangeSmallRes 

type GetRangeSmallRes struct {
	// contains filtered or unexported fields
}

GetRangeSmallRes groups the resulting values of GetRangeSmall operation.

func (GetRangeSmallRes) RangeData 

func (d GetRangeSmallRes) RangeData() []byte

RangeData returns data of the requested payload range.

type GetSmallPrm 

type GetSmallPrm struct {
	// contains filtered or unexported fields
}

GetSmallPrm groups the parameters of GetSmallPrm operation.

func (*GetSmallPrm) SetAddress 

func (a *GetSmallPrm) SetAddress(addr oid.Address)

SetAddress sets the address of the requested object.

func (*GetSmallPrm) SetBlobovniczaID 

func (v *GetSmallPrm) SetBlobovniczaID(id *blobovnicza.ID)

SetBlobovniczaID sets blobovnicza ID.

type GetSmallRes 

type GetSmallRes struct {
	// contains filtered or unexported fields
}

GetSmallRes groups the resulting values of GetSmall operation.

func (GetSmallRes) Object 

func (o GetSmallRes) Object() *object.Object

Object returns the object.

type Info 

type Info = fstree.Info

type IteratePrm 

type IteratePrm struct {
	// contains filtered or unexported fields
}

IteratePrm groups the parameters of Iterate operation.

func (*IteratePrm) IgnoreErrors added in v0.27.5 

func (i *IteratePrm) IgnoreErrors()

IgnoreErrors sets the flag signifying whether errors should be ignored.

func (*IteratePrm) SetErrorHandler added in v0.29.0 

func (i *IteratePrm) SetErrorHandler(f func(oid.Address, error) error)

SetErrorHandler sets error handler for objects that cannot be read or unmarshaled.

func (*IteratePrm) SetIterationHandler 

func (i *IteratePrm) SetIterationHandler(h IterationHandler)

SetIterationHandler sets the action to be performed on each iteration.

type IterateRes 

type IterateRes struct{}

IterateRes groups the resulting values of Iterate operation.

type IterationElement added in v0.25.0 

type IterationElement struct {
	// contains filtered or unexported fields
}

IterationElement represents a unit of elements through which Iterate operation passes.

func (IterationElement) Address added in v0.29.0 

func (x IterationElement) Address() oid.Address

Address returns the object address.

func (IterationElement) BlobovniczaID added in v0.25.0 

func (x IterationElement) BlobovniczaID() *blobovnicza.ID

BlobovniczaID returns the identifier of Blobovnicza in which object is stored. Returns nil if the object isn't in Blobovnicza.

func (IterationElement) ObjectData added in v0.25.0 

func (x IterationElement) ObjectData() []byte

ObjectData returns the stored object in a binary representation.

type IterationHandler 

type IterationHandler func(IterationElement) error

IterationHandler is a generic processor of IterationElement.

type Option 

type Option func(*cfg)

Option represents BlobStor's constructor option.

func WithBlobovniczaOpenedCacheSize 

func WithBlobovniczaOpenedCacheSize(sz int) Option

WithBlobovniczaOpenedCacheSize return option to specify maximum number of opened non-active blobovnicza's.

func WithBlobovniczaShallowDepth 

func WithBlobovniczaShallowDepth(d uint64) Option

WithBlobovniczaShallowDepth returns option to specify depth of blobovnicza directories.

func WithBlobovniczaShallowWidth 

func WithBlobovniczaShallowWidth(w uint64) Option

WithBlobovniczaShallowWidth returns option to specify width of blobovnicza directories.

func WithBlobovniczaSize 

func WithBlobovniczaSize(sz uint64) Option

WithBlobovniczaSize returns option to specify maximum volume of each blobovnicza.

func WithCompressObjects 

func WithCompressObjects(comp bool) Option

WithCompressObjects returns option to toggle compression of the stored objects.

If true, Zstandard algorithm is used for data compression.

If compressor (decompressor) creation failed, the uncompressed option will be used, and the error is recorded in the provided log.

func WithLogger 

func WithLogger(l *logger.Logger) Option

WithLogger returns option to specify BlobStor's logger.

func WithRootPath 

func WithRootPath(rootDir string) Option

WithRootPath returns option to set path to root directory of the fs tree to write the objects.

func WithRootPerm 

func WithRootPerm(perm fs.FileMode) Option

WithRootPerm returns option to set permission bits of the fs tree.

func WithShallowDepth 

func WithShallowDepth(depth int) Option

WithShallowDepth returns option to set the depth of the object file subdirectory tree.

Depth is reduced to maximum value in case of overflow.

func WithSmallSizeLimit 

func WithSmallSizeLimit(lim uint64) Option

WithSmallSizeLimit returns option to set maximum size of "small" object.

func WithUncompressableContentTypes added in v0.27.4 

func WithUncompressableContentTypes(values []string) Option

WithUncompressableContentTypes returns option to disable decompression for specific content types as seen by object.AttributeContentType attribute.

type PutPrm 

type PutPrm struct {
	// contains filtered or unexported fields
}

PutPrm groups the parameters of Put operation.

func (*PutPrm) SetObject 

func (o *PutPrm) SetObject(obj *object.Object)

SetObject sets the object.

type PutRes 

type PutRes struct {
	// contains filtered or unexported fields
}

PutRes groups the resulting values of Put operation.

func (PutRes) BlobovniczaID 

func (v PutRes) BlobovniczaID() *blobovnicza.ID

BlobovniczaID returns blobovnicza ID.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.