slicer

package
v1.0.0-rc.11 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 7, 2023 License: Apache-2.0 Imports: 19 Imported by: 2

Documentation

Overview

Package slicer provides raw data slicing into NeoFS objects.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrIncompleteHeader indicates some fields are missing in header.
	ErrIncompleteHeader = errors.New("incomplete header")
)

Functions

func Put

func Put(ctx context.Context, ow ObjectWriter, header object.Object, signer user.Signer, data io.Reader, opts Options) (oid.ID, error)

Put works similar to Slicer.Put, but allows flexible configuration of object header. The method accepts Options for adjusting max object size, epoch, session token, etc.

Types

type NetworkedClient

type NetworkedClient interface {
	ObjectWriter

	NetworkInfo(ctx context.Context, prm client.PrmNetworkInfo) (netmap.NetworkInfo, error)
}

NetworkedClient represents a virtual object recorder with possibility to get actual netmap.NetworkInfo data.

type ObjectWriter

type ObjectWriter interface {
	// ObjectPutInit initializes and returns a stream of writable data associated
	// with the object according to its header. Provided header includes at least
	// container, owner and object ID fields.
	//
	// Signer is required and must not be nil. The operation is executed on behalf of
	// the account corresponding to the specified Signer, which is taken into account, in particular, for access control.
	ObjectPutInit(ctx context.Context, hdr object.Object, signer user.Signer, prm client.PrmObjectPutInit) (client.ObjectWriter, error)
}

ObjectWriter represents a virtual object recorder.

type Options

type Options struct {
	// contains filtered or unexported fields
}

Options groups Slicer options.

func (*Options) BearerToken

func (x *Options) BearerToken() *bearer.Token

BearerToken returns bearer token.

func (*Options) CalculateHomomorphicChecksum

func (x *Options) CalculateHomomorphicChecksum()

CalculateHomomorphicChecksum makes Slicer to calculate and set homomorphic checksum of the processed objects.

func (*Options) CopiesNumber

func (x *Options) CopiesNumber() uint32

CopiesNumber returns the number of object copies.

func (*Options) CurrentNeoFSEpoch

func (x *Options) CurrentNeoFSEpoch() uint64

CurrentNeoFSEpoch returns epoch.

func (*Options) IsHomomorphicChecksumEnabled

func (x *Options) IsHomomorphicChecksumEnabled() bool

IsHomomorphicChecksumEnabled indicates homomorphic checksum calculation status.

func (*Options) ObjectPayloadLimit

func (x *Options) ObjectPayloadLimit() uint64

ObjectPayloadLimit returns required max object size.

func (*Options) Session

func (x *Options) Session() *session.Object

Session returns session object.

func (*Options) SetBearerToken

func (x *Options) SetBearerToken(bearerToken bearer.Token)

SetBearerToken allows to attach signed Extended ACL rules to the request.

func (*Options) SetCopiesNumber

func (x *Options) SetCopiesNumber(copiesNumber uint32)

SetCopiesNumber sets the minimal number of copies (out of the number specified by container placement policy) for the object PUT operation to succeed. This means that object operation will return with successful status even before container placement policy is completely satisfied.

func (*Options) SetCurrentNeoFSEpoch

func (x *Options) SetCurrentNeoFSEpoch(e uint64)

SetCurrentNeoFSEpoch sets current NeoFS epoch.

func (*Options) SetObjectPayloadLimit

func (x *Options) SetObjectPayloadLimit(l uint64)

SetObjectPayloadLimit specifies data size limit for produced physically stored objects.

func (*Options) SetSession

func (x *Options) SetSession(sess session.Object)

SetSession sets session object.

type PayloadWriter

type PayloadWriter struct {
	// contains filtered or unexported fields
}

PayloadWriter is a single-object payload stream provided by Slicer.

func InitPut

func InitPut(ctx context.Context, ow ObjectWriter, header object.Object, signer user.Signer, opts Options) (*PayloadWriter, error)

InitPut works similar to slicer.Put, but provides ObjectWriter allowing the caller to write data himself.

func (*PayloadWriter) Close

func (x *PayloadWriter) Close() error

Close finalizes object with written payload data, saves the object and closes the stream. Reference to the stored object can be obtained by ID method.

func (*PayloadWriter) ID

func (x *PayloadWriter) ID() oid.ID

ID returns unique identifier of the stored object representing its reference in the configured container.

ID MUST NOT be called before successful Close (undefined behavior otherwise).

func (*PayloadWriter) Write

func (x *PayloadWriter) Write(chunk []byte) (int, error)

Write writes next chunk of the object data. Concatenation of all chunks forms the payload of the final object. When the data is over, the PayloadWriter should be closed.

type Slicer

type Slicer struct {
	// contains filtered or unexported fields
}

Slicer converts input raw data streams into NeoFS objects. Working Slicer must be constructed via New.

func New

func New(ctx context.Context, nw NetworkedClient, signer user.Signer, cnr cid.ID, owner user.ID, sessionToken *session.Object) (*Slicer, error)

New constructs Slicer which writes sliced ready-to-go objects owned by particular user into the specified container using provided ObjectWriter. All objects are signed using provided neofscrypto.Signer.

If ObjectWriter returns data streams which provide io.Closer, they are closed in Slicer.Slice after the payload of any object has been written. In this case, Slicer.Slice fails immediately on Close error.

NetworkedClient parameter allows to extract all required network-depended information for default Slicer behavior tuning.

If payload size limit is specified via Options.SetObjectPayloadLimit, outgoing objects has payload not bigger than the limit. NeoFS stores the corresponding value in the network configuration. Ignore this option if you don't (want to) have access to it. By default, single object is limited by 1MB. Slicer uses this value to enforce the maximum object payload size limit described in the NeoFS Specification. If the total amount of data exceeds the specified limit, Slicer applies the slicing algorithm described within the same specification. The outcome will be a group of "small" objects containing a chunk of data, as well as an auxiliary linking object. All derived objects are written to the parameterized ObjectWriter. If the amount of data is within the limit, one object is produced. Note that Slicer can write multiple objects, but returns the root object ID only.

Parameter sessionToken may be nil, if no session is used.

func (*Slicer) InitPut

func (x *Slicer) InitPut(ctx context.Context, attrs []object.Attribute) (*PayloadWriter, error)

InitPut works similar to Slicer.Put but provides PayloadWriter allowing the caller to write data himself.

func (*Slicer) Put

func (x *Slicer) Put(ctx context.Context, data io.Reader, attrs []object.Attribute) (oid.ID, error)

Put creates new NeoFS object from the input data stream, associates the object with the configured container and writes the object via underlying ObjectWriter. After a successful write, Put returns an oid.ID which is a unique reference to the object in the container. Put sets all required calculated fields like payload length, checksum, etc.

Put allows you to specify object.Attribute parameters to be written to the resulting object's metadata. Keys SHOULD NOT start with system-reserved '__NEOFS__' prefix.

See New for details.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL