storage

Documentation

Overview

Package storage contains all function related to local and cloud storage

Index

• type LocalStoreClient
  • func (client *LocalStoreClient) WriteToBinFile(path, objectName string, object []byte) error
• type Manager
  • func NewStorageManager(conf *config.StorageConfig) (*Manager, error)
  • func (manager *Manager) Close()
  • func (manager *Manager) GenerateLocation(ctx context.Context, segmentName string, replicaSegmentAvro []byte) (string, cid.Cid)
  • func (manager *Manager) Store(ctx context.Context, ccid cid.Cid, filename string, data []byte) error

Types

type LocalStoreClient

type LocalStoreClient struct{}

LocalStoreClient stores file on the local filesystem

func (*LocalStoreClient) WriteToBinFile

func (client *LocalStoreClient) WriteToBinFile(path, objectName string, object []byte) error

WriteToBinFile writes to binary file in given path

type Manager

type Manager struct {
	StorageConfig *config.StorageConfig

	GcpStore   *gcp.Client
	LocalStore *LocalStoreClient
	IpfsStore  *pinner.PinnerNode
	// contains filtered or unexported fields
}

Manager composes of all the different storage types supported by the agent

func NewStorageManager

func NewStorageManager(conf *config.StorageConfig) (*Manager, error)

NewStorageManager creates and sets up a new storage manager with given config

func (*Manager) Close

func (manager *Manager) Close()

Close the stores which compose the manager

func (*Manager) GenerateLocation

func (manager *Manager) GenerateLocation(ctx context.Context, segmentName string, replicaSegmentAvro []byte) (string, cid.Cid)

GenerateLocation calculates the non-local location (gcp or ipfs) for the given segment and data, cid is returned in case of ipfs

func (*Manager) Store

func (manager *Manager) Store(ctx context.Context, ccid cid.Cid, filename string, data []byte) error

Store the given data in the stores supported by the agent. cid is needed for IFPS based stores, and cid.Undef can be passed in case IPFS is not needed.