modulestore

Documentation

Index

• type ID
  • func IDFromString(s string) (ID, error)
  • func (id *ID) MarshalJSON() ([]byte, error)
  • func (id *ID) String() string
  • func (id *ID) UnmarshalJSON(b []byte) error
• type ModuleReader
  • func (m *ModuleReader) PreferredReadSize() uint
  • func (m *ModuleReader) Size() uint
• type Store
  • func New(s3client *s3.Client, publicReadURL, s3Bucket, localCachePath string) (*Store, error)
  • func (store *Store) InsertModuleLocally(localPath string) (id ID, isNew bool, err error)
  • func (store *Store) IsPresentLocally(id ID) (bool, error)
  • func (store *Store) IsPresentRemotely(id ID) (bool, error)
  • func (store *Store) ListLocalModules() (libpf.Set[ID], error)
  • func (store *Store) ListRemoteModules() (map[ID]time.Time, error)
  • func (store *Store) OpenBufferedReadAt(id ID, cacheSizeBytes uint) (*ModuleReader, error)
  • func (store *Store) OpenReadAt(id ID) (*ModuleReader, error)
  • func (store *Store) RemoveLocalModule(id ID) error
  • func (store *Store) RemoveLocalTempFiles() error
  • func (store *Store) RemoveRemoteModule(id ID) error
  • func (store *Store) UnpackModule(id ID, out io.Writer) error
  • func (store *Store) UnpackModuleToPath(id ID, outPath string) error
  • func (store *Store) UploadModule(id ID) error

Types

type ID

type ID struct {
	// contains filtered or unexported fields
}

ID is used to uniquely identify a module in a `Store`.

func IDFromString

func IDFromString(s string) (ID, error)

IDFromString parses a string into an ID.

func (*ID) MarshalJSON

func (id *ID) MarshalJSON() ([]byte, error)

MarshalJSON encodes the ID into JSON.

func (*ID) String

func (id *ID) String() string

String implements the `fmt.Stringer` interface

func (*ID) UnmarshalJSON

func (id *ID) UnmarshalJSON(b []byte) error

UnmarshalJSON decodes JSON into an ID.

type ModuleReader

type ModuleReader struct {
	io.ReaderAt
	io.Closer
	// contains filtered or unexported fields
}

ModuleReader allows reading a module from the module store.

func (*ModuleReader) PreferredReadSize

func (m *ModuleReader) PreferredReadSize() uint

PreferredReadSize returns the preferred size and alignment of reads on this reader.

func (*ModuleReader) Size

func (m *ModuleReader) Size() uint

Size returns the uncompressed size of the module.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is a compressed storage for large binary files (modules). Upon inserting a new file, the caller receives a unique ID to identify the file by. This ID can then later be used to retrieve the module again. Files are transparently compressed upon insertion and lazily decompressed during reading. Modules can be pushed to a remote backing storage in the form of an S3 bucket. Modules present remotely but not locally are automatically downloaded when needed.

It is safe to create multiple `Store` instances for the same local directory and remote bucket at the same time, also when created within multiple different applications.

func New

func New(s3client *s3.Client, publicReadURL, s3Bucket, localCachePath string) (*Store, error)

New creates a new module storage. The modules present in the local cache are inspected and a full index of the modules in the remote S3 bucket is retrieved and cached as well.

func (*Store) InsertModuleLocally

func (store *Store) InsertModuleLocally(localPath string) (id ID, isNew bool, err error)

InsertModuleLocally places a file into the local cache, returning an ID to refer it by in the future. The file is **not** uploaded to the remote storage automatically. If the file was already previously present in the local store, the function returns the ID of the existing file.

func (*Store) IsPresentLocally

func (store *Store) IsPresentLocally(id ID) (bool, error)

IsPresentLocally checks whether a module is present in the local cache.

func (*Store) IsPresentRemotely

func (store *Store) IsPresentRemotely(id ID) (bool, error)

IsPresentRemotely checks whether a module is present in the remote data-store.

func (*Store) ListLocalModules

func (store *Store) ListLocalModules() (libpf.Set[ID], error)

ListLocalModules creates a set of all modules present in the local cache.

func (*Store) ListRemoteModules

func (store *Store) ListRemoteModules() (map[ID]time.Time, error)

ListRemoteModules creates a map of all modules present in the remote storage and their date of last change.

func (*Store) OpenBufferedReadAt

func (store *Store) OpenBufferedReadAt(id ID, cacheSizeBytes uint) (
	*ModuleReader, error)

OpenBufferedReadAt is a buffered version of `OpenReadAt`.

func (*Store) OpenReadAt

func (store *Store) OpenReadAt(id ID) (*ModuleReader, error)

OpenReadAt opens a file in the store for random-access reading.

func (*Store) RemoveLocalModule

func (store *Store) RemoveLocalModule(id ID) error

RemoveLocalModule removes a module from the local cache. No-op if not present.

func (*Store) RemoveLocalTempFiles

func (store *Store) RemoveLocalTempFiles() error

RemoveLocalTempFiles removes all lingering temporary files that were never fully committed.

If multiple instances of `Store` exist for the same cache directory, this may with uncommitted writes of the other instance.

func (*Store) RemoveRemoteModule

func (store *Store) RemoveRemoteModule(id ID) error

RemoveRemoteModule removes a module from the remote storage. No-op if not present.

func (*Store) UnpackModule

func (store *Store) UnpackModule(id ID, out io.Writer) error

UnpackModule extracts a module from the store, writing it to the given writer.

func (*Store) UnpackModuleToPath

func (store *Store) UnpackModuleToPath(id ID, outPath string) error

UnpackModuleToPath extracts a module from the store to the given local path.

func (*Store) UploadModule

func (store *Store) UploadModule(id ID) error

UploadModule uploads a module from the local storage to the remote. If the module is already present, no operation is performed.