Documentation
¶
Index ¶
- Constants
- type FileSet
- type Option
- type Reader
- type ShardFunc
- type Storage
- func (s *Storage) Merge(ctx context.Context, outputFileSet string, inputFileSets []string, ...) error
- func (s *Storage) New(ctx context.Context, fileSet string, opts ...Option) *FileSet
- func (s *Storage) NewReader(ctx context.Context, fileSet string, opts ...index.Option) *Reader
- func (s *Storage) NewWriter(ctx context.Context, fileSet string) *Writer
- func (s *Storage) Shard(ctx context.Context, fileSets []string, shardFunc ShardFunc) error
- type StorageOption
- type Writer
Constants ¶
const ( // DefaultMemoryThreshold is the default for the memory threshold that must // be met before a file set part is serialized (excluding close). DefaultMemoryThreshold = 1024 * chunk.MB // DefaultShardThreshold is the default for the size threshold that must // be met before a shard is created by the shard function. DefaultShardThreshold = 1024 * chunk.MB // Compacted is the suffix of a path that points to the compaction of the prefix. Compacted = "compacted" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type FileSet ¶ added in v1.9.6
type FileSet struct {
// contains filtered or unexported fields
}
FileSet is a set of files. This may be a full filesystem or a subfilesystem (e.g. datum / datum set / shard).
func (*FileSet) Delete ¶ added in v1.9.6
Delete deletes a file from the file set. (bryce) might need to delete ancestor directories in certain cases.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader reads the serialized format of a fileset.
type Storage ¶
type Storage struct {
// contains filtered or unexported fields
}
Storage is the abstraction that manages fileset storage.
func NewStorage ¶
NewStorage creates a new Storage.
func (*Storage) Merge ¶ added in v1.9.8
func (s *Storage) Merge(ctx context.Context, outputFileSet string, inputFileSets []string, opts ...index.Option) error
Merge merges the file sets with the passed in prefix.
func (*Storage) NewReader ¶
NewReader creates a new Reader for a file set. (bryce) expose some notion of read ahead (read a certain number of chunks in parallel). this will be necessary to speed up reading large files.
type StorageOption ¶ added in v1.9.8
type StorageOption func(s *Storage)
StorageOption configures a storage.
func ServiceEnvToOptions ¶ added in v1.9.8
func ServiceEnvToOptions(env *serviceenv.ServiceEnv) []StorageOption
ServiceEnvToOptions converts a service environment configuration (specifically the storage configuration) to a set of storage options.
func WithMemoryThreshold ¶ added in v1.9.8
func WithMemoryThreshold(threshold int64) StorageOption
WithMemoryThreshold sets the memory threshold that must be met before a file set part is serialized (excluding close).
func WithShardThreshold ¶ added in v1.9.8
func WithShardThreshold(threshold int64) StorageOption
WithShardThreshold sets the size threshold that must be met before a shard is created by the shard function.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer writes the serialized format of a fileset. The serialized format of a fileset consists of indexes and content which are both realized as compressed tar stream chunks.
func (*Writer) Close ¶
Close closes the writer. (bryce) is not closing the tar writer the right choice here? We cannot include it in the last file's range because it will effect the hash. Our indexing is what will exit the reading of the tar stream, not the end of tar entry (two empty 512 bytes).
func (*Writer) CopyFiles ¶ added in v1.9.5
CopyFiles does a cheap copy of files from a reader to a writer. (bryce) need to handle delete operations.
func (*Writer) CopyTags ¶ added in v1.9.5
CopyTags does a cheap copy of tagged file data from a reader to a writer.
func (*Writer) StartTag ¶
StartTag starts a tag for the next set of bytes (used for the reverse index, mapping file output to datums).