gotfs

package
v0.0.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2022 License: GPL-3.0 Imports: 21 Imported by: 6

README

GotFS

GotFS is a filesystem implemented on top of a key-value store. The key value store is GotKV, which is an immutable key-value store, sitting on top of a content-addressed data store. Every operation on a GotKV store returns a new reference to a new store with the change. GotFS inherits this property. All operations return a reference to a new filesystem. A GotFS filesystem is just a GotKV store with a specific key structure used to represent files and directories.

Objects

Ref

A reference to data in the content-addressed store. This type is provided by GotKV.

Extent

A part of a file. It includes a Ref, an offset, and a length. It is basically an instruction to:

  1. Get the referenced data from content-addressed storage
  2. Only include the data from offset to offset+length.

This enables packing small files into the same blob.

Info

Information about a file or directory. Most importantly the permissions and type of file.

Key Layout

All objects are represented by an Info entry at a specific key, and content stored under keys prefixed with the metadata key.

File data is stored in a content-addressed store, and references to the data are stored in GotKV.

For example: The file "test.txt" with 10B of data in it would produce the following key value pairs.

/                                -> Info (dir)
/test.txt/                       -> Info (file)
/test.txt/<NULL>< 64 bit: 10 >   -> Extent

A directory is stored as a metadata object.

/                                         -> Info (dir)
/mydir/                                   -> Info (dir)
/mydir/myfile.txt/                        -> Info (file)
/mydir/myfile.txt/<NULL><64 bit offset>   -> Part

It is possible for a file to be at the root

/                            -> Info (file)
/<NULL><64 bits      >       -> Extent
/<NULL>< next offset >       -> Extent

All Info keys end in a trailing /, including regular files. Keys for Info objects contain no NULL characters. Keys for extents contain exactly 1 NULL character 9 bytes from the end of key, separating the path from the offset. Extent keys are always prefixed by the Info key for the file they are part of.

Reading A File

To read from a file in GotFS you first lookup the Info entry for the path of the file. If there is not an entry at the path or the entry is not for a regular file, then return an error.

Then convert the offset to read from to a key. Seek to the first entry after that key. The extent referenced by that entry will end at the offset in the key, and will contain data overlapping the target range.

Documentation

Index

Constants

View Source 
const (
	DefaultMaxBlobSize          = 1 << 21
	DefaultMinBlobSizeData      = 1 << 12
	DefaultMeanBlobSizeData     = 1 << 20
	DefaultMeanBlobSizeMetadata = 1 << 13
)
View Source 
const MaxPathLen = gotkv.MaxKeySize - 2 - 8
View Source 
const Sep = '/'

Variables

View Source 
var File_models_proto protoreflect.FileDescriptor

Functions

func Dump 

func Dump(ctx context.Context, s Store, root Root, w io.Writer) error

func Equal 

func Equal(a, b Root) bool

Equal returns true if a and b contain equivalent data.

func IsEmpty 

func IsEmpty(root Root) bool

func SpanForPath 

func SpanForPath(p string) gotkv.Span

func SplitPath added in v0.0.4 

func SplitPath(p string) []string

Types

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder manages building a filesystem.

func (*Builder) BeginFile 

func (b *Builder) BeginFile(p string, mode os.FileMode) error

BeginFile creates a metadata entry for a regular file at p and directs Write calls to the content of that file.

func (*Builder) Finish 

func (b *Builder) Finish() (*Root, error)

Finish closes the builder and returns the Root to the filesystem. Finish is idempotent, and is safe to call multiple times. Not calling finish is not an error, the builder does not allocate resources other than memory.

func (*Builder) IsFinished 

func (b *Builder) IsFinished() bool

func (*Builder) Mkdir 

func (b *Builder) Mkdir(p string, mode os.FileMode) error

Mkdir creates a directory for p.

func (*Builder) Write 

func (b *Builder) Write(data []byte) (int, error)

func (*Builder) WriteExtents 

func (b *Builder) WriteExtents(ctx context.Context, exts []*Extent) error

WriteExtents adds extents to the current file. Rechunking is avoided, but the last Extent could be short, so it must be rechunked regardless. WriteExtents is useful for efficiently joining Extents from disjoint regions of a file. See also: Operator.CreateExtents

type DirEnt 

type DirEnt struct {
	Name string
	Mode os.FileMode
}

type Extent 

type Extent = gotlob.Extent

type Info added in v0.0.3 

type Info struct {
	Mode   uint32            `protobuf:"varint,1,opt,name=mode,proto3" json:"mode,omitempty"`
	Labels map[string]string `` /* 153-byte string literal not displayed */
	// contains filtered or unexported fields
}

func (*Info) Descriptor deprecated added in v0.0.3 

func (*Info) Descriptor() ([]byte, []int)

Deprecated: Use Info.ProtoReflect.Descriptor instead.

func (*Info) GetLabels added in v0.0.3 

func (x *Info) GetLabels() map[string]string

func (*Info) GetMode added in v0.0.3 

func (x *Info) GetMode() uint32

func (*Info) ProtoMessage added in v0.0.3 

func (*Info) ProtoMessage()

func (*Info) ProtoReflect added in v0.0.3 

func (x *Info) ProtoReflect() protoreflect.Message

func (*Info) Reset added in v0.0.3 

func (x *Info) Reset()

func (*Info) String added in v0.0.3 

func (x *Info) String() string

type Operator 

type Operator struct {
	// contains filtered or unexported fields
}

func NewOperator 

func NewOperator(opts ...Option) Operator

func (*Operator) AddPrefix 

func (o *Operator) AddPrefix(root Root, p string) Root

func (*Operator) Check 

func (o *Operator) Check(ctx context.Context, s Store, root Root, checkData func(ref gdat.Ref) error) error

func (*Operator) CreateExtents 

func (o *Operator) CreateExtents(ctx context.Context, ds Store, r io.Reader) ([]*Extent, error)

CreateExtents returns a list of extents created from r

func (*Operator) CreateFile 

func (o *Operator) CreateFile(ctx context.Context, ms, ds Store, x Root, p string, r io.Reader) (*Root, error)

CreateFile creates a file at p with data from r If there is an entry at p CreateFile returns an error ms is the store used for metadata ds is the store used for data.

func (*Operator) CreateFileRoot 

func (o *Operator) CreateFileRoot(ctx context.Context, ms, ds Store, r io.Reader) (*Root, error)

CreateFileRoot creates a new filesystem with the contents read from r at the root

func (*Operator) ForEach 

func (o *Operator) ForEach(ctx context.Context, s cadata.Store, root Root, p string, fn func(p string, md *Info) error) error

func (*Operator) ForEachLeaf added in v0.0.4 

func (o *Operator) ForEachLeaf(ctx context.Context, s cadata.Store, root Root, p string, fn func(p string, md *Info) error) error

ForEachLeaf calls fn with each regular file in root, beneath p.

func (*Operator) GetDirInfo added in v0.0.3 

func (o *Operator) GetDirInfo(ctx context.Context, s Store, x Root, p string) (*Info, error)

GetDirInfo returns directory metadata at p if it exists, and errors otherwise

func (*Operator) GetFileInfo added in v0.0.3 

func (o *Operator) GetFileInfo(ctx context.Context, s Store, x Root, p string) (*Info, error)

GetFileInfo returns the file metadata at p if it exists, and errors otherwise

func (*Operator) GetInfo added in v0.0.3 

func (o *Operator) GetInfo(ctx context.Context, s Store, x Root, p string) (*Info, error)

GetInfo retrieves the metadata at p if it exists and errors otherwise

func (*Operator) Graft 

func (o *Operator) Graft(ctx context.Context, ms, ds cadata.Store, root Root, p string, branch Root) (*Root, error)

Graft places branch at p in root. If p == "" then branch is returned unaltered.

func (*Operator) MaxInfo added in v0.0.4 

func (o *Operator) MaxInfo(ctx context.Context, ms cadata.Store, root Root, span Span) (string, *Info, error)

MaxInfo returns the maximum path and the corresponding Info for the path. If no Info entry can be found MaxInfo returns ("", nil, nil)

func (*Operator) MeanBlobSizeData added in v0.0.4 

func (o *Operator) MeanBlobSizeData() int

func (*Operator) MeanBlobSizeMetadata added in v0.0.4 

func (o *Operator) MeanBlobSizeMetadata() int

func (*Operator) Mkdir 

func (o *Operator) Mkdir(ctx context.Context, s Store, x Root, p string) (*Root, error)

Mkdir creates a directory at path p

func (*Operator) MkdirAll 

func (o *Operator) MkdirAll(ctx context.Context, s Store, x Root, p string) (*Root, error)

MkdirAll creates the directory p and any of p's ancestors if necessary.

func (*Operator) NewBuilder 

func (o *Operator) NewBuilder(ctx context.Context, ms, ds Store) *Builder

func (*Operator) NewEmpty 

func (o *Operator) NewEmpty(ctx context.Context, s Store) (*Root, error)

NewEmpty creates a new filesystem with nothing in it.

func (*Operator) NewReader 

func (o *Operator) NewReader(ctx context.Context, ms, ds Store, x Root, p string) (*Reader, error)

NewReader returns an io.Reader | io.Seeker | io.ReaderAt

func (*Operator) Populate added in v0.0.4 

func (o *Operator) Populate(ctx context.Context, s Store, root Root, mdSet, dataSet cadata.Set) error

Populate adds the ID for all the metadata blobs to mdSet and all the data blobs to dataSet

func (*Operator) PutInfo added in v0.0.3 

func (o *Operator) PutInfo(ctx context.Context, s Store, x Root, p string, md *Info) (*Root, error)

PutInfo assigns metadata to p

func (*Operator) ReadDir 

func (o *Operator) ReadDir(ctx context.Context, s Store, x Root, p string, fn func(e DirEnt) error) error

ReadDir calls fn for every child of the directory at p.

func (*Operator) ReadFileAt 

func (o *Operator) ReadFileAt(ctx context.Context, ms, ds Store, x Root, p string, start int64, buf []byte) (int, error)

ReadFileAt fills `buf` with data in the file at `p` starting at offset `start`

func (*Operator) RemoveAll 

func (o *Operator) RemoveAll(ctx context.Context, s Store, x Root, p string) (*Root, error)

func (*Operator) Select 

func (o *Operator) Select(ctx context.Context, s cadata.Store, root Root, p string) (*Root, error)

Select returns a new root containing everything under p, shifted to the root.

func (*Operator) SizeOfFile 

func (o *Operator) SizeOfFile(ctx context.Context, s Store, x Root, p string) (uint64, error)

SizeOfFile returns the size of the file at p in bytes.

func (*Operator) Splice 

func (o *Operator) Splice(ctx context.Context, ms, ds Store, segs []Segment) (*Root, error)

func (*Operator) Sync added in v0.0.4 

func (o *Operator) Sync(ctx context.Context, srcMeta, srcData, dstMeta, dstData Store, root Root) error

Sync ensures dst has all the data reachable from root dst and src should both be metadata stores. copyData will be called to sync metadata

type Option 

type Option func(o *Operator)

func WithContentCacheSize 

func WithContentCacheSize(n int) Option

WithContentCacheSize sets the size of the cache for raw data

func WithMetaCacheSize 

func WithMetaCacheSize(n int) Option

WithMetaCacheSize sets the size of the cache for metadata

func WithSalt added in v0.0.4 

func WithSalt(salt *[32]byte) Option

type Reader added in v0.0.4 

type Reader = gotlob.Reader

type Ref 

type Ref = gotkv.Ref

type Root 

type Root = gotkv.Root

type Segment 

type Segment struct {
	Span gotkv.Span
	Root Root
}

Segment is a span of a GotFS instance.

func ChangesOnBase 

func ChangesOnBase(base Root, changes []Segment) []Segment

ChangesOnBase inserts segments from base between each Segment in changes.

func (Segment) String 

func (s Segment) String() string

type Span added in v0.0.4 

type Span = gotkv.Span

type Store 

type Store = gotkv.Store

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.