hashfs

package
v0.0.0-...-7220889 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 25, 2025 License: BSD-3-Clause Imports: 34 Imported by: 0

Documentation

Overview

Package hashfs provides a filesystem with digest hash.

Index

Constants

This section is empty.

Variables

View Source 
var DigestSemaphore = semaphore.New("file-digest", runtimex.NumCPU())

DigestSemaphore is a semaphore to control concurrent digest calculation.

View Source 
var FlushSemaphore = semaphore.New("fs-flush", runtimex.NumCPU()*2)

FlushSemaphore is a semaphore to control concurrent flushes.

View Source 
var ForgetMissingsSemaphore = semaphore.New("fs-forget", runtimex.NumCPU()*2)

ForgetMissingsSemaphore is a semaphore to control concurrent ForgetMissings. os.Lstat in ForgetMissings would create lots of thread. b/325565625

Functions

func Load 

func Load(ctx context.Context, opts Option) (*pb.State, error)

Load loads a HashFS's state.

func Save 

func Save(ctx context.Context, state *pb.State, opts Option) error

Save persists state in fname.

func StateMap 

func StateMap(s *pb.State) map[string]*pb.Entry

Types

type DataSource 

type DataSource interface {
	Source(context.Context, digest.Digest, string) digest.Source
}

DataSource is an interface to get digest source for digest and its name.

type Dir 

type Dir struct {
	// contains filtered or unexported fields
}

Dir implements https://pkg.go.dev/io/fs#ReadDirFile.

func (*Dir) Close 

func (d *Dir) Close() error

Close closes the directory.

func (*Dir) Read 

func (d *Dir) Read(buf []byte) (int, error)

Read reads contents from the dir (permission denied).

func (*Dir) ReadDir 

func (d *Dir) ReadDir(n int) ([]fs.DirEntry, error)

ReadDir reads directory entries from the dir.

func (*Dir) Stat 

func (d *Dir) Stat() (fs.FileInfo, error)

Stat returns a FileInfo describing the directory.

type DirEntry 

type DirEntry struct {
	// contains filtered or unexported fields
}

DirEntry implements https://pkg.go.dev/io/fs#DirEntry.

func (DirEntry) Info 

func (de DirEntry) Info() (fs.FileInfo, error)

Info returns a FileInfo.

func (DirEntry) IsDir 

func (de DirEntry) IsDir() bool

IsDir returns true if it is a directory.

func (DirEntry) Name 

func (de DirEntry) Name() string

Name is a base name in the directory.

func (DirEntry) Type 

func (de DirEntry) Type() fs.FileMode

Type returns a file type.

type FSMonitor 

type FSMonitor interface {
	// ClockToken returns a token that represents the check time.
	ClockToken(context.Context) (string, error)

	// Scan returns FileInfoer from last checked token.
	Scan(context.Context, string) (FileInfoer, error)
}

FSMonitor provides a way to access file info on the disk for the entry.

type File 

type File struct {
	// contains filtered or unexported fields
}

File implements https://pkg.go.dev/io/fs#File. This is an in-memory representation of the file.

func (*File) Close 

func (f *File) Close() error

Close closes the file.

func (*File) Read 

func (f *File) Read(buf []byte) (int, error)

Read reads contents from the file.

func (*File) Stat 

func (f *File) Stat() (fs.FileInfo, error)

Stat returns a FileInfo describing the file.

type FileInfo 

type FileInfo struct {
	// contains filtered or unexported fields
}

FileInfo implements https://pkg.go.dev/io/fs#FileInfo.

func (FileInfo) Action 

func (fi FileInfo) Action() digest.Digest

Action returns a digest of action that created the file.

func (FileInfo) CmdHash 

func (fi FileInfo) CmdHash() []byte

CmdHash returns a cmdhash that created the file.

func (FileInfo) IsChanged 

func (fi FileInfo) IsChanged() bool

IsChanged returns true if file has been changed in the session.

func (FileInfo) IsDir 

func (fi FileInfo) IsDir() bool

IsDir returns true if it is the directory.

func (FileInfo) IsMissingChecked 

func (fi FileInfo) IsMissingChecked() bool

IsMissingChecked returns true if file has been checked existence for ForgetMissings.

func (FileInfo) ModTime 

func (fi FileInfo) ModTime() time.Time

ModTime is a modification time of the file.

func (FileInfo) Mode 

func (fi FileInfo) Mode() fs.FileMode

Mode is a file mode of the file.

func (FileInfo) Name 

func (fi FileInfo) Name() string

Name is a base name of the file.

func (*FileInfo) Path 

func (fi *FileInfo) Path() string

func (FileInfo) Size 

func (fi FileInfo) Size() int64

Size is a size of the file.

func (FileInfo) Sys 

func (fi FileInfo) Sys() any

Sys returns merkletree Entry of the file. For local file, digest may not be calculated yet. Use Entries to get correct merkletree.Entry.

func (FileInfo) Target 

func (fi FileInfo) Target() string

Target returns a symlink target of the file, or empty if it is not symlink.

func (FileInfo) UpdatedTime 

func (fi FileInfo) UpdatedTime() time.Time

UpdatedTime is a update time of the file. Usually it is the same with ModTime, but may differ for restat=1.

type FileInfoer 

type FileInfoer interface {
	FileInfo(context.Context, *pb.Entry) (fs.FileInfo, error)
}

FileInfoer get fs.FileInfo for *pb.Entry from the disk.

type FileSystem 

type FileSystem struct {
	// contains filtered or unexported fields
}

FileSystem provides fs.{FS,ReadDirFS,ReadFileFS,StatFS,SubFS} interfaces.

func (FileSystem) Open 

func (fsys FileSystem) Open(name string) (fs.File, error)

Open opens a file for name.

func (FileSystem) ReadDir 

func (fsys FileSystem) ReadDir(name string) ([]fs.DirEntry, error)

ReadDir reads directory at name.

func (FileSystem) ReadFile 

func (fsys FileSystem) ReadFile(name string) ([]byte, error)

ReadFile reads contents of name.

func (FileSystem) Stat 

func (fsys FileSystem) Stat(name string) (fs.FileInfo, error)

Stat gets stat of name.

func (FileSystem) Sub 

func (fsys FileSystem) Sub(dir string) (fs.FS, error)

Sub returns an FS corresponding to the subtree rooted at dir.

type HashFS 

type HashFS struct {

	// OS wraps of OS I/O operations in the HashFS.
	OS *osfs.OSFS
	// contains filtered or unexported fields
}

HashFS is a filesystem for digest hash.

func New 

func New(ctx context.Context, opt Option) (*HashFS, error)

New creates a HashFS.

func (*HashFS) Availables 

func (hfs *HashFS) Availables(ctx context.Context, root string, inputs []string) []string

Availables returns valid inputs (i.e. exist in hashfs).

func (*HashFS) Close 

func (hfs *HashFS) Close(ctx context.Context) error

Close closes the HashFS. Persists current state in opt.StateFile.

func (*HashFS) Copy 

func (hfs *HashFS) Copy(ctx context.Context, root, src, dst string, mtime time.Time, cmdhash []byte) error

Copy copies a file from root/src to root/dst with mtime and cmdhash. if src is dir, returns error.

func (*HashFS) DataSource 

func (hfs *HashFS) DataSource() DataSource

DataSource returns DataSource of the HashFS.

func (*HashFS) Entries 

func (hfs *HashFS) Entries(ctx context.Context, root string, inputs []string) ([]merkletree.Entry, error)

Entries gets merkletree entries for inputs at root. it won't return entries symlink escaped from root.

func (*HashFS) FileSystem 

func (hfs *HashFS) FileSystem(ctx context.Context, dir string) FileSystem

FileSystem returns FileSystem interface at dir.

func (*HashFS) Flush 

func (hfs *HashFS) Flush(ctx context.Context, execRoot string, files []string) error

Flush flushes cached information for files under execRoot to local disk.

func (*HashFS) Forget 

func (hfs *HashFS) Forget(ctx context.Context, root string, inputs []string)

Forget forgets cached entry for inputs under root.

func (*HashFS) ForgetMissings 

func (hfs *HashFS) ForgetMissings(ctx context.Context, root string, inputs []string) []string

ForgetMissings forgets cached entry for input under root if it doesn't exist on local disk, and returns valid inputs. It is currently used for deps=msvc only to workaround clang-cl issue. https://github.com/llvm/llvm-project/issues/58726

func (*HashFS) ForgetMissingsInDir 

func (hfs *HashFS) ForgetMissingsInDir(ctx context.Context, root, dir string)

ForgetMissingsInDir forgets cached entry under root/dir if it isn't generated files/dirs by any steps and doesn't exist on local disk. It is used for a step that removes files under a dir. b/350662100

func (*HashFS) IsClean 

func (hfs *HashFS) IsClean() bool

IsClean returns whether hashfs is clean (i.e. sync with local disk).

func (*HashFS) LoadErr 

func (hfs *HashFS) LoadErr() error

LoadErr returns load error.

func (*HashFS) Mkdir 

func (hfs *HashFS) Mkdir(ctx context.Context, root, dirname string, cmdhash []byte) error

Mkdir makes a directory at root/dirname.

func (*HashFS) NeedFlush 

func (hfs *HashFS) NeedFlush(ctx context.Context, execRoot, fname string) bool

NeedFlush returns whether the fname need to be flushed based on OutputLocal option.

func (*HashFS) Notify 

func (hfs *HashFS) Notify(f NotifyFunc)

Notify causes the hashfs to relay filesystem motification to f.

func (*HashFS) OnCog 

func (hfs *HashFS) OnCog() bool

OnCog returns whether it is on Cog or not.

func (*HashFS) PreviouslyGeneratedFiles 

func (hfs *HashFS) PreviouslyGeneratedFiles() []string

PreviouslyGeneratedFiles returns a list of generated files (i.e. has cmdhash) in the previous builds. It will reset internal data, so next call will return nil

func (*HashFS) ReadDir 

func (hfs *HashFS) ReadDir(ctx context.Context, root, name string) (dents []DirEntry, err error)

ReadDir returns directory entries of root/name.

func (*HashFS) ReadFile 

func (hfs *HashFS) ReadFile(ctx context.Context, root, fname string) ([]byte, error)

ReadFile reads a contents of root/fname.

func (*HashFS) Refresh 

func (hfs *HashFS) Refresh(ctx context.Context, execRoot string) error

Refresh refreshes cached file entries under execRoot.

func (*HashFS) Remove 

func (hfs *HashFS) Remove(ctx context.Context, root, fname string) error

Remove removes a file at root/fname.

func (*HashFS) RemoveAll 

func (hfs *HashFS) RemoveAll(ctx context.Context, root, name string) error

RemoveAll removes all files under root/name. Also removes from the disk at the same time.

func (*HashFS) RetrieveUpdateEntries 

func (hfs *HashFS) RetrieveUpdateEntries(ctx context.Context, root string, fnames []string) []UpdateEntry

RetrieveUpdateEntries gets UpdateEntry for fnames at root.

func (*HashFS) RetrieveUpdateEntriesFromLocal 

func (hfs *HashFS) RetrieveUpdateEntriesFromLocal(ctx context.Context, root string, fnames []string) []UpdateEntry

RetrieveUpdateEntriesFromLocal gets UpdateEntry for fnames at root from local disk. It is intended to be used for local execution outputs in cmd.RecordOutputsFromLocal. It won't wait for digest calculation for entries, so UpdateEntry's Entry will be nil. It will forget recorded enties when err (doesn't exist or so).

func (*HashFS) SetExecutables 

func (hfs *HashFS) SetExecutables(m map[string]bool)

SetExecutables sets a map of full paths for files to be considered as executable, even if it is not executable on local disk.

func (*HashFS) SetState 

func (hfs *HashFS) SetState(ctx context.Context, state *pb.State) error

SetState sets states to the HashFS.

func (*HashFS) Stat 

func (hfs *HashFS) Stat(ctx context.Context, root, fname string) (FileInfo, error)

Stat returns a FileInfo at root/fname.

func (*HashFS) State 

func (hfs *HashFS) State(ctx context.Context) *pb.State

State returns a State of the HashFS. 

func (hfs *HashFS) Symlink(ctx context.Context, root, target, linkpath string, mtime time.Time, cmdhash []byte) error

Symlink creates a symlink to target at root/linkpath with mtime and cmdhash.

func (*HashFS) TaintedFiles 

func (hfs *HashFS) TaintedFiles() []string

TaintedFiles returns a list of manually modified generated files.

func (*HashFS) Update 

func (hfs *HashFS) Update(ctx context.Context, execRoot string, entries []UpdateEntry) error

Update updates cache information for entries under execRoot.

func (*HashFS) WaitReady 

func (hfs *HashFS) WaitReady(ctx context.Context) error

WaitReady waits fs state is updated in memory.

func (*HashFS) WriteFile 

func (hfs *HashFS) WriteFile(ctx context.Context, root, fname string, b []byte, isExecutable bool, mtime time.Time, cmdhash []byte) error

WriteFile writes a contents in root/fname with mtime and cmdhash.

type IgnoreFunc 

type IgnoreFunc func(context.Context, string) bool

IgnoreFunc returns true if given fname should be ignored in hashfs.

type NotifyFunc 

type NotifyFunc func(context.Context, *FileInfo)

NotifyFunc is the type of the function to notify the filesystem changes.

type Option 

type Option struct {
	StateFile       string // filename that HashFS saves its state to
	GzipUsesBgzf    bool   // use bgzf for gzip compression
	CompressZstd    bool   // compress fs state using zstd instead of gzip
	CompressLevel   int    // compression level (0 = uncompressed, 1 = fastest, 10 = best)
	CompressThreads int    // number of threads to use for data compression

	KeepTainted bool // keep manually modified generated file

	OSFSOption osfs.Option

	FSMonitor FSMonitor

	DataSource  DataSource
	OutputLocal OutputLocalFunc
	Ignore      IgnoreFunc
	CogFS       *cogutil.Client
	ArtFS       *artfsutil.Client
}

Option is an option for HashFS.

func (*Option) RegisterFlags 

func (o *Option) RegisterFlags(flagSet *flag.FlagSet)

RegisterFlags registers flags for the option.

type OutputLocalFunc 

type OutputLocalFunc func(context.Context, string) bool

OutputLocalFunc returns true if given fname needs to be on local disk.

type UpdateEntry 

type UpdateEntry struct {
	Name string

	// if Entry is nil, use local disk (from RetrieveUpdateEntriesFromLocal), so need to calculate digest from file.
	// If Entry is not nil, use digest in Entry, rather than calculating digest from file.
	Entry *merkletree.Entry

	Mode        fs.FileMode
	ModTime     time.Time
	CmdHash     []byte
	Action      digest.Digest
	UpdatedTime time.Time
	IsChanged   bool

	// IsLocal=true uses Entry, but assumes file exists on local,
	// to avoid unnecessary flush operation.
	IsLocal bool
}

UpdateEntry is an entry for Update.

func (UpdateEntry) String 

func (e UpdateEntry) String() string

Source Files

View all Source files

Directories

Path Synopsis
Package osfs provides OS Filesystem access.
 Package osfs provides OS Filesystem access.
Package proto provides protocol buffer message for hashfs.
 Package proto provides protocol buffer message for hashfs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.