filesystem

package
v1.5.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2022 License: MIT Imports: 20 Imported by: 0

README

Filesystem

The Filesystem is responsible for ensuring that all of its supported file formats can be accessed in a threadsafe manner. It doesn't handle any persistence directly but instead relies on the underlying format's package to handle that itself. This also means that methods like Delete and Rename should not be called on the underlying types directly but be called through the Filesystem's corresponding wrappers instead.

To refer to a file or folder within the Filesystem, so-called SiaPaths are used. They are unix-like paths relative to the specified root of the Filesystem except for the fact that they don't start with a leading slash. Ideally all parts of the siad codebase only have to interact with SiaPaths instead of system paths and the Filesystem would handle all of the translations between SiaPaths and regular system paths. The Filesystem also enforces that files and folders can't share the same name.

The Filesystem is an in-memory tree-like data structure of nodes. The nodes can either be files or directories. Directory nodes potentially point to other directory nodes or file nodes while file nodes can't have any children. When opening a file or folder, the Filesystem is traversed starting from the root until the required file or folder is found, opening all the nodes on demand and adding them to the tree. The tree will be pruned again as nodes are no longer needed. To do so a special locking convention was introduced which will be explained in its own paragraph.

Locking Conventions

Due to the nature of the Filesystem custom locking conventions were required to optimize the performance of the data structure.

  • A locked child node can't grab a parent's lock
  • A locked parent can grab its children's locks
  • A parent needs to be locked before adding/removing children
  • A parent doesn't need to be locked before modifying any children

Locking like this avoids a lot of lock contention and enables us to easily and efficiently delete and rename folders.

Submodules

The Filesystem has several submodules that each perform a specific function for the Renter. This README will provide brief overviews of the submodules, but for more detailed descriptions of the inner workings of the submodules the respective README files should be reviewed.

  • SiaDir
  • SiaFile
SiaDir

The SiaDir module is the code that defines what a directory is on the Sia network. It also manages accesses and updates to the file, ensuring safety and ACIDity when performing file operations.

SiaFile

The SiaFile module is the code that defines what a file is on the Sia network. It also manages accesses and updates to the file, ensuring safety and ACIDity when performing file operations.

Subsystems

The Filesystem has the following subsystems.

Filesystem

Key Files

The Filesystem subsystem contains Filesystem specific errors, the definition for the common node type as well as all the exported methods which are called by other modules.

It is a special DirNode which acts as the root of the Filesystem, is created upon startup and is always kept in memory instead of being pruned like the other nodes. The job of the FileSystem is to translate SiaPaths into regular paths and pass calls to its exported methods on to the correct child if possible. It also implements some high level methods which might require interacting with multiple nodes like RenameDir for example.

DirNode

Key Files

The DirNode extends the common node by fields relevant to a directory. Namely children and an embedded SiaDir. The special thing about the embedded SiaDir is that it is loaded lazily. That means it will only be loaded from disk once a method is called that requires it. The SiaDir will also be removed from memory again once the DirNode is no longer being accessed. That way we avoid caching the SiaDir for too long.

FileNode

Key Files

The FileNode is similar to the DirNode but it only extends the node by a single embedded Siafile field. Apart from that it contains wrappers for the SiaFile methods which correctly modify the parent directory when the underlying file is moved or deleted.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNotExist is returned when a file or folder can't be found on disk.
	ErrNotExist = errors.New("path does not exist")

	// ErrExists is returned when a file or folder already exists at a given
	// location.
	ErrExists = errors.New("a file or folder already exists at the specified path")

	// ErrDeleteFileIsDir is returned when the file delete method is used but
	// the filename corresponds to a directory
	ErrDeleteFileIsDir = errors.New("cannot delete file, file is a directory")
)

Functions

This section is empty.

Types

type DirNode 

type DirNode struct {
	// contains filtered or unexported fields
}

DirNode is a node which references a SiaDir.

func (*DirNode) Close 

func (n *DirNode) Close() error

Close calls close on the DirNode and also removes the dNode from its parent if it's no longer being used and if it doesn't have any children which are currently in use. This happens iteratively for all parent as long as removing a child resulted in them not having any children left.

func (*DirNode) Delete 

func (n *DirNode) Delete() error

Delete is a wrapper for SiaDir.Delete.

func (*DirNode) Deleted 

func (n *DirNode) Deleted() (bool, error)

Deleted is a wrapper for SiaDir.Deleted.

func (*DirNode) Dir 

func (n *DirNode) Dir(name string) (*DirNode, error)

Dir will return a child dir of this directory if it exists.

func (*DirNode) DirReader 

func (n *DirNode) DirReader() (*siadir.DirReader, error)

DirReader is a wrapper for SiaDir.DirReader.

func (*DirNode) File 

func (n *DirNode) File(name string) (*FileNode, error)

File will return a child file of this directory if it exists.

func (*DirNode) Inode 

func (n *DirNode) Inode() uint64

NID returns the node's unique identifier.

func (*DirNode) Metadata 

func (n *DirNode) Metadata() (siadir.Metadata, error)

Metadata is a wrapper for SiaDir.Metadata.

func (*DirNode) Path 

func (n *DirNode) Path() (string, error)

Path is a wrapper for SiaDir.Path.

func (*DirNode) UpdateBubbledMetadata 

func (n *DirNode) UpdateBubbledMetadata(md siadir.Metadata) error

UpdateBubbledMetadata is a wrapper for SiaDir.UpdateBubbledMetadata.

func (*DirNode) UpdateLastHealthCheckTime 

func (n *DirNode) UpdateLastHealthCheckTime(aggregateLastHealthCheckTime, lastHealthCheckTime time.Time) error

UpdateLastHealthCheckTime is a wrapper for SiaDir.UpdateLastHealthCheckTime.

func (*DirNode) UpdateMetadata 

func (n *DirNode) UpdateMetadata(md siadir.Metadata) error

UpdateMetadata is a wrapper for SiaDir.UpdateMetadata.

type FileNode 

type FileNode struct {
	*siafile.SiaFile
	// contains filtered or unexported fields
}

FileNode is a node which references a SiaFile.

func (*FileNode) AddPiece 

func (n *FileNode) AddPiece(pk types.SiaPublicKey, chunkIndex, pieceIndex uint64, merkleRoot crypto.Hash) (err error)

AddPiece wraps siafile.AddPiece to guarantee that it's not called when the fileNode was already closed.

func (*FileNode) Close 

func (n *FileNode) Close() error

Close calls close on the FileNode and also removes the FileNode from its parent if it's no longer being used and if it doesn't have any children which are currently in use. This happens iteratively for all parent as long as removing a child resulted in them not having any children left.

func (*FileNode) Copy 

func (n *FileNode) Copy() *FileNode

Copy copies a file node and returns the copy.

func (*FileNode) Inode 

func (n *FileNode) Inode() uint64

NID returns the node's unique identifier.

type FileSystem 

type FileSystem struct {
	DirNode
}

FileSystem implements a thread-safe filesystem for Sia for loading SiaFiles, SiaDirs and potentially other supported Sia types in the future.

func New 

func New(root string, log *persist.Logger, wal *writeaheadlog.WAL) (*FileSystem, error)

New creates a new FileSystem at the specified root path. The folder will be created if it doesn't exist already.

func (*FileSystem) AddSiaFileFromReader 

func (fs *FileSystem) AddSiaFileFromReader(rs io.ReadSeeker, siaPath modules.SiaPath) (err error)

AddSiaFileFromReader adds an existing SiaFile to the set and stores it on disk. If the exact same file already exists, this is a no-op. If a file already exists with a different UID, the UID will be updated and a unique path will be chosen. If no file exists, the UID will be updated but the path remains the same.

func (*FileSystem) CachedFileInfo 

func (fs *FileSystem) CachedFileInfo(siaPath modules.SiaPath) (modules.FileInfo, error)

CachedFileInfo returns the cached File Information of the siafile

func (*FileSystem) CachedList 

func (fs *FileSystem) CachedList(siaPath modules.SiaPath, recursive bool, flf modules.FileListFunc, dlf modules.DirListFunc) error

CachedList lists the files and directories within a SiaDir.

func (*FileSystem) CachedListOnNode 

func (fs *FileSystem) CachedListOnNode(d *DirNode) (fis []modules.FileInfo, dis []modules.DirectoryInfo, err error)

CachedListOnNode will return the files and directories within a given siadir node in a non-recursive way.

func (*FileSystem) DeleteDir 

func (fs *FileSystem) DeleteDir(siaPath modules.SiaPath) error

DeleteDir deletes a dir from the filesystem. The dir will be marked as 'deleted' which should cause all remaining instances of the dir to be close shortly. Only when all instances of the dir are closed it will be removed from the tree. This means that as long as the deletion is in progress, no new file of the same path can be created and the existing file can't be opened until all instances of it are closed.

func (*FileSystem) DeleteFile 

func (fs *FileSystem) DeleteFile(siaPath modules.SiaPath) error

DeleteFile deletes a file from the filesystem. The file will be marked as 'deleted' which should cause all remaining instances of the file to be closed shortly. Only when all instances of the file are closed it will be removed from the tree. This means that as long as the deletion is in progress, no new file of the same path can be created and the existing file can't be opened until all instances of it are closed.

func (*FileSystem) DirExists 

func (fs *FileSystem) DirExists(siaPath modules.SiaPath) (bool, error)

DirExists checks to see if a dir with the provided siaPath already exists in the renter.

func (*FileSystem) DirInfo 

func (fs *FileSystem) DirInfo(siaPath modules.SiaPath) (_ modules.DirectoryInfo, err error)

DirInfo returns the Directory Information of the siadir

func (*FileSystem) DirNodeInfo 

func (fs *FileSystem) DirNodeInfo(n *DirNode) (modules.DirectoryInfo, error)

DirNodeInfo will return the DirectoryInfo of a siadir given the node. This is more efficient than calling fs.DirInfo.

func (*FileSystem) DirPath 

func (fs *FileSystem) DirPath(siaPath modules.SiaPath) string

DirPath converts a SiaPath into a dir's system path.

func (*FileSystem) DirSiaPath 

func (fs *FileSystem) DirSiaPath(n *DirNode) (sp modules.SiaPath)

DirSiaPath returns the SiaPath of a dir node.

func (*FileSystem) FileExists 

func (fs *FileSystem) FileExists(siaPath modules.SiaPath) (bool, error)

FileExists checks to see if a file with the provided siaPath already exists in the renter.

func (*FileSystem) FileInfo 

func (fs *FileSystem) FileInfo(siaPath modules.SiaPath, offline map[string]bool, goodForRenew map[string]bool, contracts map[string]modules.RenterContract) (modules.FileInfo, error)

FileInfo returns the File Information of the siafile

func (*FileSystem) FileNodeInfo 

func (fs *FileSystem) FileNodeInfo(n *FileNode) (modules.FileInfo, error)

FileNodeInfo returns the FileInfo of a siafile given the node for the siafile. This is faster than calling fs.FileInfo.

func (*FileSystem) FilePath 

func (fs *FileSystem) FilePath(siaPath modules.SiaPath) string

FilePath converts a SiaPath into a file's system path.

func (*FileSystem) FileSiaPath 

func (fs *FileSystem) FileSiaPath(n *FileNode) (sp modules.SiaPath)

FileSiaPath returns the SiaPath of a file node.

func (*FileSystem) Inode 

func (n *FileSystem) Inode() uint64

NID returns the node's unique identifier.

func (*FileSystem) List 

func (fs *FileSystem) List(siaPath modules.SiaPath, recursive bool, offlineMap, goodForRenewMap map[string]bool, contractsMap map[string]modules.RenterContract, flf modules.FileListFunc, dlf modules.DirListFunc) error

List lists the files and directories within a SiaDir.

func (*FileSystem) NewSiaDir 

func (fs *FileSystem) NewSiaDir(siaPath modules.SiaPath, mode os.FileMode) error

NewSiaDir creates the folder for the specified siaPath.

func (*FileSystem) NewSiaFile 

func (fs *FileSystem) NewSiaFile(siaPath modules.SiaPath, source string, ec modules.ErasureCoder, mk crypto.CipherKey, fileSize uint64, fileMode os.FileMode, disablePartialUpload bool) error

NewSiaFile creates a SiaFile at the specified siaPath.

func (*FileSystem) NewSiaFileFromLegacyData 

func (fs *FileSystem) NewSiaFileFromLegacyData(fd siafile.FileData) (_ *FileNode, err error)

NewSiaFileFromLegacyData creates a new SiaFile from data that was previously loaded from a legacy file.

func (*FileSystem) OpenSiaDir 

func (fs *FileSystem) OpenSiaDir(siaPath modules.SiaPath) (*DirNode, error)

OpenSiaDir opens a SiaDir and adds it and all of its parents to the filesystem tree.

func (*FileSystem) OpenSiaDirCustom 

func (fs *FileSystem) OpenSiaDirCustom(siaPath modules.SiaPath, create bool) (*DirNode, error)

OpenSiaDirCustom opens a SiaDir and adds it and all of its parents to the filesystem tree. If create is true it will create the dir if it doesn't exist.

func (*FileSystem) OpenSiaFile 

func (fs *FileSystem) OpenSiaFile(siaPath modules.SiaPath) (*FileNode, error)

OpenSiaFile opens a SiaFile and adds it and all of its parents to the filesystem tree.

func (*FileSystem) ReadDir 

func (fs *FileSystem) ReadDir(siaPath modules.SiaPath) ([]os.FileInfo, error)

ReadDir reads all the fileinfos of the specified dir.

func (*FileSystem) RenameDir 

func (fs *FileSystem) RenameDir(oldSiaPath, newSiaPath modules.SiaPath) error

RenameDir takes an existing directory and changes the path. The original directory must exist, and there must not be any directory that already has the replacement path. All sia files within directory will also be renamed

func (*FileSystem) RenameFile 

func (fs *FileSystem) RenameFile(oldSiaPath, newSiaPath modules.SiaPath) (err error)

RenameFile renames the file with oldSiaPath to newSiaPath.

func (*FileSystem) Root 

func (fs *FileSystem) Root() string

Root returns the root system path of the FileSystem.

func (*FileSystem) Stat 

func (fs *FileSystem) Stat(siaPath modules.SiaPath) (os.FileInfo, error)

Stat is a wrapper for os.Stat which takes a SiaPath as an argument instead of a system path.

func (*FileSystem) UpdateDirMetadata 

func (fs *FileSystem) UpdateDirMetadata(siaPath modules.SiaPath, metadata siadir.Metadata) (err error)

UpdateDirMetadata updates the metadata of a SiaDir.

func (*FileSystem) Walk 

func (fs *FileSystem) Walk(siaPath modules.SiaPath, walkFn filepath.WalkFunc) error

Walk is a wrapper for filepath.Walk which takes a SiaPath as an argument instead of a system path.

func (*FileSystem) WriteFile 

func (fs *FileSystem) WriteFile(siaPath modules.SiaPath, data []byte, perm os.FileMode) error

WriteFile is a wrapper for ioutil.WriteFile which takes a SiaPath as an argument instead of a system path.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.