README ¶
Filesystem
The Filesystem is responsible for ensuring that all of its supported file
formats can be accessed in a threadsafe manner. It doesn't handle any
persistence directly but instead relies on the underlying format's package to
handle that itself. This also means that methods like Delete
and Rename
should not be called on the underlying types directly but be called through
the Filesystem's corresponding wrappers instead.
To refer to a file or folder within the Filesystem, so-called SiaPaths are used. They are unix-like paths relative to the specified root of the Filesystem except for the fact that they don't start with a leading slash. Ideally all parts of the siad codebase only have to interact with SiaPaths instead of system paths and the Filesystem would handle all of the translations between SiaPaths and regular system paths. The Filesystem also enforces that files and folders can't share the same name.
The Filesystem is an in-memory tree-like data structure of nodes. The nodes can either be files or directories. Directory nodes potentially point to other directory nodes or file nodes while file nodes can't have any children. When opening a file or folder, the Filesystem is traversed starting from the root until the required file or folder is found, opening all the nodes on demand and adding them to the tree. The tree will be pruned again as nodes are no longer needed. To do so a special locking convention was introduced which will be explained in its own paragraph.
Locking Conventions
Due to the nature of the Filesystem custom locking conventions were required to optimize the performance of the data structure.
- A locked child node can't grab a parent's lock
- A locked parent can grab its children's locks
- A parent needs to be locked before adding/removing children
- A parent doesn't need to be locked before modifying any children
Locking like this avoids a lot of lock contention and enables us to easily and efficiently delete and rename folders.
Submodules
The Filesystem has several submodules that each perform a specific function for the Renter. This README will provide brief overviews of the submodules, but for more detailed descriptions of the inner workings of the submodules the respective README files should be reviewed.
- SiaDir
- SiaFile
SiaDir
The SiaDir module is the code that defines what a directory is on the Sia network. It also manages accesses and updates to the file, ensuring safety and ACIDity when performing file operations.
SiaFile
The SiaFile module is the code that defines what a file is on the Sia network. It also manages accesses and updates to the file, ensuring safety and ACIDity when performing file operations.
Subsystems
The Filesystem has the following subsystems.
Filesystem
Key Files
The Filesystem subsystem contains Filesystem specific errors, the definition
for the common node
type as well as all the exported methods which are
called by other modules.
It is a special DirNode
which acts as the root of the Filesystem, is
created upon startup and is always kept in memory instead of being pruned
like the other nodes. The job of the FileSystem is to translate SiaPaths into
regular paths and pass calls to its exported methods on to the correct child
if possible. It also implements some high level methods which might require
interacting with multiple nodes like RenameDir
for example.
DirNode
Key Files
The DirNode extends the common node
by fields relevant to a directory.
Namely children and an embedded SiaDir
. The special thing about the
embedded SiaDir
is that it is loaded lazily. That means it will only be
loaded from disk once a method is called that requires it. The SiaDir
will
also be removed from memory again once the DirNode is no longer being
accessed. That way we avoid caching the SiaDir
for too long.
FileNode
Key Files
The FileNode is similar to the DirNode but it only extends the node
by a
single embedded Siafile
field. Apart from that it contains wrappers for the
SiaFile
methods which correctly modify the parent directory when the
underlying file is moved or deleted.
Documentation ¶
Index ¶
- Variables
- type DirNode
- func (n *DirNode) Close() error
- func (n *DirNode) Delete() error
- func (n *DirNode) Deleted() (bool, error)
- func (n *DirNode) Dir(name string) (*DirNode, error)
- func (n *DirNode) DirReader() (*siadir.DirReader, error)
- func (n *DirNode) File(name string) (*FileNode, error)
- func (n *DirNode) Inode() uint64
- func (n *DirNode) Metadata() (siadir.Metadata, error)
- func (n *DirNode) Path() (string, error)
- func (n *DirNode) UpdateBubbledMetadata(md siadir.Metadata) error
- func (n *DirNode) UpdateLastHealthCheckTime(aggregateLastHealthCheckTime, lastHealthCheckTime time.Time) error
- func (n *DirNode) UpdateMetadata(md siadir.Metadata) error
- type FileNode
- type FileSystem
- func (fs *FileSystem) AddSiaFileFromReader(rs io.ReadSeeker, siaPath modules.SiaPath) (err error)
- func (fs *FileSystem) CachedFileInfo(siaPath modules.SiaPath) (modules.FileInfo, error)
- func (fs *FileSystem) CachedList(siaPath modules.SiaPath, recursive bool, flf modules.FileListFunc, ...) error
- func (fs *FileSystem) CachedListOnNode(d *DirNode) (fis []modules.FileInfo, dis []modules.DirectoryInfo, err error)
- func (fs *FileSystem) DeleteDir(siaPath modules.SiaPath) error
- func (fs *FileSystem) DeleteFile(siaPath modules.SiaPath) error
- func (fs *FileSystem) DirExists(siaPath modules.SiaPath) (bool, error)
- func (fs *FileSystem) DirInfo(siaPath modules.SiaPath) (_ modules.DirectoryInfo, err error)
- func (fs *FileSystem) DirNodeInfo(n *DirNode) (modules.DirectoryInfo, error)
- func (fs *FileSystem) DirPath(siaPath modules.SiaPath) string
- func (fs *FileSystem) DirSiaPath(n *DirNode) (sp modules.SiaPath)
- func (fs *FileSystem) FileExists(siaPath modules.SiaPath) (bool, error)
- func (fs *FileSystem) FileInfo(siaPath modules.SiaPath, offline map[string]bool, goodForRenew map[string]bool, ...) (modules.FileInfo, error)
- func (fs *FileSystem) FileNodeInfo(n *FileNode) (modules.FileInfo, error)
- func (fs *FileSystem) FilePath(siaPath modules.SiaPath) string
- func (fs *FileSystem) FileSiaPath(n *FileNode) (sp modules.SiaPath)
- func (n *FileSystem) Inode() uint64
- func (fs *FileSystem) List(siaPath modules.SiaPath, recursive bool, ...) error
- func (fs *FileSystem) NewSiaDir(siaPath modules.SiaPath, mode os.FileMode) error
- func (fs *FileSystem) NewSiaFile(siaPath modules.SiaPath, source string, ec modules.ErasureCoder, ...) error
- func (fs *FileSystem) NewSiaFileFromLegacyData(fd siafile.FileData) (_ *FileNode, err error)
- func (fs *FileSystem) OpenSiaDir(siaPath modules.SiaPath) (*DirNode, error)
- func (fs *FileSystem) OpenSiaDirCustom(siaPath modules.SiaPath, create bool) (*DirNode, error)
- func (fs *FileSystem) OpenSiaFile(siaPath modules.SiaPath) (*FileNode, error)
- func (fs *FileSystem) ReadDir(siaPath modules.SiaPath) ([]os.FileInfo, error)
- func (fs *FileSystem) RenameDir(oldSiaPath, newSiaPath modules.SiaPath) error
- func (fs *FileSystem) RenameFile(oldSiaPath, newSiaPath modules.SiaPath) (err error)
- func (fs *FileSystem) Root() string
- func (fs *FileSystem) Stat(siaPath modules.SiaPath) (os.FileInfo, error)
- func (fs *FileSystem) UpdateDirMetadata(siaPath modules.SiaPath, metadata siadir.Metadata) (err error)
- func (fs *FileSystem) Walk(siaPath modules.SiaPath, walkFn filepath.WalkFunc) error
- func (fs *FileSystem) WriteFile(siaPath modules.SiaPath, data []byte, perm os.FileMode) error
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNotExist is returned when a file or folder can't be found on disk. ErrNotExist = errors.New("path does not exist") // ErrExists is returned when a file or folder already exists at a given // location. ErrExists = errors.New("a file or folder already exists at the specified path") // ErrDeleteFileIsDir is returned when the file delete method is used but // the filename corresponds to a directory ErrDeleteFileIsDir = errors.New("cannot delete file, file is a directory") )
Functions ¶
This section is empty.
Types ¶
type DirNode ¶
type DirNode struct {
// contains filtered or unexported fields
}
DirNode is a node which references a SiaDir.
func (*DirNode) Close ¶
Close calls close on the DirNode and also removes the dNode from its parent if it's no longer being used and if it doesn't have any children which are currently in use. This happens iteratively for all parent as long as removing a child resulted in them not having any children left.
func (*DirNode) UpdateBubbledMetadata ¶ added in v1.4.8
UpdateBubbledMetadata is a wrapper for SiaDir.UpdateBubbledMetadata.
type FileNode ¶
FileNode is a node which references a SiaFile.
func (*FileNode) AddPiece ¶ added in v1.5.5
func (n *FileNode) AddPiece(pk types.SiaPublicKey, chunkIndex, pieceIndex uint64, merkleRoot crypto.Hash) (err error)
AddPiece wraps siafile.AddPiece to guarantee that it's not called when the fileNode was already closed.
func (*FileNode) Close ¶
Close calls close on the FileNode and also removes the FileNode from its parent if it's no longer being used and if it doesn't have any children which are currently in use. This happens iteratively for all parent as long as removing a child resulted in them not having any children left.
type FileSystem ¶
type FileSystem struct {
DirNode
}
FileSystem implements a thread-safe filesystem for Sia for loading SiaFiles, SiaDirs and potentially other supported Sia types in the future.
func New ¶
func New(root string, log *persist.Logger, wal *writeaheadlog.WAL) (*FileSystem, error)
New creates a new FileSystem at the specified root path. The folder will be created if it doesn't exist already.
func (*FileSystem) AddSiaFileFromReader ¶
func (fs *FileSystem) AddSiaFileFromReader(rs io.ReadSeeker, siaPath modules.SiaPath) (err error)
AddSiaFileFromReader adds an existing SiaFile to the set and stores it on disk. If the exact same file already exists, this is a no-op. If a file already exists with a different UID, the UID will be updated and a unique path will be chosen. If no file exists, the UID will be updated but the path remains the same.
func (*FileSystem) CachedFileInfo ¶
CachedFileInfo returns the cached File Information of the siafile
func (*FileSystem) CachedList ¶
func (fs *FileSystem) CachedList(siaPath modules.SiaPath, recursive bool, flf modules.FileListFunc, dlf modules.DirListFunc) error
CachedList lists the files and directories within a SiaDir.
func (*FileSystem) CachedListOnNode ¶
func (fs *FileSystem) CachedListOnNode(d *DirNode) (fis []modules.FileInfo, dis []modules.DirectoryInfo, err error)
CachedListOnNode will return the files and directories within a given siadir node in a non-recursive way.
func (*FileSystem) DeleteDir ¶
func (fs *FileSystem) DeleteDir(siaPath modules.SiaPath) error
DeleteDir deletes a dir from the filesystem. The dir will be marked as 'deleted' which should cause all remaining instances of the dir to be close shortly. Only when all instances of the dir are closed it will be removed from the tree. This means that as long as the deletion is in progress, no new file of the same path can be created and the existing file can't be opened until all instances of it are closed.
func (*FileSystem) DeleteFile ¶
func (fs *FileSystem) DeleteFile(siaPath modules.SiaPath) error
DeleteFile deletes a file from the filesystem. The file will be marked as 'deleted' which should cause all remaining instances of the file to be closed shortly. Only when all instances of the file are closed it will be removed from the tree. This means that as long as the deletion is in progress, no new file of the same path can be created and the existing file can't be opened until all instances of it are closed.
func (*FileSystem) DirExists ¶ added in v1.4.11
func (fs *FileSystem) DirExists(siaPath modules.SiaPath) (bool, error)
DirExists checks to see if a dir with the provided siaPath already exists in the renter.
func (*FileSystem) DirInfo ¶
func (fs *FileSystem) DirInfo(siaPath modules.SiaPath) (_ modules.DirectoryInfo, err error)
DirInfo returns the Directory Information of the siadir
func (*FileSystem) DirNodeInfo ¶
func (fs *FileSystem) DirNodeInfo(n *DirNode) (modules.DirectoryInfo, error)
DirNodeInfo will return the DirectoryInfo of a siadir given the node. This is more efficient than calling fs.DirInfo.
func (*FileSystem) DirPath ¶
func (fs *FileSystem) DirPath(siaPath modules.SiaPath) string
DirPath converts a SiaPath into a dir's system path.
func (*FileSystem) DirSiaPath ¶
func (fs *FileSystem) DirSiaPath(n *DirNode) (sp modules.SiaPath)
DirSiaPath returns the SiaPath of a dir node.
func (*FileSystem) FileExists ¶
func (fs *FileSystem) FileExists(siaPath modules.SiaPath) (bool, error)
FileExists checks to see if a file with the provided siaPath already exists in the renter.
func (*FileSystem) FileInfo ¶
func (fs *FileSystem) FileInfo(siaPath modules.SiaPath, offline map[string]bool, goodForRenew map[string]bool, contracts map[string]modules.RenterContract) (modules.FileInfo, error)
FileInfo returns the File Information of the siafile
func (*FileSystem) FileNodeInfo ¶
func (fs *FileSystem) FileNodeInfo(n *FileNode) (modules.FileInfo, error)
FileNodeInfo returns the FileInfo of a siafile given the node for the siafile. This is faster than calling fs.FileInfo.
func (*FileSystem) FilePath ¶
func (fs *FileSystem) FilePath(siaPath modules.SiaPath) string
FilePath converts a SiaPath into a file's system path.
func (*FileSystem) FileSiaPath ¶
func (fs *FileSystem) FileSiaPath(n *FileNode) (sp modules.SiaPath)
FileSiaPath returns the SiaPath of a file node.
func (*FileSystem) Inode ¶
func (n *FileSystem) Inode() uint64
NID returns the node's unique identifier.
func (*FileSystem) List ¶
func (fs *FileSystem) List(siaPath modules.SiaPath, recursive bool, offlineMap, goodForRenewMap map[string]bool, contractsMap map[string]modules.RenterContract, flf modules.FileListFunc, dlf modules.DirListFunc) error
List lists the files and directories within a SiaDir.
func (*FileSystem) NewSiaFile ¶
func (fs *FileSystem) NewSiaFile(siaPath modules.SiaPath, source string, ec modules.ErasureCoder, mk crypto.CipherKey, fileSize uint64, fileMode os.FileMode, disablePartialUpload bool) error
NewSiaFile creates a SiaFile at the specified siaPath.
func (*FileSystem) NewSiaFileFromLegacyData ¶
func (fs *FileSystem) NewSiaFileFromLegacyData(fd siafile.FileData) (_ *FileNode, err error)
NewSiaFileFromLegacyData creates a new SiaFile from data that was previously loaded from a legacy file.
func (*FileSystem) OpenSiaDir ¶
func (fs *FileSystem) OpenSiaDir(siaPath modules.SiaPath) (*DirNode, error)
OpenSiaDir opens a SiaDir and adds it and all of its parents to the filesystem tree.
func (*FileSystem) OpenSiaDirCustom ¶ added in v1.5.1
OpenSiaDirCustom opens a SiaDir and adds it and all of its parents to the filesystem tree. If create is true it will create the dir if it doesn't exist.
func (*FileSystem) OpenSiaFile ¶
func (fs *FileSystem) OpenSiaFile(siaPath modules.SiaPath) (*FileNode, error)
OpenSiaFile opens a SiaFile and adds it and all of its parents to the filesystem tree.
func (*FileSystem) RenameDir ¶
func (fs *FileSystem) RenameDir(oldSiaPath, newSiaPath modules.SiaPath) error
RenameDir takes an existing directory and changes the path. The original directory must exist, and there must not be any directory that already has the replacement path. All sia files within directory will also be renamed
func (*FileSystem) RenameFile ¶
func (fs *FileSystem) RenameFile(oldSiaPath, newSiaPath modules.SiaPath) (err error)
RenameFile renames the file with oldSiaPath to newSiaPath.
func (*FileSystem) Root ¶
func (fs *FileSystem) Root() string
Root returns the root system path of the FileSystem.
func (*FileSystem) Stat ¶
Stat is a wrapper for os.Stat which takes a SiaPath as an argument instead of a system path.
func (*FileSystem) UpdateDirMetadata ¶
func (fs *FileSystem) UpdateDirMetadata(siaPath modules.SiaPath, metadata siadir.Metadata) (err error)
UpdateDirMetadata updates the metadata of a SiaDir.