filesystem

package
v0.0.0-...-c98baff Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 30, 2020 License: MIT Imports: 18 Imported by: 0

README

Filesystem

The Filesystem is responsible for ensuring that all of its supported file formats can be accessed in a threadsafe manner. It doesn't handle any persistence directly but instead relies on the underlying format's package to handle that itself. This also means that methods like Delete and Rename should not be called on the underlying types directly but be called through the Filesystem's corresponding wrappers instead.

To refer to a file or folder within the Filesystem, so-called I3vPaths are used. They are unix-like paths relative to the specified root of the Filesystem except for the fact that they don't start with a leading slash. Ideally all parts of the i3vd codebase only have to interact with I3vPaths instead of system paths and the Filesystem would handle all of the translations between I3vPaths and regular system paths. The Filesystem also enforces that files and folders can't share the same name.

The Filesystem is an in-memory tree-like data structure of nodes. The nodes can either be files or directories. Directory nodes potentially point to other directory nodes or file nodes while file nodes can't have any children. When opening a file or folder, the Filesystem is traversed starting from the root until the required file or folder is found, opening all the nodes on demand and adding them to the tree. The tree will be pruned again as nodes are no longer needed. To do so a special locking convention was introduced which will be explained in its own paragraph.

Locking Conventions

Due to the nature of the Filesystem custom locking conventions were required to optimize the performance of the data structure.

  • A locked child node can't grab a parent's lock
  • A locked parent can grab its children's locks
  • A parent needs to be locked before adding/removing children
  • A parent doesn't need to be locked before modifying any children

Locking like this avoids a lot of lock contention and enables us to easily and efficiently delete and rename folders.

Subsystems

The Filesystem has the following subsystems.

Filesystem

Key Files

The Filesystem subsystem contains Filesystem specific errors, the definition for the common node type as well as all the exported methods which are called by other modules.

It is a special DirNode which acts as the root of the Filesystem, is created upon startup and is always kept in memory instead of being pruned like the other nodes. The job of the FileSystem is to translate I3vPaths into regular paths and pass calls to its exported methods on to the correct child if possible. It also implements some high level methods which might require interacting with multiple nodes like RenameDir for example.

DirNode

Key Files

The DirNode extends the common node by fields relevant to a directory. Namely children and an embedded I3vDir. The special thing about the embedded I3vDir is that it is loaded lazily. That means it will only be loaded from disk once a method is called that requires it. The I3vDir will also be removed from memory again once the DirNode is no longer being accessed. That way we avoid caching the I3vDir for too long.

FileNode

Key Files

The FileNode is similar to the DirNode but it only extends the node by a single embedded I3vfile field. Apart from that it contains wrappers for the I3vFile methods which correctly modify the parent directory when the underlying file is moved or deleted.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNotExist is returned when a file or folder can't be found on disk.
	ErrNotExist = errors.New("path does not exist")

	// ErrExists is returned when a file or folder already exists at a given
	// location.
	ErrExists = errors.New("a file or folder already exists at the specified path")

	// ErrDeleteFileIsDir is returned when the file delete method is used but
	// the filename corresponds to a directory
	ErrDeleteFileIsDir = errors.New("cannot delete file, file is a directory")
)

Functions

This section is empty.

Types

type DirNode 

type DirNode struct {
	// contains filtered or unexported fields
}

DirNode is a node which references a I3vDir.

func (*DirNode) Close 

func (n *DirNode) Close() error

Close calls close on the DirNode and also removes the dNode from its parent if it's no longer being used and if it doesn't have any children which are currently in use. This happens iteratively for all parent as long as removing a child resulted in them not having any children left.

func (*DirNode) Delete 

func (n *DirNode) Delete() error

Delete is a wrapper for I3vDir.Delete.

func (*DirNode) Dir 

func (n *DirNode) Dir(name string) (*DirNode, error)

Dir will return a child dir of this directory if it exists.

func (*DirNode) DirReader 

func (n *DirNode) DirReader() (*i3vdir.DirReader, error)

DirReader is a wrapper for I3vDir.DirReader.

func (*DirNode) File 

func (n *DirNode) File(name string) (*FileNode, error)

File will return a child file of this directory if it exists.

func (*DirNode) Inode 

func (n *DirNode) Inode() uint64

NID returns the node's unique identifier.

func (*DirNode) Metadata 

func (n *DirNode) Metadata() (i3vdir.Metadata, error)

Metadata is a wrapper for I3vDir.Metadata.

func (*DirNode) Path 

func (n *DirNode) Path() (string, error)

Path is a wrapper for I3vDir.Path.

func (*DirNode) UpdateMetadata 

func (n *DirNode) UpdateMetadata(md i3vdir.Metadata) error

UpdateMetadata is a wrapper for I3vDir.UpdateMetadata.

type FileNode 

type FileNode struct {
	*i3vfile.I3vFile
	// contains filtered or unexported fields
}

FileNode is a node which references a I3vFile.

func (*FileNode) Close 

func (n *FileNode) Close() error

Close calls close on the FileNode and also removes the FileNode from its parent if it's no longer being used and if it doesn't have any children which are currently in use. This happens iteratively for all parent as long as removing a child resulted in them not having any children left.

func (*FileNode) Copy 

func (n *FileNode) Copy() *FileNode

Copy copies a file node and returns the copy.

func (*FileNode) Inode 

func (n *FileNode) Inode() uint64

NID returns the node's unique identifier.

type FileSystem 

type FileSystem struct {
	DirNode
}

FileSystem implements a thread-safe filesystem for I3v for loading I3vFiles, I3vDirs and potentially other supported I3v types in the future.

func New 

func New(root string, log *persist.Logger, wal *writeaheadlog.WAL) (*FileSystem, error)

New creates a new FileSystem at the specified root path. The folder will be created if it doesn't exist already.

func (*FileSystem) AddI3vFileFromReader 

func (fs *FileSystem) AddI3vFileFromReader(rs io.ReadSeeker, i3vPath modules.I3vPath) error

AddI3vFileFromReader adds an existing I3vFile to the set and stores it on disk. If the exact same file already exists, this is a no-op. If a file already exists with a different UID, the UID will be updated and a unique path will be chosen. If no file exists, the UID will be updated but the path remains the same.

func (*FileSystem) CachedFileInfo 

func (fs *FileSystem) CachedFileInfo(i3vPath modules.I3vPath) (modules.FileInfo, error)

CachedFileInfo returns the cached File Information of the i3vfile

func (*FileSystem) CachedList 

func (fs *FileSystem) CachedList(i3vPath modules.I3vPath, recursive bool) ([]modules.FileInfo, []modules.DirectoryInfo, error)

CachedList lists the files and directories within a I3vDir.

func (*FileSystem) CachedListOnNode 

func (fs *FileSystem) CachedListOnNode(d *DirNode, recursive bool) ([]modules.FileInfo, []modules.DirectoryInfo, error)

CachedListOnNode will return the files and directories within a given i3vdir node.

func (*FileSystem) DeleteDir 

func (fs *FileSystem) DeleteDir(i3vPath modules.I3vPath) error

DeleteDir deletes a dir from the filesystem. The dir will be marked as 'deleted' which should cause all remaining instances of the dir to be close shortly. Only when all instances of the dir are closed it will be removed from the tree. This means that as long as the deletion is in progress, no new file of the same path can be created and the existing file can't be opened until all instances of it are closed.

func (*FileSystem) DeleteFile 

func (fs *FileSystem) DeleteFile(i3vPath modules.I3vPath) error

DeleteFile deletes a file from the filesystem. The file will be marked as 'deleted' which should cause all remaining instances of the file to be closed shortly. Only when all instances of the file are closed it will be removed from the tree. This means that as long as the deletion is in progress, no new file of the same path can be created and the existing file can't be opened until all instances of it are closed.

func (*FileSystem) DirI3vPath 

func (fs *FileSystem) DirI3vPath(n *DirNode) (sp modules.I3vPath)

DirI3vPath returns the I3vPath of a dir node.

func (*FileSystem) DirInfo 

func (fs *FileSystem) DirInfo(i3vPath modules.I3vPath) (modules.DirectoryInfo, error)

DirInfo returns the Directory Information of the i3vdir

func (*FileSystem) DirNodeInfo 

func (fs *FileSystem) DirNodeInfo(n *DirNode) (modules.DirectoryInfo, error)

DirNodeInfo will return the DirectoryInfo of a i3vdir given the node. This is more efficient than calling fs.DirInfo.

func (*FileSystem) DirPath 

func (fs *FileSystem) DirPath(i3vPath modules.I3vPath) string

DirPath converts a I3vPath into a dir's system path.

func (*FileSystem) FileExists 

func (fs *FileSystem) FileExists(i3vPath modules.I3vPath) (bool, error)

FileExists checks to see if a file with the provided i3vPath already exists in the renter.

func (*FileSystem) FileI3vPath 

func (fs *FileSystem) FileI3vPath(n *FileNode) (sp modules.I3vPath)

FileI3vPath returns the I3vPath of a file node.

func (*FileSystem) FileInfo 

func (fs *FileSystem) FileInfo(i3vPath modules.I3vPath, offline map[string]bool, goodForRenew map[string]bool, contracts map[string]modules.RenterContract) (modules.FileInfo, error)

FileInfo returns the File Information of the i3vfile

func (*FileSystem) FileNodeInfo 

func (fs *FileSystem) FileNodeInfo(n *FileNode) (modules.FileInfo, error)

FileNodeInfo returns the FileInfo of a i3vfile given the node for the i3vfile. This is faster than calling fs.FileInfo.

func (*FileSystem) FilePath 

func (fs *FileSystem) FilePath(i3vPath modules.I3vPath) string

FilePath converts a I3vPath into a file's system path.

func (*FileSystem) Inode 

func (n *FileSystem) Inode() uint64

NID returns the node's unique identifier.

func (*FileSystem) List 

func (fs *FileSystem) List(i3vPath modules.I3vPath, recursive bool, offlineMap, goodForRenewMap map[string]bool, contractsMap map[string]modules.RenterContract) ([]modules.FileInfo, []modules.DirectoryInfo, error)

List lists the files and directories within a I3vDir.

func (*FileSystem) NewI3vDir 

func (fs *FileSystem) NewI3vDir(i3vPath modules.I3vPath, mode os.FileMode) error

NewI3vDir creates the folder for the specified i3vPath.

func (*FileSystem) NewI3vFile 

func (fs *FileSystem) NewI3vFile(i3vPath modules.I3vPath, source string, ec modules.ErasureCoder, mk crypto.CipherKey, fileSize uint64, fileMode os.FileMode, disablePartialUpload bool) error

NewI3vFile creates a I3vFile at the specified i3vPath.

func (*FileSystem) NewI3vFileFromLegacyData 

func (fs *FileSystem) NewI3vFileFromLegacyData(fd i3vfile.FileData) (*FileNode, error)

NewI3vFileFromLegacyData creates a new I3vFile from data that was previously loaded from a legacy file.

func (*FileSystem) OpenI3vDir 

func (fs *FileSystem) OpenI3vDir(i3vPath modules.I3vPath) (*DirNode, error)

OpenI3vDir opens a I3vDir and adds it and all of its parents to the filesystem tree.

func (*FileSystem) OpenI3vFile 

func (fs *FileSystem) OpenI3vFile(i3vPath modules.I3vPath) (*FileNode, error)

OpenI3vFile opens a I3vFile and adds it and all of its parents to the filesystem tree.

func (*FileSystem) ReadDir 

func (fs *FileSystem) ReadDir(i3vPath modules.I3vPath) ([]os.FileInfo, error)

ReadDir is a wrapper of ioutil.ReadDir which takes a I3vPath as an argument instead of a system path.

func (*FileSystem) RenameDir 

func (fs *FileSystem) RenameDir(oldI3vPath, newI3vPath modules.I3vPath) error

RenameDir takes an existing directory and changes the path. The original directory must exist, and there must not be any directory that already has the replacement path. All i3v files within directory will also be renamed

func (*FileSystem) RenameFile 

func (fs *FileSystem) RenameFile(oldI3vPath, newI3vPath modules.I3vPath) error

RenameFile renames the file with oldI3vPath to newI3vPath.

func (*FileSystem) Root 

func (fs *FileSystem) Root() string

Root returns the root system path of the FileSystem.

func (*FileSystem) Stat 

func (fs *FileSystem) Stat(i3vPath modules.I3vPath) (os.FileInfo, error)

Stat is a wrapper for os.Stat which takes a I3vPath as an argument instead of a system path.

func (*FileSystem) UpdateDirMetadata 

func (fs *FileSystem) UpdateDirMetadata(i3vPath modules.I3vPath, metadata i3vdir.Metadata) error

UpdateDirMetadata updates the metadata of a I3vDir.

func (*FileSystem) Walk 

func (fs *FileSystem) Walk(i3vPath modules.I3vPath, walkFn filepath.WalkFunc) error

Walk is a wrapper for filepath.Walk which takes a I3vPath as an argument instead of a system path.

func (*FileSystem) WriteFile 

func (fs *FileSystem) WriteFile(i3vPath modules.I3vPath, data []byte, perm os.FileMode) error

WriteFile is a wrapper for ioutil.WriteFile which takes a I3vPath as an argument instead of a system path.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.