siadir

README

SiaDir

The SiaDir module is responsible for creating and maintaining the directory metadata information stored in the .siadir files on disk. This includes all disk interaction and metadata definition. These siadirs represent directories on the Sia network.

Structure of the SiaDir

The SiaDir is a dir on the Sia network and the siadir metadata is a JSON formatted metadata file that contains aggregate and non-aggregate fields. The aggregate fields are the totals of the siadir and any sub siadirs, or are calculated based on all the values in the subtree. The non-aggregate fields are information specific to the siadir that is not an aggregate of the entire sub directory tree

Subsystems

The following subsystems help the SiaDir module execute its responsibilities:

• Persistence Subsystem
• File Format Subsystem
• SiaDirSet Subsystem
• DirReader Subsystem

Persistence Subsystem

Key Files

• persist.go
• persistwal.go

The Persistence subsystem is responsible for the disk interaction with the .siadir files and ensuring safe and performant ACID operations by using the writeaheadlog package. There are two WAL updates that are used, deletion and metadata updates.

The WAL deletion update removes all the contents of the directory including the directory itself.

The WAL metadata update re-writes the entire metadata, which is stored as JSON. This is used whenever the metadata changes and needs to be saved as well as when a new siadir is created.

Exports

• ApplyUpdates
• CreateAndApplyTransaction
• IsSiaDirUpdate
• New
• LoadSiaDir
• UpdateMetadata

Inbound Complexities

• callDelete deletes a SiaDir from disk
  • SiaDirSet.Delete uses callDelete
• LoadSiaDir loads a SiaDir from disk
  • SiaDirSet.open uses LoadSiaDir

File Format Subsystem

Key Files

• siadir.go

The file format subsystem contains the type definitions for the SiaDir format and methods that return information about the SiaDir.

Exports

• Deleted
• Metatdata
• SiaPath

SiaDirSet Subsystem

Key Files

• siadirset.go

A SiaDir object is threadsafe by itself, and to ensure that when a SiaDir is accessed by multiple threads that it is still threadsafe, SiaDirs should always be accessed through the SiaDirSet. The SiaDirSet was created as a pool of SiaDirs which is used by other packages to get access to SiaDirEntries which are wrappers for SiaDirs containing some extra information about how many threads are using it at a certain time. If a SiaDir was already loaded the SiaDirSet will hand out the existing object, otherwise it will try to load it from disk.

Exports

• HealthPercentage
• NewSiaDirSet
• Close
• Delete
• DirInfo
• DirList
• Exists
• InitRootDir
• NewSiaDir
• Open
• Rename

Outbound Complexities

• Delete will use callDelete to delete the SiaDir once it has been acquired in the set
• open calls LoadSiaDir to load the SiaDir from disk

DirReader Subsystem

Key Files

• dirreader.go

The DirReader Subsystem creates the DirReader which is used as a helper to read raw .siadir from disk

Exports

• Close
• Read
• Stat
• DirReader

Documentation

Index

• Constants
• func ApplyUpdates(updates ...writeaheadlog.Update) error
• func CreateAndApplyTransaction(wal *writeaheadlog.WAL, updates ...writeaheadlog.Update) error
• func IsSiaDirUpdate(update writeaheadlog.Update) bool
• type DirReader
  • func (sdr *DirReader) Close() error
  • func (sdr *DirReader) Read(b []byte) (int, error)
  • func (sdr *DirReader) Stat() (os.FileInfo, error)
• type Metadata
• type SiaDir
  • func LoadSiaDir(path string, deps modules.Dependencies, wal *writeaheadlog.WAL) (sd *SiaDir, err error)
  • func New(path, rootPath string, mode os.FileMode, wal *writeaheadlog.WAL) (*SiaDir, error)
  • func (sd *SiaDir) Delete() error
  • func (sd *SiaDir) Deleted() bool
  • func (sd *SiaDir) DirReader() (*DirReader, error)
  • func (sd *SiaDir) MDPath() string
  • func (sd *SiaDir) Metadata() Metadata
  • func (sd *SiaDir) Path() string
  • func (sd *SiaDir) Rename(targetPath string) error
  • func (sd *SiaDir) SetPath(targetPath string)
  • func (sd *SiaDir) UpdateBubbledMetadata(metadata Metadata) error
  • func (sd *SiaDir) UpdateLastHealthCheckTime(aggregateLastHealthCheckTime, lastHealthCheckTime time.Time) error
  • func (sd *SiaDir) UpdateMetadata(metadata Metadata) error

Constants

const (
	// SiaDirExtension is the name of the metadata file for the sia directory
	SiaDirExtension = ".siadir"

	// DefaultDirHealth is the default health for the directory and the fall
	// back value when there is an error. This is to protect against falsely
	// trying to repair directories that had a read error
	DefaultDirHealth = float64(0)

	// DefaultDirRedundancy is the default redundancy for the directory and the
	// fall back value when there is an error. This is to protect against
	// falsely trying to repair directories that had a read error
	DefaultDirRedundancy = float64(-1)
)

Functions

func ApplyUpdates

func ApplyUpdates(updates ...writeaheadlog.Update) error

ApplyUpdates applies a number of writeaheadlog updates to the corresponding SiaDir. This method can apply updates from different SiaDirs and should only be run before the SiaDirs are loaded from disk right after the startup of siad. Otherwise we might run into concurrency issues.

func CreateAndApplyTransaction

func CreateAndApplyTransaction(wal *writeaheadlog.WAL, updates ...writeaheadlog.Update) error

CreateAndApplyTransaction is a helper method that creates a writeaheadlog transaction and applies it.

func IsSiaDirUpdate

func IsSiaDirUpdate(update writeaheadlog.Update) bool

IsSiaDirUpdate is a helper method that makes sure that a wal update belongs to the SiaDir package.

Types

type DirReader

type DirReader struct {
	// contains filtered or unexported fields
}

DirReader is a helper type that allows reading a raw .siadir from disk while keeping the file in memory locked.

func (*DirReader) Close

func (sdr *DirReader) Close() error

Close closes the underlying file.

func (*DirReader) Read

func (sdr *DirReader) Read(b []byte) (int, error)

Read calls Read on the underlying file.

func (*DirReader) Stat

func (sdr *DirReader) Stat() (os.FileInfo, error)

Stat returns the FileInfo of the underlying file.

type Metadata

type Metadata struct {

	// The following fields are aggregate values of the siadir. These values are
	// the totals of the siadir and any sub siadirs, or are calculated based on
	// all the values in the subtree
	AggregateHealth              float64   `json:"aggregatehealth"`
	AggregateLastHealthCheckTime time.Time `json:"aggregatelasthealthchecktime"`
	AggregateMinRedundancy       float64   `json:"aggregateminredundancy"`
	AggregateModTime             time.Time `json:"aggregatemodtime"`
	AggregateNumFiles            uint64    `json:"aggregatenumfiles"`
	AggregateNumStuckChunks      uint64    `json:"aggregatenumstuckchunks"`
	AggregateNumSubDirs          uint64    `json:"aggregatenumsubdirs"`
	AggregateRemoteHealth        float64   `json:"aggregateremotehealth"`
	AggregateSize                uint64    `json:"aggregatesize"`
	AggregateStuckHealth         float64   `json:"aggregatestuckhealth"`

	// The following fields are information specific to the siadir that is not
	// an aggregate of the entire sub directory tree
	Health              float64     `json:"health"`
	LastHealthCheckTime time.Time   `json:"lasthealthchecktime"`
	MinRedundancy       float64     `json:"minredundancy"`
	Mode                os.FileMode `json:"mode"`
	ModTime             time.Time   `json:"modtime"`
	NumFiles            uint64      `json:"numfiles"`
	NumStuckChunks      uint64      `json:"numstuckchunks"`
	NumSubDirs          uint64      `json:"numsubdirs"`
	RemoteHealth        float64     `json:"remotehealth"`
	Size                uint64      `json:"size"`
	StuckHealth         float64     `json:"stuckhealth"`

	// Version is the used version of the header file.
	Version string `json:"version"`
}

Metadata is the metadata that is saved to disk as a .siadir file

type SiaDir

type SiaDir struct {
	// contains filtered or unexported fields
}

SiaDir contains the metadata information about a renter directory

func LoadSiaDir

func LoadSiaDir(path string, deps modules.Dependencies, wal *writeaheadlog.WAL) (sd *SiaDir, err error)

LoadSiaDir loads the directory metadata from disk

func New

func New(path, rootPath string, mode os.FileMode, wal *writeaheadlog.WAL) (*SiaDir, error)

New creates a new directory in the renter directory and makes sure there is a metadata file in the directory and creates one as needed. This method will also make sure that all the parent directories are created and have metadata files as well and will return the SiaDir containing the information for the directory that matches the siaPath provided

func (*SiaDir) Delete

func (sd *SiaDir) Delete() error

Delete removes the directory from disk and marks it as deleted. Once the directory is deleted, attempting to access the directory will return an error.

func (*SiaDir) Deleted

func (sd *SiaDir) Deleted() bool

Deleted returns the deleted field of the siaDir

func (*SiaDir) DirReader

func (sd *SiaDir) DirReader() (*DirReader, error)

DirReader creates a io.ReadCloser that can be used to read the raw SiaDir from disk.

func (*SiaDir) MDPath

func (sd *SiaDir) MDPath() string

MDPath returns the path of the SiaDir's metadata on disk.

func (*SiaDir) Metadata

func (sd *SiaDir) Metadata() Metadata

Metadata returns the metadata of the SiaDir

func (*SiaDir) Path

func (sd *SiaDir) Path() string

Path returns the path of the SiaDir on disk.

func (*SiaDir) Rename

func (sd *SiaDir) Rename(targetPath string) error

Rename renames the SiaDir to targetPath.

func (*SiaDir) SetPath

func (sd *SiaDir) SetPath(targetPath string)

SetPath sets the path field of the dir.

func (*SiaDir) UpdateBubbledMetadata

func (sd *SiaDir) UpdateBubbledMetadata(metadata Metadata) error

UpdateBubbledMetadata updates the SiaDir Metadata that is bubbled and saves the changes to disk. For fields that are not bubbled, this method sets them to the current values in the SiaDir metadata

func (*SiaDir) UpdateLastHealthCheckTime

func (sd *SiaDir) UpdateLastHealthCheckTime(aggregateLastHealthCheckTime, lastHealthCheckTime time.Time) error

UpdateLastHealthCheckTime updates the SiaDir LastHealthCheckTime and AggregateLastHealthCheckTime and saves the changes to disk

func (*SiaDir) UpdateMetadata

func (sd *SiaDir) UpdateMetadata(metadata Metadata) error

UpdateMetadata updates the SiaDir metadata on disk