storage

package
v1.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 11, 2024 License: MIT Imports: 18 Imported by: 1

README

Storage Manager

A Storage Manager is responsible for storing and retrieving sectors uploaded to the host. Sectors can now come from two different sources: contracts and temporary storage.

Responsibilities

  • Managing stored sectors
  • Reporting storage inconsistencies
  • Deleting sectors that are no longer referenced

Storage

The RHP packages interact directly with a storage manager to write and read sectors from persistent storage.

Temp Storage

Currently the storage manager also manages temporary storage, but that may not always be the case.

Volume Manager

In hostd the Volume Manager is the default implementation of a storage manager. It is responsible for managing the storage of sectors on disk. It stores metadata in a SQLite database and sector data in a flat file.

Responsibilities

  • Managing volume metadata
  • Managing sector metadata
  • Managing volume data
  • Managing temporary storage
  • Reporting disk errors
  • Migrating volume data

Requirements

  • Stored sectors must stay consistent between storage manager, contract manager, and temp storage
  • Metadata must be ACID between contract revisions in RHP2 and program finalization in RHP3.
  • Support for setting volumes to read-only
  • Support for checking on-disk consistency and reporting errors
  • Support for resizing and removing volumes
  • During resize, sectors must be able to be moved within the same volume
  • Avoid locking a folder during resize or remove operations; sectors not being actively moved should be readable.
  • Support for multiple volumes with dynamic sizes
  • Provide alerts and status information on long-running operations

Documentation

Index

Constants

View Source 
const (
	VolumeStatusUnavailable = "unavailable"
	VolumeStatusCreating    = "creating"
	VolumeStatusResizing    = "resizing"
	VolumeStatusRemoving    = "removing"
	VolumeStatusReady       = "ready"
)

VolumeStatus is the status of a volume.

View Source 
const (
	// MaxTempSectorBlocks is the maximum number of blocks that a temp sector
	// can be stored for.
	MaxTempSectorBlocks = 144 * 7 // 7 days
)

Variables

View Source 
var (
	// ErrMigrationFailed is returned when a volume fails to migrate all
	// of its sectors.
	ErrMigrationFailed = errors.New("migration failed")
	// ErrNotEnoughStorage is returned when there is not enough storage space to
	// store a sector.
	ErrNotEnoughStorage = errors.New("not enough storage")
	// ErrSectorNotFound is returned when a sector is not found.
	ErrSectorNotFound = errors.New("sector not found")
	// ErrVolumeNotEmpty is returned when trying to remove or shrink a volume
	// that has not been emptied.
	ErrVolumeNotEmpty = errors.New("volume is not empty")
	// ErrVolumeNotFound is returned when a volume is not found.
	ErrVolumeNotFound = errors.New("volume not found")
)
View Source 
var ErrVolumeNotAvailable = errors.New("volume not available")

ErrVolumeNotAvailable is returned when a volume is not available

Functions

This section is empty.

Types

type Alerts 

type Alerts interface {
	Register(alerts.Alert)
	Dismiss(...types.Hash256)
}

Alerts can be used to register alerts.

type ChainManager 

type ChainManager interface {
	TipState() consensus.State
	Subscribe(s modules.ConsensusSetSubscriber, ccID modules.ConsensusChangeID, cancel <-chan struct{}) error
}

A ChainManager is used to get the current consensus state.

type MigrateFunc added in v1.0.3 

type MigrateFunc func(location SectorLocation) error

MigrateFunc is a callback function that is called for each sector that needs to be migrated If the function returns an error, the sector should be skipped and migration should continue.

type SectorLocation 

type SectorLocation struct {
	ID     int64
	Volume int64
	Index  uint64
	Root   types.Hash256
}

A SectorLocation is a location of a sector within a volume.

type SectorReference added in v1.0.0 

type SectorReference struct {
	Contracts   []types.FileContractID `json:"contracts"`
	TempStorage int                    `json:"tempStorage"`
	Locks       int                    `json:"locks"`
}

A SectorReference contains the references to a sector.

type TempSector 

type TempSector struct {
	Root       types.Hash256
	Expiration uint64
}

A TempSector is a stored sector that is not attached to a contract. It will be deleted after the expiration height unless it is appended to a contract.

type Volume 

type Volume struct {
	ID           int64  `json:"id"`
	LocalPath    string `json:"localPath"`
	UsedSectors  uint64 `json:"usedSectors"`
	TotalSectors uint64 `json:"totalSectors"`
	ReadOnly     bool   `json:"readOnly"`
	Available    bool   `json:"available"`
}

A Volume stores and retrieves sector data

type VolumeManager 

type VolumeManager struct {
	// contains filtered or unexported fields
}

A VolumeManager manages storage using local volumes.

func NewVolumeManager 

func NewVolumeManager(vs VolumeStore, a Alerts, cm ChainManager, log *zap.Logger, sectorCacheSize uint32) (*VolumeManager, error)

NewVolumeManager creates a new VolumeManager.

func (*VolumeManager) AddTemporarySectors 

func (vm *VolumeManager) AddTemporarySectors(sectors []TempSector) error

AddTemporarySectors adds sectors to the temporary store. The sectors are not referenced by a contract and will be removed at the expiration height.

func (*VolumeManager) AddVolume 

func (vm *VolumeManager) AddVolume(ctx context.Context, localPath string, maxSectors uint64, result chan<- error) (Volume, error)

AddVolume adds a new volume to the storage manager

func (*VolumeManager) CacheStats 

func (vm *VolumeManager) CacheStats() (hits, misses uint64)

CacheStats returns the number of cache hits and misses.

func (*VolumeManager) Close 

func (vm *VolumeManager) Close() error

Close gracefully shutsdown the volume manager.

func (*VolumeManager) LockSector 

func (vm *VolumeManager) LockSector(root types.Hash256) (func() error, error)

LockSector prevents the sector with the given root from being pruned. If the sector does not exist, an error is returned. Release must be called when the sector is no longer needed.

func (*VolumeManager) ProcessConsensusChange added in v0.2.0 

func (vm *VolumeManager) ProcessConsensusChange(cc modules.ConsensusChange)

ProcessConsensusChange is called when the consensus set changes.

func (*VolumeManager) Read 

func (vm *VolumeManager) Read(root types.Hash256) (*[rhp2.SectorSize]byte, error)

Read reads the sector with the given root

func (*VolumeManager) RemoveSector 

func (vm *VolumeManager) RemoveSector(root types.Hash256) error

RemoveSector deletes a sector's metadata and zeroes its data.

func (*VolumeManager) RemoveVolume 

func (vm *VolumeManager) RemoveVolume(ctx context.Context, id int64, force bool, result chan<- error) error

RemoveVolume removes a volume from the manager.

func (*VolumeManager) ResizeCache 

func (vm *VolumeManager) ResizeCache(size uint32)

ResizeCache resizes the cache to the given size.

func (*VolumeManager) ResizeVolume 

func (vm *VolumeManager) ResizeVolume(ctx context.Context, id int64, maxSectors uint64, result chan<- error) error

ResizeVolume resizes a volume to the specified size.

func (*VolumeManager) SectorReferences added in v1.0.0 

func (vm *VolumeManager) SectorReferences(root types.Hash256) (SectorReference, error)

SectorReferences returns the references to a sector.

func (*VolumeManager) SetReadOnly 

func (vm *VolumeManager) SetReadOnly(id int64, readOnly bool) error

SetReadOnly sets the read-only status of a volume.

func (*VolumeManager) Sync 

func (vm *VolumeManager) Sync() error

Sync syncs the data files of changed volumes.

func (*VolumeManager) Usage 

func (vm *VolumeManager) Usage() (usedSectors uint64, totalSectors uint64, err error)

Usage returns the total and used storage space, in sectors, from the storage manager.

func (*VolumeManager) Volume 

func (vm *VolumeManager) Volume(id int64) (VolumeMeta, error)

Volume returns a volume by its ID.

func (*VolumeManager) Volumes 

func (vm *VolumeManager) Volumes() ([]VolumeMeta, error)

Volumes returns a list of all volumes in the storage manager.

func (*VolumeManager) Write 

func (vm *VolumeManager) Write(root types.Hash256, data *[rhp2.SectorSize]byte) (func() error, error)

Write writes a sector to a volume. release should only be called after the contract roots have been committed to prevent the sector from being deleted.

type VolumeMeta 

type VolumeMeta struct {
	Volume
	VolumeStats
}

VolumeMeta contains the metadata of a volume.

type VolumeStats 

type VolumeStats struct {
	FailedReads      uint64  `json:"failedReads"`
	FailedWrites     uint64  `json:"failedWrites"`
	SuccessfulReads  uint64  `json:"successfulReads"`
	SuccessfulWrites uint64  `json:"successfulWrites"`
	Status           string  `json:"status"`
	Errors           []error `json:"errors"`
}

VolumeStats contains statistics about a volume

type VolumeStore 

type VolumeStore interface {
	// StorageUsage returns the number of used and total bytes in all volumes
	StorageUsage() (usedBytes, totalBytes uint64, _ error)
	// Volumes returns a list of all volumes in the volume store.
	Volumes() ([]Volume, error)
	// Volume returns a volume in the store by its id
	Volume(id int64) (Volume, error)
	// AddVolume initializes a new storage volume and adds it to the volume
	// store. GrowVolume must be called afterwards to initialize the volume
	// to its desired size.
	AddVolume(localPath string, readOnly bool) (int64, error)
	// RemoveVolume removes a storage volume from the volume store. If there
	// are used sectors in the volume, ErrVolumeNotEmpty is returned. If
	// force is true, the volume is removed even if it is not empty.
	RemoveVolume(volumeID int64, force bool) error
	// GrowVolume grows a storage volume's metadata to maxSectors. If the
	// number of sectors in the volume is already greater than maxSectors,
	// nil is returned.
	GrowVolume(volumeID int64, maxSectors uint64) error
	// ShrinkVolume shrinks a storage volume's metadata to maxSectors. If
	// there are used sectors in the shrink range, an error is returned.
	ShrinkVolume(volumeID int64, maxSectors uint64) error

	// SetReadOnly sets the read-only flag on a volume.
	SetReadOnly(volumeID int64, readOnly bool) error
	// SetAvailable sets the available flag on a volume.
	SetAvailable(volumeID int64, available bool) error

	// MigrateSectors returns a new location for each occupied sector of a
	// volume starting at min. The sector data should be copied to the new
	// location and synced to disk during migrateFn. If migrateFn returns an
	// error, migration will continue, but that sector is not migrated.
	MigrateSectors(ctx context.Context, volumeID int64, min uint64, migrateFn MigrateFunc) (migrated, failed int, err error)
	// StoreSector calls fn with an empty location in a writable volume. If
	// the sector root already exists, fn is called with the existing
	// location and exists is true. Unless exists is true, The sector must
	// be written to disk within fn. If fn returns an error, the metadata is
	// rolled back. If no space is available, ErrNotEnoughStorage is
	// returned. The location is locked until release is called.
	//
	// The sector should be referenced by either a contract or temp store
	// before release is called to prevent Prune() from removing it.
	StoreSector(root types.Hash256, fn func(loc SectorLocation, exists bool) error) (release func() error, err error)
	// RemoveSector removes the metadata of a sector and returns its
	// location in the volume.
	RemoveSector(root types.Hash256) error
	// SectorLocation returns the location of a sector or an error if the
	// sector is not found. The location is locked until release is
	// called.
	SectorLocation(root types.Hash256) (loc SectorLocation, release func() error, err error)
	// AddTemporarySectors adds a list of sectors to the temporary store.
	// The sectors are not referenced by a contract and will be removed
	// at the expiration height.
	AddTemporarySectors(sectors []TempSector) error
	// ExpireTempSectors removes all temporary sectors that expired before
	// the given height.
	ExpireTempSectors(height uint64) error
	// IncrementSectorStats increments sector stats
	IncrementSectorStats(reads, writes, cacheHit, cacheMiss uint64) error
	// SectorReferences returns the references to a sector
	SectorReferences(types.Hash256) (SectorReference, error)
}

A VolumeStore stores and retrieves information about storage volumes.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.