storage

package
v2.1.0-beta.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 11, 2025 License: MIT Imports: 17 Imported by: 0

README

Storage Manager

A Storage Manager is responsible for storing and retrieving sectors uploaded to the host. Sectors can now come from two different sources: contracts and temporary storage.

Responsibilities

  • Managing stored sectors
  • Reporting storage inconsistencies
  • Deleting sectors that are no longer referenced

Storage

The RHP packages interact directly with a storage manager to write and read sectors from persistent storage.

Temp Storage

Currently the storage manager also manages temporary storage, but that may not always be the case.

Volume Manager

In hostd the Volume Manager is the default implementation of a storage manager. It is responsible for managing the storage of sectors on disk. It stores metadata in a SQLite database and sector data in a flat file.

Responsibilities

  • Managing volume metadata
  • Managing sector metadata
  • Managing volume data
  • Managing temporary storage
  • Reporting disk errors
  • Migrating volume data

Requirements

  • Stored sectors must stay consistent between storage manager, contract manager, and temp storage
  • Metadata must be ACID between contract revisions in RHP2 and program finalization in RHP3.
  • Support for setting volumes to read-only
  • Support for checking on-disk consistency and reporting errors
  • Support for resizing and removing volumes
  • During resize, sectors must be able to be moved within the same volume
  • Avoid locking a folder during resize or remove operations; sectors not being actively moved should be readable.
  • Support for multiple volumes with dynamic sizes
  • Provide alerts and status information on long-running operations

Documentation

Index

Constants

View Source 
const (
	VolumeStatusUnavailable = "unavailable"
	VolumeStatusCreating    = "creating"
	VolumeStatusResizing    = "resizing"
	VolumeStatusRemoving    = "removing"
	VolumeStatusReady       = "ready"
)

VolumeStatus is the status of a volume.

View Source 
const (
	// MaxTempSectorBlocks is the maximum number of blocks that a temp sector
	// can be stored for.
	MaxTempSectorBlocks = 144 * 7 // 7 days
)

Variables

View Source 
var (
	// ErrMigrationFailed is returned when a volume fails to migrate all
	// of its sectors.
	ErrMigrationFailed = errors.New("migration failed")
	// ErrNotEnoughStorage is returned when there is not enough storage space to
	// store a sector.
	ErrNotEnoughStorage = errors.New("not enough storage")
	// ErrSectorNotFound is returned when a sector is not found.
	ErrSectorNotFound = errors.New("sector not found")
	// ErrVolumeNotEmpty is returned when trying to remove or shrink a volume
	// that has not been emptied.
	ErrVolumeNotEmpty = errors.New("volume is not empty")
	// ErrVolumeNotFound is returned when a volume is not found.
	ErrVolumeNotFound = errors.New("volume not found")
)
View Source 
var ErrVolumeNotAvailable = errors.New("volume not available")

ErrVolumeNotAvailable is returned when a volume is not available

Functions

This section is empty.

Types

type Alerts 

type Alerts interface {
	Register(alerts.Alert)
	Dismiss(...types.Hash256)
}

Alerts can be used to register alerts.

type MigrateFunc 

type MigrateFunc func(from, to SectorLocation) error

MigrateFunc is called for every sector that needs migration. The sector should be migrated from 'from' to 'to' during migrate func.

type SectorLocation 

type SectorLocation struct {
	ID     int64
	Volume int64
	Index  uint64
	Root   types.Hash256
}

A SectorLocation is a location of a sector within a volume.

type SectorReference 

type SectorReference struct {
	Contracts   []types.FileContractID `json:"contracts"`
	TempStorage int                    `json:"tempStorage"`
}

A SectorReference contains the references to a sector.

type StoreFunc 

type StoreFunc func(loc SectorLocation) error

StoreFunc is called for every sector that needs written to disk.

type TempSector 

type TempSector struct {
	Root       types.Hash256
	Expiration uint64
}

A TempSector is a stored sector that is not attached to a contract. It will be deleted after the expiration height unless it is appended to a contract.

type Volume 

type Volume struct {
	ID           int64  `json:"id"`
	LocalPath    string `json:"localPath"`
	UsedSectors  uint64 `json:"usedSectors"`
	TotalSectors uint64 `json:"totalSectors"`
	ReadOnly     bool   `json:"readOnly"`
	Available    bool   `json:"available"`
}

A Volume stores and retrieves sector data

type VolumeManager 

type VolumeManager struct {
	// contains filtered or unexported fields
}

A VolumeManager manages storage using local volumes.

func NewVolumeManager 

func NewVolumeManager(vs VolumeStore, opts ...VolumeManagerOption) (*VolumeManager, error)

NewVolumeManager creates a new VolumeManager.

func (*VolumeManager) AddTemporarySectors 

func (vm *VolumeManager) AddTemporarySectors(sectors []TempSector) error

AddTemporarySectors adds sectors to the temporary store. The sectors are not referenced by a contract and will be removed at the expiration height.

func (*VolumeManager) AddVolume 

func (vm *VolumeManager) AddVolume(ctx context.Context, localPath string, maxSectors uint64, result chan<- error) (Volume, error)

AddVolume adds a new volume to the storage manager

func (*VolumeManager) CacheStats 

func (vm *VolumeManager) CacheStats() (hits, misses uint64)

CacheStats returns the number of cache hits and misses.

func (*VolumeManager) Close 

func (vm *VolumeManager) Close() error

Close gracefully shutsdown the volume manager.

func (*VolumeManager) HasSector 

func (vm *VolumeManager) HasSector(root types.Hash256) (bool, error)

HasSector returns true if the host is storing a sector

func (*VolumeManager) ProcessActions 

func (vm *VolumeManager) ProcessActions(index types.ChainIndex) error

ProcessActions processes the actions for the given chain index.

func (*VolumeManager) ReadSector 

func (vm *VolumeManager) ReadSector(root types.Hash256) (*[proto2.SectorSize]byte, error)

ReadSector reads the sector with the given root from disk

func (*VolumeManager) RemoveSector 

func (vm *VolumeManager) RemoveSector(root types.Hash256) error

RemoveSector deletes a sector's metadata and zeroes its data.

func (*VolumeManager) RemoveVolume 

func (vm *VolumeManager) RemoveVolume(ctx context.Context, id int64, force bool, result chan<- error) error

RemoveVolume removes a volume from the manager.

func (*VolumeManager) ResizeCache 

func (vm *VolumeManager) ResizeCache(size uint32)

ResizeCache resizes the cache to the given size.

func (*VolumeManager) ResizeVolume 

func (vm *VolumeManager) ResizeVolume(ctx context.Context, id int64, maxSectors uint64, result chan<- error) error

ResizeVolume resizes a volume to the specified size.

func (*VolumeManager) SectorReferences 

func (vm *VolumeManager) SectorReferences(root types.Hash256) (SectorReference, error)

SectorReferences returns the references to a sector.

func (*VolumeManager) SetReadOnly 

func (vm *VolumeManager) SetReadOnly(id int64, readOnly bool) error

SetReadOnly sets the read-only status of a volume.

func (*VolumeManager) StoreSector 

func (vm *VolumeManager) StoreSector(root types.Hash256, data *[proto4.SectorSize]byte, expiration uint64) error

StoreSector writes a sector to disk and adds it to temporary storage

func (*VolumeManager) Sync 

func (vm *VolumeManager) Sync() error

Sync syncs the data files of changed volumes.

func (*VolumeManager) Usage 

func (vm *VolumeManager) Usage() (usedSectors uint64, totalSectors uint64, err error)

Usage returns the total and used storage space, in sectors, from the storage manager.

func (*VolumeManager) Volume 

func (vm *VolumeManager) Volume(id int64) (VolumeMeta, error)

Volume returns a volume by its ID.

func (*VolumeManager) Volumes 

func (vm *VolumeManager) Volumes() ([]VolumeMeta, error)

Volumes returns a list of all volumes in the storage manager.

func (*VolumeManager) Write 

func (vm *VolumeManager) Write(root types.Hash256, data *[proto2.SectorSize]byte) error

Write writes a sector to a volume.

type VolumeManagerOption 

type VolumeManagerOption func(*VolumeManager)

A VolumeManagerOption configures a VolumeManager.

func WithAlerter 

func WithAlerter(a Alerts) VolumeManagerOption

WithAlerter sets the alerter for the manager.

func WithCacheSize 

func WithCacheSize(cacheSize int) VolumeManagerOption

WithCacheSize sets the sector cache size for the manager.

func WithLogger 

func WithLogger(l *zap.Logger) VolumeManagerOption

WithLogger sets the logger for the manager.

func WithPruneInterval 

func WithPruneInterval(d time.Duration) VolumeManagerOption

WithPruneInterval sets the time between cleaning up dereferenced sectors.

type VolumeMeta 

type VolumeMeta struct {
	Volume
	VolumeStats
}

VolumeMeta contains the metadata of a volume.

type VolumeStats 

type VolumeStats struct {
	FailedReads      uint64  `json:"failedReads"`
	FailedWrites     uint64  `json:"failedWrites"`
	SuccessfulReads  uint64  `json:"successfulReads"`
	SuccessfulWrites uint64  `json:"successfulWrites"`
	Status           string  `json:"status"`
	Errors           []error `json:"errors"`
}

VolumeStats contains statistics about a volume

type VolumeStore 

type VolumeStore interface {
	// StorageUsage returns the number of used and total bytes in all volumes
	StorageUsage() (usedBytes, totalBytes uint64, _ error)
	// Volumes returns a list of all volumes in the volume store.
	Volumes() ([]Volume, error)
	// Volume returns a volume in the store by its id
	Volume(id int64) (Volume, error)
	// AddVolume initializes a new storage volume and adds it to the volume
	// store. GrowVolume must be called afterwards to initialize the volume
	// to its desired size.
	AddVolume(localPath string, readOnly bool) (int64, error)
	// RemoveVolume removes a storage volume from the volume store. If there
	// are used sectors in the volume, ErrVolumeNotEmpty is returned. If
	// force is true, the volume is removed even if it is not empty.
	RemoveVolume(volumeID int64, force bool) error
	// GrowVolume grows a storage volume's metadata to maxSectors. If the
	// number of sectors in the volume is already greater than maxSectors,
	// nil is returned.
	GrowVolume(volumeID int64, maxSectors uint64) error
	// ShrinkVolume shrinks a storage volume's metadata to maxSectors. If
	// there are used sectors in the shrink range, an error is returned.
	ShrinkVolume(volumeID int64, maxSectors uint64) error

	// SetReadOnly sets the read-only flag on a volume.
	SetReadOnly(volumeID int64, readOnly bool) error
	// SetAvailable sets the available flag on a volume.
	SetAvailable(volumeID int64, available bool) error

	// PruneSectors removes all sectors that have not been accessed since
	// lastAccess and are no longer referenced by a contract or temp storage.
	// If the context is canceled, pruning is stopped and the function returns
	// with the error.
	PruneSectors(ctx context.Context, lastAccess time.Time) error

	// MigrateSectors returns a new location for each occupied sector of a
	// volume starting at min. The sector data should be copied to the new
	// location and synced to disk during migrateFn. If migrateFn returns an
	// error, migration will continue, but that sector is not migrated.
	MigrateSectors(ctx context.Context, volumeID int64, min uint64, fn MigrateFunc) (migrated, failed int, err error)
	// StoreSector calls fn with an empty location in a writable volume. If
	// the sector root already exists, nil is returned. The sector should be
	// written to disk within fn. If fn returns an error, the metadata is
	// rolled back and the error is returned. If no space is available,
	// ErrNotEnoughStorage is returned.
	StoreSector(root types.Hash256, fn StoreFunc) error
	// RemoveSector removes the metadata of a sector and returns its
	// location in the volume.
	RemoveSector(root types.Hash256) error
	// HasSector returns true if the sector is stored by the host.
	HasSector(root types.Hash256) (bool, error)
	// SectorLocation returns the location of a sector or an error if the
	// sector is not found. The location is locked until release is
	// called.
	SectorLocation(root types.Hash256) (loc SectorLocation, err error)
	// AddTempSector adds a sector to temporary storage. The sectors will be deleted
	// after the expiration height
	AddTempSector(root types.Hash256, expiration uint64) error
	// AddTemporarySectors adds a list of sectors to the temporary store.
	// The sectors are not referenced by a contract and will be removed
	// at the expiration height.
	//
	// Deprecated: use AddTempSector
	AddTemporarySectors(sectors []TempSector) error
	// ExpireTempSectors removes all temporary sectors that expired before
	// the given height.
	ExpireTempSectors(height uint64) error
	// IncrementSectorStats increments sector stats
	IncrementSectorStats(reads, writes, cacheHit, cacheMiss uint64) error
	// SectorReferences returns the references to a sector
	SectorReferences(types.Hash256) (SectorReference, error)
}

A VolumeStore stores and retrieves information about storage volumes.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.