storiface

package
v1.24.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 11, 2024 License: Apache-2.0, MIT Imports: 15 Imported by: 0

Documentation

Index

Constants

View Source 
const FSOverheadDen = 10

FSOverheadDen represents the constant value 10, which is used to calculate the overhead in various storage space utilization calculations.

View Source 
const IDSep = "."
View Source 
const TempSuffix = ".tmp"

TempSuffix is appended to file names when they are worked on before being atomically moved to their final location. Local Path GC should be aware of this suffix and have adequate cleanup logic.

Variables

View Source 
var ErrSectorNotFound = errors.New("sector not found")
View Source 
var FSOverheadSeal = map[SectorFileType]int{
	FTUnsealed:    FSOverheadDen,
	FTSealed:      FSOverheadDen,
	FTUpdate:      FSOverheadDen,
	FTUpdateCache: FSOverheadDen*2 + 1,
	FTCache:       141,
	FTPiece:       FSOverheadDen,
	FTKey:         FSOverheadDen*11 + 1,
}

FSOverheadSeal is a map that represents the overheads for different SectorFileType in sectors which are being sealed.

View Source 
var FTAll = func() (out SectorFileType) {
	for _, pathType := range PathTypes {
		out |= pathType
	}
	return out
}()

FTAll represents the combination of all available sector file. It is a variable of type SectorFileType. The value of FTAll is calculated by iterating over the PathTypes slice and using the |= operator to perform a bitwise OR operation on each path type. The result is assigned to the variable out and returned. FTAll is immediately invoked as a function using the anonymous function syntax, so the result is returned as soon as it is calculated.

View Source 
var FsOverheadFinalized = map[SectorFileType]int{
	FTUnsealed:    FSOverheadDen,
	FTSealed:      FSOverheadDen,
	FTUpdate:      FSOverheadDen,
	FTUpdateCache: 1,
	FTCache:       1,
	FTPiece:       FSOverheadDen,
	FTKey:         FSOverheadDen,
}

FsOverheadFinalized is a map that represents the finalized overhead for different types of SectorFileType. The keys in the map are the SectorFileType values, and the values are integers representing the overhead. It is used to calculate the storage space usage for different types of sectors, as shown in the example below: The overhead value is retrieved from FsOverheadFinalized by using the SectorFileType value as the key. If the overhead value is not found in the map, an error is returned indicating that there is no finalized overhead information for the given sector type.

View Source 
var PathTypes = []SectorFileType{FTUnsealed, FTSealed, FTCache, FTUpdate, FTUpdateCache, FTPiece, FTKey}

PathTypes is a slice of SectorFileType that represents different types of sector file paths. It contains the following types of sector file paths: - FTUnsealed: represents regular unsealed sectors - FTSealed: represents sealed sectors - FTCache: represents cache sectors - FTUpdate: represents snap sectors - FTUpdateCache: represents snap cache sectors - FTPiece: represents Piece Park sectors

Functions

func ParseSectorID 

func ParseSectorID(baseName string) (abi.SectorID, error)

ParseSectorID parses a sector ID from a given base name. It expects the format "s-t0%d-%d", where the first %d represents the miner ID and the second %d represents the sector number.

Parameters: - baseName: The base name from which to parse the sector ID.

Returns: - abi.SectorID: The parsed sector ID. - error: An error if parsing fails.

Example usage: 

id, err := ParseSectorID(baseName)
if err != nil {
  // handle error
}
// use id

func PathByType 

func PathByType(sps SectorPaths, fileType SectorFileType) string

PathByType returns the path associated with the specified fileType in the given SectorPaths. It panics if the requested path type is unknown.

func SectorName 

func SectorName(sid abi.SectorID) string

SectorName returns the name of a sector in the format "s-t0<Miner>-<Number>"

Parameters:

  • sid: The sector ID

Returns:

  • The name of the sector as a string

func SetPathByType 

func SetPathByType(sps *SectorPaths, fileType SectorFileType, p string)

Types

type AcquireMode 

type AcquireMode string
const (
	AcquireMove AcquireMode = "move"
	AcquireCopy AcquireMode = "copy"
)

type AcquireOption 

type AcquireOption func(*AcquireSettings)

func AcquireInto 

func AcquireInto(pathIDs PathsWithIDs) AcquireOption

type AcquireSettings 

type AcquireSettings struct {
	Into *PathsWithIDs
}

type CallError 

type CallError struct {
	Code    ErrorCode
	Message string
	// contains filtered or unexported fields
}

func Err 

func Err(code ErrorCode, sub error) *CallError

func (*CallError) ErrCode 

func (c *CallError) ErrCode() ErrorCode

func (*CallError) Error 

func (c *CallError) Error() string

func (*CallError) Unwrap 

func (c *CallError) Unwrap() error

type Data 

type Data = io.Reader

type Decl 

type Decl struct {
	abi.SectorID
	SectorFileType
}

type ErrorCode 

type ErrorCode int
const (
	// Temp Errors
	ErrTempUnknown ErrorCode = iota + 100
	ErrTempWorkerRestart
	ErrTempAllocateSpace
)
const (
	ErrUnknown ErrorCode = iota
)

type FallbackChallenges 

type FallbackChallenges struct {
	Sectors    []abi.SectorNumber
	Challenges map[abi.SectorNumber][]uint64
}

type Group 

type Group = string

type HealthReport 

type HealthReport struct {
	Stat fsutil.FsStat
	Err  string
}

type ID 

type ID string

ID identifies sector storage by UUID. One sector storage should map to one 

filesystem, local or networked / shared by multiple machines

type IDList 

type IDList []ID

func ParseIDList 

func ParseIDList(s string) IDList

func (IDList) String 

func (il IDList) String() string

type LocalPath 

type LocalPath struct {
	Path string
}

type LocalStorageMeta 

type LocalStorageMeta struct {
	ID ID

	// A high weight means data is more likely to be stored in this path
	Weight uint64 // 0 = readonly

	// Intermediate data for the sealing process will be stored here
	CanSeal bool

	// Finalized sectors that will be proved over time will be stored here
	CanStore bool

	// MaxStorage specifies the maximum number of bytes to use for sector storage
	// (0 = unlimited)
	MaxStorage uint64

	// List of storage groups this path belongs to
	Groups []string

	// List of storage groups to which data from this path can be moved. If none
	// are specified, allow to all
	AllowTo []string

	// AllowTypes lists sector file types which are allowed to be put into this
	// path. If empty, all file types are allowed.
	//
	// Valid values:
	// - "unsealed"
	// - "sealed"
	// - "cache"
	// - "update"
	// - "update-cache"
	// Any other value will generate a warning and be ignored.
	AllowTypes []string

	// DenyTypes lists sector file types which aren't allowed to be put into this
	// path.
	//
	// Valid values:
	// - "unsealed"
	// - "sealed"
	// - "cache"
	// - "update"
	// - "update-cache"
	// Any other value will generate a warning and be ignored.
	DenyTypes []string

	// AllowMiners lists miner IDs which are allowed to store their sector data into
	// this path. If empty, all miner IDs are allowed
	AllowMiners []string

	// DenyMiners lists miner IDs which are denied to store their sector data into
	// this path
	DenyMiners []string
}

LocalStorageMeta path/sectorstore.json

type PaddedByteIndex 

type PaddedByteIndex uint64

type PathType 

type PathType string
const (
	PathStorage PathType = "storage"
	PathSealing PathType = "sealing"
)

type PathsWithIDs 

type PathsWithIDs struct {
	Paths SectorPaths
	IDs   SectorPaths
}

PathsWithIDs represents paths and IDs for sector files.

func (PathsWithIDs) HasAllSet 

func (p PathsWithIDs) HasAllSet(ft SectorFileType) bool

HasAllSet checks if all paths and IDs in PathsWithIDs have a corresponding path set for the specified SectorFileType. It returns true if all paths and IDs are set, and false otherwise.

type PieceNumber 

type PieceNumber uint64

PieceNumber is a reference to a piece in the storage system; mapping between pieces in the storage system and piece CIDs is maintained by the storage index

func (PieceNumber) Ref 

func (pn PieceNumber) Ref() SectorRef

type PostSectorChallenge 

type PostSectorChallenge struct {
	SealProof    abi.RegisteredSealProof
	SectorNumber abi.SectorNumber
	SealedCID    cid.Cid
	Challenge    []uint64
	Update       bool
}

type PreCommit1Out 

type PreCommit1Out []byte

type Prover 

type Prover interface {
	AggregateSealProofs(aggregateInfo proof.AggregateSealVerifyProofAndInfos, proofs [][]byte) ([]byte, error)
}

Prover contains cheap proving-related methods

type RGetter 

type RGetter func(ctx context.Context, id abi.SectorID) (sealed cid.Cid, update bool, err error)

type ReplicaUpdateProof 

type ReplicaUpdateProof []byte

type SecDataHttpHeader 

type SecDataHttpHeader struct {
	Key   string
	Value string
}

func (*SecDataHttpHeader) MarshalCBOR 

func (t *SecDataHttpHeader) MarshalCBOR(w io.Writer) error

func (*SecDataHttpHeader) UnmarshalCBOR 

func (t *SecDataHttpHeader) UnmarshalCBOR(r io.Reader) (err error)

type SectorCids 

type SectorCids struct {
	Unsealed cid.Cid
	Sealed   cid.Cid
}

type SectorFileType 

type SectorFileType int

SectorFileType represents the type of a sector file TypeFromString converts a string to a SectorFileType 

const (
	// "regular" sectors
	FTUnsealed SectorFileType = 1 << iota
	FTSealed
	FTCache

	// snap
	FTUpdate
	FTUpdateCache

	// Piece Park
	FTPiece

	// Curio unseal
	FTKey

	FileTypes = iota
)

FTUnsealed represents an unsealed sector file. FTSealed represents a sealed sector file. FTCache represents a cache sector file. FTUpdate represents an update sector file. FTUpdateCache represents an update cache sector file. FTPiece represents a Piece Park sector file. FileTypes represents the total number of file.

The SectorFileType type is an integer type that represents different sector file. It has several methods to manipulate and query the file. The String method returns a string representation of the file. The Strings method returns a slice of string representations of all the file that are set in the receiver object. The AllSet method returns a slice of all the file that are set in the receiver object. The Has method checks whether a specific file type is set in the receiver object. The SealSpaceUse method calculates the space used by the receiver object in sealing a sector of a given size. The SubAllowed method removes selected file from the receiver object based on a list of allowed and denied file. The Unset method removes selected file from the receiver object. The AnyAllowed method checks whether any file in the receiver object are allowed based on a list of allowed and denied file. The Allowed method checks whether all file in the receiver object are allowed based on a list of allowed and denied file. The StoreSpaceUse method calculates the space used by the receiver object in storing a sector of a given size. The All method returns an array that represents which file are set in the receiver object. The IsNone method checks whether the receiver object represents no file. 

const (
	FTNone SectorFileType = 0
)

FTNone represents a sector file type of none. This constant is used in the StorageLock method to specify that a sector should not have any file locked. Example usage: err := m.index.StorageLock(ctx, sector.ID, storiface.FTNone, storiface.FTSealed|storiface.FTUnsealed|storiface.FTCache)

func TypeFromString 

func TypeFromString(s string) (SectorFileType, error)

TypeFromString converts a string representation of a SectorFileType to its corresponding value. It returns the SectorFileType and nil error if the string matches one of the existing types. If the string does not match any type, it returns 0 and an error.

func (SectorFileType) All 

func (t SectorFileType) All() [FileTypes]bool

All returns an array indicating whether each FileTypes flag is set in the SectorFileType.

func (SectorFileType) AllSet 

func (t SectorFileType) AllSet() []SectorFileType

AllSet returns a slice of SectorFileType values that are set in the SectorFileType receiver value

func (SectorFileType) Allowed 

func (t SectorFileType) Allowed(allowTypes []string, denyTypes []string) bool

Allowed checks if the SectorFileType is allowed based on the given allowTypes and denyTypes. Returns true if the SectorFileType is allowed, otherwise false.

func (SectorFileType) AnyAllowed 

func (t SectorFileType) AnyAllowed(allowTypes []string, denyTypes []string) bool

AnyAllowed checks if the SectorFileType has any allowed types and no denied types.

func (SectorFileType) Has 

func (t SectorFileType) Has(singleType SectorFileType) bool

Has checks if the SectorFileType has a specific singleType.

func (SectorFileType) IsNone 

func (t SectorFileType) IsNone() bool

IsNone checks if the SectorFileType value is equal to zero. It returns true if the value is zero, indicating that the type is none.

func (SectorFileType) SealSpaceUse 

func (t SectorFileType) SealSpaceUse(ssize abi.SectorSize) (uint64, error)

SealSpaceUse calculates the amount of space needed for sealing the sector based on the given sector size. It iterates over the different path types and calculates the space needed for each path type using the FSOverheadSeal map. The overhead value is multiplied by the sector size and divided by the FSOverheadDen constant. The total space needed is accumulated and returned. If there is no seal overhead information for a particular path type, an error is returned.

Example usage: 

fileType := FTSealed | FTCache
sectorSize := abi.SectorSize(32 << 20) // 32 MiB
spaceNeeded, err := fileType.SealSpaceUse(sectorSize)

Parameters: 

ssize: The size of the sector

Returns: 

uint64: The amount of space needed for sealing the sector
error: If there is no seal overhead information for a path type

func (SectorFileType) StoreSpaceUse 

func (t SectorFileType) StoreSpaceUse(ssize abi.SectorSize) (uint64, error)

StoreSpaceUse calculates the space used for storing sectors of a specific file type. It takes the sector size as input and returns the total space needed in bytes and an error, if any. The calculation is based on the finalized overhead information for the file type. If the overhead information is not available for a particular file type, an error will be returned.

func (SectorFileType) String 

func (t SectorFileType) String() string

String returns a string representation of the SectorFileType.

func (SectorFileType) Strings 

func (t SectorFileType) Strings() []string

Strings returns a slice of strings representing the names of the SectorFileType values that are set in the receiver value. Example usage: 

fileType := SectorFileType(FTSealed | FTCache)
names := fileType.Strings() // names = ["sealed", "cache"]
fmt.Println(names)

func (SectorFileType) SubAllowed 

func (t SectorFileType) SubAllowed(allowTypes []string, denyTypes []string) SectorFileType

SubAllowed takes in two parameters: allowTypes and denyTypes, both of which are slices of strings. If allowTypes is not empty, the method sets a denyMask with all bits set to 1, and then iterates over each allowType, converting it to a SectorFileType using the TypeFromString function and unsetting the corresponding bit in the denyMask. If a string in allowTypes cannot be converted to a valid SectorFileType, it is ignored. After processing allowTypes, the method iterates over each denyType, converting it to a SectorFileType using the TypeFromString function and setting the corresponding bit in the denyMask. If a string in denyTypes cannot be converted to a valid SectorFileType, it is ignored. Finally, the method returns the bitwise AND of the original SectorFileType and the denyMask. The returned SectorFileType will only allow the types specified in allowTypes and exclude the types specified in denyTypes.`

func (SectorFileType) Unset 

func (t SectorFileType) Unset(toUnset SectorFileType) SectorFileType

Unset removes the specified sector file type(s) from the current SectorFileType value. It performs a bitwise AND operation between the current value and the bitwise complement of the toUnset value. The result is returned as a new SectorFileType value. Any bits that are set in toUnset will be cleared in the result. Usage: result = value.Unset(typesToUnset)

type SectorLocation 

type SectorLocation struct {
	// Local when set to true indicates to lotus that sector data is already
	// available locally; When set lotus will skip fetching sector data, and
	// only check that sector data exists in sector storage
	Local bool

	// URL to the sector data
	// For sealed/unsealed sector, lotus expects octet-stream
	// For cache, lotus expects a tar archive with cache files
	// Valid schemas:
	// - http:// / https://
	URL string

	// optional http headers to use when requesting sector data
	Headers []SecDataHttpHeader
}

func (*SectorLocation) HttpHeaders 

func (sd *SectorLocation) HttpHeaders() http.Header

func (*SectorLocation) MarshalCBOR 

func (t *SectorLocation) MarshalCBOR(w io.Writer) error

func (*SectorLocation) UnmarshalCBOR 

func (t *SectorLocation) UnmarshalCBOR(r io.Reader) (err error)

type SectorLock 

type SectorLock struct {
	Sector abi.SectorID
	Write  [FileTypes]uint
	Read   [FileTypes]uint
}

type SectorLocks 

type SectorLocks struct {
	Locks []SectorLock
}

type SectorPaths 

type SectorPaths struct {
	ID abi.SectorID

	Unsealed    string
	Sealed      string
	Cache       string
	Update      string
	UpdateCache string
	Piece       string
	Key         string
}

SectorPaths represents the paths for different sector files.

func (SectorPaths) HasAllSet 

func (sp SectorPaths) HasAllSet(ft SectorFileType) bool

HasAllSet checks if all paths of a SectorPaths struct are set for a given SectorFileType.

func (SectorPaths) Subset 

func (sp SectorPaths) Subset(filter SectorFileType) SectorPaths

Subset returns a new instance of SectorPaths that contains only the paths specified by the filter SectorFileType. It iterates over each fileType in the filter, retrieves the corresponding path from the original SectorPaths instance, and sets it in the new instance. Finally, it sets the ID field of the new instance to be the same as the original instance.

type SectorRef 

type SectorRef struct {
	ID        abi.SectorID
	ProofType abi.RegisteredSealProof
}

type SectorStorageInfo 

type SectorStorageInfo struct {
	ID       ID
	URLs     []string // TODO: Support non-http transports
	BaseURLs []string
	Weight   uint64

	CanSeal  bool
	CanStore bool

	Primary bool

	AllowTypes  []string
	DenyTypes   []string
	AllowMiners []string
	DenyMiners  []string
}

type StorageConfig 

type StorageConfig struct {
	StoragePaths []LocalPath
}

StorageConfig .lotusstorage/storage.json

type StorageInfo 

type StorageInfo struct {
	// ID is the UUID of the storage path
	ID ID

	// URLs for remote access
	URLs []string // TODO: Support non-http transports

	// Storage path weight; higher number means that the path will be preferred more often
	Weight uint64

	// MaxStorage is the number of bytes allowed to be used by files in the
	// storage path
	MaxStorage uint64

	// CanStore is true when the path is allowed to be used for io-intensive
	// sealing operations
	CanSeal bool

	// CanStore is true when the path is allowed to be used for long-term storage
	CanStore bool

	// Groups is the list of path groups this path belongs to
	Groups []Group

	// AllowTo is the list of paths to which data from this path can be moved to
	AllowTo []Group

	// AllowTypes lists sector file types which are allowed to be put into this
	// path. If empty, all file types are allowed.
	//
	// Valid values:
	// - "unsealed"
	// - "sealed"
	// - "cache"
	// - "update"
	// - "update-cache"
	// Any other value will generate a warning and be ignored.
	AllowTypes []string

	// DenyTypes lists sector file types which aren't allowed to be put into this
	// path.
	//
	// Valid values:
	// - "unsealed"
	// - "sealed"
	// - "cache"
	// - "update"
	// - "update-cache"
	// Any other value will generate a warning and be ignored.
	DenyTypes []string

	// AllowMiners lists miner IDs which are allowed to store their sector data into
	// this path. If empty, all miner IDs are allowed
	AllowMiners []string

	// DenyMiners lists miner IDs which are denied to store their sector data into
	// this path
	DenyMiners []string
}

type StoragePath 

type StoragePath struct {
	ID     ID
	Weight uint64

	LocalPath string

	CanSeal  bool
	CanStore bool
}

type UnpaddedByteIndex 

type UnpaddedByteIndex uint64

func UnpaddedCeil 

func UnpaddedCeil(n uint64) UnpaddedByteIndex

func UnpaddedFloor 

func UnpaddedFloor(n uint64) UnpaddedByteIndex

func (UnpaddedByteIndex) Padded 

func (i UnpaddedByteIndex) Padded() PaddedByteIndex

func (UnpaddedByteIndex) Valid 

func (i UnpaddedByteIndex) Valid() error

type Verifier 

type Verifier interface {
	VerifySeal(proof.SealVerifyInfo) (bool, error)
	VerifyAggregateSeals(aggregate proof.AggregateSealVerifyProofAndInfos) (bool, error)
	VerifyReplicaUpdate(update proof.ReplicaUpdateInfo) (bool, error)
	VerifyWinningPoSt(ctx context.Context, info proof.WinningPoStVerifyInfo) (bool, error)
	VerifyWindowPoSt(ctx context.Context, info proof.WindowPoStVerifyInfo) (bool, error)

	GenerateWinningPoStSectorChallenge(context.Context, abi.RegisteredPoStProof, abi.ActorID, abi.PoStRandomness, uint64) ([]uint64, error)
}

type WindowPoStResult 

type WindowPoStResult struct {
	PoStProofs proof.PoStProof
	Skipped    []abi.SectorID
}

type WorkError 

type WorkError interface {
	ErrCode() ErrorCode
}

type WorkerJob 

type WorkerJob struct{} // dummy

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.