filestore

package
v1.110.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 6, 2024 License: AGPL-3.0 Imports: 26 Imported by: 6

Documentation

Index

Constants

View Source 
const (
	// FormatV0 is the identifier for storage format v0, which also corresponds to an absence of
	// format version information.
	FormatV0 blobstore.FormatVersion = 0
	// FormatV1 is the identifier for storage format v1.
	FormatV1 blobstore.FormatVersion = 1
)
View Source 
const (
	// MaxFormatVersionSupported is the highest supported storage format version for reading, and
	// the only supported storage format version for writing. If stored blobs claim a higher
	// storage format version than this, or a caller requests _writing_ a storage format version
	// which is not this, this software will not know how to perform the read or write and an error
	// will be returned.
	MaxFormatVersionSupported = FormatV1

	// MinFormatVersionSupported is the lowest supported storage format version for reading. If
	// stored blobs claim a lower storage format version than this, this software will not know how
	// to perform the read and an error will be returned.
	MinFormatVersionSupported = FormatV0

	// MinFormatVersionSupportedInTrash is the lowest supported storage format that can be used
	// for storage in the trash.
	MinFormatVersionSupportedInTrash = FormatV1
)
View Source 
const (

	// TrashUsesDayDirsIndicator is the name of a file whose presence under
	// trashdir indicates per-day directories can be used. absence of this file
	// means there is still trash in trash/$namespace/?? directories that needs
	// to be migrated to per-day directories.
	TrashUsesDayDirsIndicator = ".trash-uses-day-dirs-indicator"
)

Variables

View Source 
var (
	// Error is the default filestore error class.
	Error = errs.Class("filestore error")
	// ErrIsDir is the error returned when we encounter a directory named like a blob file
	// while traversing a blob namespace.
	ErrIsDir = Error.New("file is a directory")
)
View Source 
var DefaultConfig = Config{
	WriteBufferSize: 128 * memory.KiB,
	ForceSync:       false,
}

DefaultConfig is the default value for Config.

View Source 
var PathEncoding = base32.NewEncoding("abcdefghijklmnopqrstuvwxyz234567").WithPadding(base32.NoPadding)

PathEncoding is the encoding used for the namespace and key in the filestore.

Functions

func MonFileInTrash added in v1.99.1 

func MonFileInTrash(namespace []byte) *monkit.Meter

MonFileInTrash returns a monkit meter which counts the times a requested blob is found in the trash. It is exported so that it can be activated from outside this package for backwards compatibility. (It is no longer activated from inside this package.)

func New 

func New(log *zap.Logger, dir *Dir, config Config) blobstore.Blobs

New creates a new disk blob store in the specified directory.

func NewAt 

func NewAt(log *zap.Logger, path string, config Config) (blobstore.Blobs, error)

NewAt creates a new disk blob store in the specified directory.

func OpenAt added in v1.100.2 

func OpenAt(log *zap.Logger, path string, config Config) (blobstore.Blobs, error)

OpenAt opens an existing disk blob store in the specified directory.

Types

type Config 

type Config struct {
	WriteBufferSize memory.Size `help:"in-memory buffer for uploads" default:"128KiB"`
	ForceSync       bool        `help:"if true, force disk synchronization and atomic writes" default:"false"`
}

Config is configuration for the blob store.

type CorruptDataError 

type CorruptDataError struct {
	// contains filtered or unexported fields
}

CorruptDataError represents a filesystem or disk error which indicates data corruption.

We use a custom error type here so that we can add explanatory information and wrap the original error at the same time.

func (CorruptDataError) Error 

func (cde CorruptDataError) Error() string

Error returns an error string describing the condition.

func (CorruptDataError) Path 

func (cde CorruptDataError) Path() string

Path returns the path at which the error was encountered.

func (CorruptDataError) Unwrap 

func (cde CorruptDataError) Unwrap() error

Unwrap unwraps the error.

type Dir 

type Dir struct {
	// contains filtered or unexported fields
}

Dir represents single folder for storing blobs.

func NewDir 

func NewDir(log *zap.Logger, path string) (dir *Dir, err error)

NewDir returns folder for storing blobs.

func OpenDir 

func OpenDir(log *zap.Logger, path string, now time.Time) (*Dir, error)

OpenDir opens existing folder for storing blobs.

func (*Dir) Commit 

func (dir *Dir) Commit(ctx context.Context, file *os.File, sync bool, ref blobstore.BlobRef, formatVersion blobstore.FormatVersion) (err error)

Commit commits the temporary file to permanent storage.

func (*Dir) CreateNamedFile added in v1.104.1 

func (dir *Dir) CreateNamedFile(ctx context.Context, ref blobstore.BlobRef, formatVersion blobstore.FormatVersion) (file *os.File, err error)

CreateNamedFile creates a preallocated file in the correct destination directory.

func (*Dir) CreateTemporaryFile 

func (dir *Dir) CreateTemporaryFile(ctx context.Context) (_ *os.File, err error)

CreateTemporaryFile creates a preallocated temporary file in the temp directory.

func (*Dir) CreateVerificationFile 

func (dir *Dir) CreateVerificationFile(ctx context.Context, id storj.NodeID) error

CreateVerificationFile creates a file to be used for storage directory verification.

func (*Dir) Delete 

func (dir *Dir) Delete(ctx context.Context, ref blobstore.BlobRef) (err error)

Delete deletes blobs with the specified ref (in all supported storage formats).

It doesn't return an error if the blob is not found for any reason.

func (*Dir) DeleteNamespace 

func (dir *Dir) DeleteNamespace(ctx context.Context, ref []byte) (err error)

DeleteNamespace deletes blobs folder for a specific namespace.

func (*Dir) DeleteTemporary 

func (dir *Dir) DeleteTemporary(ctx context.Context, file *os.File) (err error)

DeleteTemporary deletes a temporary file.

func (*Dir) DeleteTrashNamespace added in v1.88.2 

func (dir *Dir) DeleteTrashNamespace(ctx context.Context, namespace []byte) (err error)

DeleteTrashNamespace deletes an entire namespace under the trash dir.

func (*Dir) DeleteWithStorageFormat 

func (dir *Dir) DeleteWithStorageFormat(ctx context.Context, ref blobstore.BlobRef, formatVer blobstore.FormatVersion) (err error)

DeleteWithStorageFormat deletes the blob with the specified ref for one specific format version.

It doesn't return an error if the blob isn't found for any reason.

func (*Dir) EmptyTrash 

func (dir *Dir) EmptyTrash(ctx context.Context, namespace []byte, trashedBefore time.Time) (bytesEmptied int64, deletedKeys [][]byte, err error)

EmptyTrash iterates through the toplevel trash directories for the given namespace and recursively deletes any of them more than 24h older than trashedBefore.

func (*Dir) Info 

func (dir *Dir) Info(ctx context.Context) (blobstore.DiskInfo, error)

Info returns information about the current state of the dir.

func (*Dir) ListNamespaces 

func (dir *Dir) ListNamespaces(ctx context.Context) (ids [][]byte, err error)

ListNamespaces finds all known namespace IDs in use in local storage. They are not guaranteed to contain any blobs.

func (*Dir) Open 

func (dir *Dir) Open(ctx context.Context, ref blobstore.BlobRef) (_ *os.File, _ blobstore.FormatVersion, err error)

Open opens the file with the specified ref. It may need to check in more than one location in order to find the blob, if it was stored with an older version of the storage node software. In cases where the storage format version of a blob is already known, OpenWithStorageFormat() will generally be a better choice.

func (*Dir) OpenWithStorageFormat 

func (dir *Dir) OpenWithStorageFormat(ctx context.Context, ref blobstore.BlobRef, formatVer blobstore.FormatVersion) (_ *os.File, err error)

OpenWithStorageFormat opens an already-located blob file with a known storage format version, which avoids the potential need to search through multiple storage formats to find the blob.

func (*Dir) Path 

func (dir *Dir) Path() string

Path returns the directory path.

func (*Dir) RestoreTrash 

func (dir *Dir) RestoreTrash(ctx context.Context, namespace []byte) (keysRestored [][]byte, err error)

RestoreTrash moves every blob in the trash folder back into blobsdir.

func (*Dir) Stat 

func (dir *Dir) Stat(ctx context.Context, ref blobstore.BlobRef) (_ blobstore.BlobInfo, err error)

Stat looks up disk metadata on the blob file. It may need to check in more than one location in order to find the blob, if it was stored with an older version of the storage node software. In cases where the storage format version of a blob is already known, StatWithStorageFormat() will generally be a better choice.

func (*Dir) StatWithStorageFormat 

func (dir *Dir) StatWithStorageFormat(ctx context.Context, ref blobstore.BlobRef, formatVer blobstore.FormatVersion) (_ blobstore.BlobInfo, err error)

StatWithStorageFormat looks up disk metadata on the blob file with the given storage format version. This avoids the need for checking for the file in multiple different storage format types.

func (*Dir) Trash 

func (dir *Dir) Trash(ctx context.Context, ref blobstore.BlobRef, timestamp time.Time) (err error)

Trash moves the blob specified by ref to the trash for every format version.

func (*Dir) TrashWithStorageFormat 

func (dir *Dir) TrashWithStorageFormat(ctx context.Context, ref blobstore.BlobRef, formatVer blobstore.FormatVersion, timestamp time.Time) (err error)

TrashWithStorageFormat moves the blob specified by ref to the trash for the specified format version.

func (*Dir) TryRestoreTrashBlob added in v1.99.1 

func (dir *Dir) TryRestoreTrashBlob(ctx context.Context, ref blobstore.BlobRef) (err error)

TryRestoreTrashBlob attempts to restore a blob from the trash if it exists. It returns nil if the blob was restored, or an error if the blob was not in the trash or could not be restored.

func (*Dir) Verify 

func (dir *Dir) Verify(ctx context.Context, id storj.NodeID) error

Verify verifies that the storage directory is correct by checking for the existence and validity of the verification file.

func (*Dir) WalkNamespace 

func (dir *Dir) WalkNamespace(ctx context.Context, namespace []byte, startFromPrefix string, walkFunc func(blobstore.BlobInfo) error) (err error)

WalkNamespace executes walkFunc for each locally stored blob, stored with storage format V1 or greater, in the given namespace. If walkFunc returns a non-nil error, WalkNamespace will stop iterating and return the error immediately. The ctx parameter is intended specifically to allow canceling iteration early.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.