Documentation ¶
Index ¶
- Constants
- Variables
- func Clone(srcFS, dstFS FS, srcPath, dstPath string, opts ...CloneOption) (bool, error)
- func Copy(fs FS, oldname, newname string) error
- func CopyAcrossFS(srcFS FS, oldname string, dstFS FS, newname string) error
- func IsNoSpaceError(err error) bool
- func LimitedCopy(fs FS, oldname, newname string, maxBytes int64) error
- func LinkOrCopy(fs FS, oldname, newname string) error
- type CloneOption
- type DiskSlowInfo
- type DiskUsage
- type FS
- type File
- type LogFn
- type MemFS
- func (y *MemFS) Create(fullname string) (File, error)
- func (*MemFS) GetDiskUsage(string) (DiskUsage, error)
- func (y *MemFS) Link(oldname, newname string) error
- func (y *MemFS) List(dirname string) ([]string, error)
- func (y *MemFS) Lock(fullname string) (io.Closer, error)
- func (y *MemFS) MkdirAll(dirname string, perm os.FileMode) error
- func (y *MemFS) Open(fullname string, opts ...OpenOption) (File, error)
- func (y *MemFS) OpenDir(fullname string) (File, error)
- func (y *MemFS) OpenReadWrite(fullname string, opts ...OpenOption) (File, error)
- func (*MemFS) PathBase(p string) string
- func (*MemFS) PathDir(p string) string
- func (*MemFS) PathJoin(elem ...string) string
- func (y *MemFS) Remove(fullname string) error
- func (y *MemFS) RemoveAll(fullname string) error
- func (y *MemFS) Rename(oldname, newname string) error
- func (y *MemFS) ResetToSyncedState()
- func (y *MemFS) ReuseForWrite(oldname, newname string) (File, error)
- func (y *MemFS) SetIgnoreSyncs(ignoreSyncs bool)
- func (y *MemFS) Stat(name string) (os.FileInfo, error)
- func (y *MemFS) String() string
- func (y *MemFS) UseWindowsSemantics(windowsSemantics bool)
- type OpType
- type OpenOption
- type SyncingFileOptions
Constants ¶
const InvalidFd uintptr = ^(uintptr(0))
InvalidFd is a special value returned by File.Fd() when the file is not backed by an OS descriptor. Note: the special value is consistent with what os.File implementation returns on a nil receiver.
Variables ¶
var ErrUnsupported = errors.New("pebble: not supported")
ErrUnsupported may be returned a FS when it does not support an operation.
Functions ¶
func Clone ¶
func Clone(srcFS, dstFS FS, srcPath, dstPath string, opts ...CloneOption) (bool, error)
Clone recursively copies a directory structure from srcFS to dstFS. srcPath specifies the path in srcFS to copy from and must be compatible with the srcFS path format. dstDir is the target directory in dstFS and must be compatible with the dstFS path format. Returns (true,nil) on a successful copy, (false,nil) if srcPath does not exist, and (false,err) if an error occurred.
func Copy ¶
Copy copies the contents of oldname to newname. If newname exists, it will be overwritten.
func CopyAcrossFS ¶
CopyAcrossFS copies the contents of oldname on srcFS to newname dstFS. If newname exists, it will be overwritten.
func IsNoSpaceError ¶
IsNoSpaceError returns true if the given error indicates that the disk is out of space.
func LimitedCopy ¶
LimitedCopy copies up to maxBytes from oldname to newname. If newname exists, it will be overwritten.
func LinkOrCopy ¶
LinkOrCopy creates newname as a hard link to the oldname file. If creating the hard link fails, LinkOrCopy falls back to copying the file (which may also fail if oldname doesn't exist or newname already exists).
Types ¶
type CloneOption ¶
type CloneOption func(*cloneOpts)
A CloneOption configures the behavior of Clone.
var CloneSync CloneOption = func(o *cloneOpts) { o.sync = true }
CloneSync configures Clone to sync files and directories.
var CloneTryLink CloneOption = func(o *cloneOpts) { o.tryLink = true }
CloneTryLink configures Clone to link files to the destination if the source and destination filesystems are the same. If the source and destination filesystems are not the same or the filesystem does not support linking, then Clone falls back to copying.
func CloneSkip ¶
func CloneSkip(fn func(string) bool) CloneOption
CloneSkip configures Clone to skip files for which the provided function returns true when passed the file's path.
type DiskSlowInfo ¶
type DiskSlowInfo struct { // Path of file being written to. Path string // Operation being performed on the file. OpType OpType // Size of write in bytes, if the write is sized. WriteSize int // Duration that has elapsed since this disk operation started. Duration time.Duration }
DiskSlowInfo captures info about detected slow operations on the vfs.
func (DiskSlowInfo) SafeFormat ¶
func (i DiskSlowInfo) SafeFormat(w redact.SafePrinter, _ rune)
SafeFormat implements redact.SafeFormatter.
func (DiskSlowInfo) String ¶
func (i DiskSlowInfo) String() string
type DiskUsage ¶
type DiskUsage struct { // Total disk space available to the current process in bytes. AvailBytes uint64 // Total disk space in bytes. TotalBytes uint64 // Used disk space in bytes. UsedBytes uint64 }
DiskUsage summarizes disk space usage on a filesystem.
type FS ¶
type FS interface { // Create creates the named file for reading and writing. If a file // already exists at the provided name, it's removed first ensuring the // resulting file descriptor points to a new inode. Create(name string) (File, error) // Link creates newname as a hard link to the oldname file. Link(oldname, newname string) error // Open opens the named file for reading. openOptions provides Open(name string, opts ...OpenOption) (File, error) // OpenReadWrite opens the named file for reading and writing. If the file // does not exist, it is created. OpenReadWrite(name string, opts ...OpenOption) (File, error) // OpenDir opens the named directory for syncing. OpenDir(name string) (File, error) // Remove removes the named file or directory. Remove(name string) error // Remove removes the named file or directory and any children it // contains. It removes everything it can but returns the first error it // encounters. RemoveAll(name string) error // Rename renames a file. It overwrites the file at newname if one exists, // the same as os.Rename. Rename(oldname, newname string) error // ReuseForWrite attempts to reuse the file with oldname by renaming it to newname and opening // it for writing without truncation. It is acceptable for the implementation to choose not // to reuse oldname, and simply create the file with newname -- in this case the implementation // should delete oldname. If the caller calls this function with an oldname that does not exist, // the implementation may return an error. ReuseForWrite(oldname, newname string) (File, error) // MkdirAll creates a directory and all necessary parents. The permission // bits perm have the same semantics as in os.MkdirAll. If the directory // already exists, MkdirAll does nothing and returns nil. MkdirAll(dir string, perm os.FileMode) error // Lock locks the given file, creating the file if necessary, and // truncating the file if it already exists. The lock is an exclusive lock // (a write lock), but locked files should neither be read from nor written // to. Such files should have zero size and only exist to co-ordinate // ownership across processes. // // A nil Closer is returned if an error occurred. Otherwise, close that // Closer to release the lock. // // On Linux and OSX, a lock has the same semantics as fcntl(2)'s advisory // locks. In particular, closing any other file descriptor for the same // file will release the lock prematurely. // // Attempting to lock a file that is already locked by the current process // returns an error and leaves the existing lock untouched. // // Lock is not yet implemented on other operating systems, and calling it // will return an error. Lock(name string) (io.Closer, error) // List returns a listing of the given directory. The names returned are // relative to dir. List(dir string) ([]string, error) // Stat returns an os.FileInfo describing the named file. Stat(name string) (os.FileInfo, error) // PathBase returns the last element of path. Trailing path separators are // removed before extracting the last element. If the path is empty, PathBase // returns ".". If the path consists entirely of separators, PathBase returns a // single separator. PathBase(path string) string // PathJoin joins any number of path elements into a single path, adding a // separator if necessary. PathJoin(elem ...string) string // PathDir returns all but the last element of path, typically the path's directory. PathDir(path string) string // GetDiskUsage returns disk space statistics for the filesystem where // path is any file or directory within that filesystem. GetDiskUsage(path string) (DiskUsage, error) }
FS is a namespace for files.
The names are filepath names: they may be / separated or \ separated, depending on the underlying operating system.
var Default FS = defaultFS{}
Default is a FS implementation backed by the underlying operating system's file system.
func NewSyncingFS ¶
func NewSyncingFS(fs FS, syncOpts SyncingFileOptions) FS
NewSyncingFS wraps a vfs.FS with one that wraps newly created files with vfs.NewSyncingFile.
func OnDiskFull ¶
OnDiskFull wraps the provided FS with an FS that examines returned errors, looking for ENOSPC errors. It invokes the provided callback when the underlying filesystem returns an error signifying the storage is out of disk space.
All new writes to the filesystem are blocked while the callback executes, so care must be taken to avoid expensive work from within the callback.
Once the callback completes, any write-oriented operations that encountered ENOSPC are retried exactly once. Once the callback completes, it will not be invoked again until a new operation that began after the callback returned encounters an ENOSPC error.
OnDiskFull may be used to automatically manage a ballast file, which is removed from the filesystem from within the callback. Note that if managing a ballast, the caller should maintain a reference to the inner FS and remove the ballast on the unwrapped FS.
func Root ¶
Root returns the base FS implementation, unwrapping all nested FSs that expose an Unwrap method.
func WithDiskHealthChecks ¶
func WithDiskHealthChecks( innerFS FS, diskSlowThreshold time.Duration, onSlowDisk func(info DiskSlowInfo), ) (FS, io.Closer)
WithDiskHealthChecks wraps an FS and ensures that all write-oriented operations on the FS are wrapped with disk health detection checks. Disk operations that are observed to take longer than diskSlowThreshold trigger an onSlowDisk call.
A threshold of zero disables disk-health checking.
func WithLogging ¶
WithLogging wraps an FS and logs filesystem modification operations to the given logFn.
type File ¶
type File interface { io.Closer io.Reader io.ReaderAt // Unlike the specification for io.Writer.Write(), the vfs.File.Write() // method *is* allowed to modify the slice passed in, whether temporarily // or permanently. Callers of Write() need to take this into account. io.Writer // WriteAt() is only supported for files that were opened with FS.OpenReadWrite. io.WriterAt // Preallocate optionally preallocates storage for `length` at `offset` // within the file. Implementations may choose to do nothing. Preallocate(offset, length int64) error Stat() (os.FileInfo, error) Sync() error // SyncTo requests that a prefix of the file's data be synced to stable // storage. The caller passes provides a `length`, indicating how many bytes // to sync from the beginning of the file. SyncTo is a no-op for // directories, and therefore always returns false. // // SyncTo returns a fullSync return value, indicating one of two possible // outcomes. // // If fullSync is false, the first `length` bytes of the file was queued to // be synced to stable storage. The syncing of the file prefix may happen // asynchronously. No persistence guarantee is provided. // // If fullSync is true, the entirety of the file's contents were // synchronously synced to stable storage, and a persistence guarantee is // provided. In this outcome, any modified metadata for the file is not // guaranteed to be synced unless that metadata is needed in order to allow // a subsequent data retrieval to be correctly handled. SyncTo(length int64) (fullSync bool, err error) // SyncData requires that all written data be persisted. File metadata is // not required to be synced. Unsophisticated implementations may call Sync. SyncData() error // Prefetch signals the OS (on supported platforms) to fetch the next length // bytes in file (as returned by os.File.Fd()) after offset into cache. Any // subsequent reads in that range will not issue disk IO. Prefetch(offset int64, length int64) error // Fd returns the raw file descriptor when a File is backed by an *os.File. // It can be used for specific functionality like Prefetch. // Returns InvalidFd if not supported. Fd() uintptr }
File is a readable, writable sequence of bytes.
Typically, it will be an *os.File, but test code may choose to substitute memory-backed implementations.
Write-oriented operations (Write, Sync) must be called sequentially: At most 1 call to Write or Sync may be executed at any given time.
func NewMemFile ¶
NewMemFile returns a memory-backed File implementation. The memory-backed file takes ownership of data.
func NewSyncingFile ¶
func NewSyncingFile(f File, opts SyncingFileOptions) File
NewSyncingFile wraps a writable file and ensures that data is synced periodically as it is written. The syncing does not provide persistency guarantees for these periodic syncs, but is used to avoid latency spikes if the OS automatically decides to write out a large chunk of dirty filesystem buffers. The underlying file is fully synced upon close.
type LogFn ¶
type LogFn func(fmt string, args ...interface{})
LogFn is a function that is used to capture a log when WithLogging is used.
type MemFS ¶
type MemFS struct {
// contains filtered or unexported fields
}
MemFS implements FS.
func NewStrictMem ¶
func NewStrictMem() *MemFS
NewStrictMem returns a "strict" memory-backed FS implementation. The behaviour is strict wrt needing a Sync() call on files or directories for the state changes to be finalized. Any changes that are not finalized are visible to reads until MemFS.ResetToSyncedState() is called, at which point they are discarded and no longer visible.
Expected usage:
strictFS := NewStrictMem() db := Open(..., &Options{FS: strictFS}) // Do and commit various operations. ... // Prevent any more changes to finalized state. strictFS.SetIgnoreSyncs(true) // This will finish any ongoing background flushes, compactions but none of these writes will // be finalized since syncs are being ignored. db.Close() // Discard unsynced state. strictFS.ResetToSyncedState() // Allow changes to finalized state. strictFS.SetIgnoreSyncs(false) // Open the DB. This DB should have the same state as if the earlier strictFS operations and // db.Close() were not called. db := Open(..., &Options{FS: strictFS})
func (*MemFS) GetDiskUsage ¶
GetDiskUsage implements FS.GetDiskUsage.
func (*MemFS) Open ¶
func (y *MemFS) Open(fullname string, opts ...OpenOption) (File, error)
Open implements FS.Open.
func (*MemFS) OpenReadWrite ¶ added in v1.1.0
func (y *MemFS) OpenReadWrite(fullname string, opts ...OpenOption) (File, error)
OpenReadWrite implements FS.OpenReadWrite.
func (*MemFS) ResetToSyncedState ¶
func (y *MemFS) ResetToSyncedState()
ResetToSyncedState discards state in the FS that is not synced. See the usage comment with NewStrictMem() for details.
func (*MemFS) ReuseForWrite ¶
ReuseForWrite implements FS.ReuseForWrite.
func (*MemFS) SetIgnoreSyncs ¶
SetIgnoreSyncs sets the MemFS.ignoreSyncs field. See the usage comment with NewStrictMem() for details.
func (*MemFS) UseWindowsSemantics ¶
UseWindowsSemantics configures whether the MemFS implements Windows-style semantics, in particular with respect to whether any of an open file's links may be removed. Windows semantics default to off.
type OpType ¶
type OpType uint8
OpType is the type of IO operation being monitored by a diskHealthCheckingFile.
const ( OpTypeUnknown OpType = iota OpTypeWrite OpTypeSync OpTypeSyncData OpTypeSyncTo OpTypeCreate OpTypeLink OpTypeMkdirAll OpTypePreallocate OpTypeRemove OpTypeRemoveAll OpTypeRename OpTypeReuseForWrite )
The following OpTypes is limited to the subset of file system operations that a diskHealthCheckingFile supports (namely writes and syncs).
type OpenOption ¶
type OpenOption interface { // Apply is called on the file handle after it's opened. Apply(File) }
OpenOption provide an interface to do work on file handles in the Open() call.
var RandomReadsOption OpenOption = &randomReadsOption{}
RandomReadsOption is an OpenOption that optimizes opened file handle for random reads, by calling fadvise() with POSIX_FADV_RANDOM on Linux systems to disable readahead.
var SequentialReadsOption OpenOption = &sequentialReadsOption{}
SequentialReadsOption is an OpenOption that optimizes opened file handle for sequential reads, by calling fadvise() with POSIX_FADV_SEQUENTIAL on Linux systems to enable readahead.
type SyncingFileOptions ¶
type SyncingFileOptions struct { // NoSyncOnClose elides the automatic Sync during Close if it's not possible // to sync the remainder of the file in a non-blocking way. NoSyncOnClose bool BytesPerSync int PreallocateSize int }
SyncingFileOptions holds the options for a syncingFile.