fio

package module
v0.5.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 27, 2024 License: GPL-2.0 Imports: 17 Imported by: 1

README

go-fio - optimized file I/O routines

Description

This is a collection of cross platform file I/O and File system functions. These use concurrency to speed up the underlying functions.

  • fio.Info: a serializable version of Stat/Lstat that uses OS specific functions
  • Utilities to copy a file or dir along with all its metadata (including XATTR) The file copy functions will use the best underlying primitive like reflink(2); and fallback to using mmap based copy.
  • Safefile: a wrapper over os.OpenFile that atomically commits the writes. This uses a temporary file for all the I/O and calling Close() atomically renames the temporary file to the intended file.
Subpackages
  • cmp: compares two directory trees and returns their differences
  • clone: clones a source directory tree to a destination - skipping over identical files.
  • walk: A concurrent directory tree traversal library

Documentation

Index

Constants

View Source 
const (
	OPT_OVERWRITE uint32 = 1 << iota
)

Variables

View Source 
var ErrCompleted = errors.New("workpool: workpool closed")

Error returned if new work is submitted after Wait() or if Wait() is called multiple times.

View Source 
var ErrNotClosed = errors.New("workpool: workpool not closed before waiting")

Error returned if Wait() is called without closing the work submission

View Source 
var (
	ErrTooSmall = errors.New("buffer is not big enough")
)

Functions

func ClearXattr added in v0.2.0 

func ClearXattr(nm string) error

ClearXattr deletes all the extended attributes of a file.

func CopyFd 

func CopyFd(dst, src *os.File) error

CopyFd copies open files 'src' to 'dst' using the most efficient OS primitive available on the runtime platform. CopyFile will use copy-on-write facailities if the underlying file-system implements it. It will fallback to copying via memory mapping 'src' and writing the blocks to 'dst'.

func CopyFile 

func CopyFile(dst, src string, perm fs.FileMode) error

CopyFile copies files 'src' to 'dst' using the most efficient OS primitive available on the runtime platform. CopyFile will use copy-on-write facilities if the underlying file-system implements it. It will fallback to copying via memory mapping 'src' and writing the blocks to 'dst'.

func DelXattr added in v0.2.0 

func DelXattr(nm string, keys ...string) error

DelXattr deletes one or more extended attributes of a file.

func LclearXattr added in v0.2.0 

func LclearXattr(nm string) error

ClearXattr deletes all the extended attributes of a file. If 'nm' points to a symlink, LSetXattr will delete the extended attributes of the symlink and *not* the target.

func LdelXattr added in v0.2.0 

func LdelXattr(nm string, keys ...string) error

LDelXattr deletes one or more extended attributes of a file. If 'nm' points to a symlink, LSetXattr will delete the extended attributes of the symlink and *not* the target.

func LreplaceXattr added in v0.2.0 

func LreplaceXattr(nm string, x Xattr) error

LReplaceXattr replaces all the extended attributes of 'nm' with new attributes in 'x'. This function is a combination of LclearXattr() and LsetXattr(). If 'nm' points to a symlink, LReplaceXattr will set/update the extended attributes of the symlink and *not* the target.

func LsetXattr added in v0.2.0 

func LsetXattr(nm string, x Xattr) error

LSetXattr sets/updates the xattr list for a given file. If 'nm' points to a symlink, LSetXattr will set/update the extended attributes of the symlink and *not* the target.

func Lstatm added in v0.2.0 

func Lstatm(nm string, fi *Info) error

Lstatm is like Lstat except it uses the caller's supplied memory.

func ReplaceXattr added in v0.2.0 

func ReplaceXattr(nm string, x Xattr) error

ReplaceXattr replaces all the extended attributes of 'nm' with new attributes in 'x'. This function is a combination of ClearXattr() and SetXattr().

func SetXattr added in v0.2.0 

func SetXattr(nm string, x Xattr) error

SetXattr sets/updates the xattr list for a given file.

func Statm added in v0.2.0 

func Statm(nm string, fi *Info) error

Statm is like Stat above - except it uses caller supplied memory for the stat(2) info

Types

type CopyError added in v0.4.0 

type CopyError struct {
	Op  string
	Src string
	Dst string
	Err error
}

CopyError represents the errors returned by CopyFile and CopyFd

func (*CopyError) Error added in v0.4.0 

func (e *CopyError) Error() string

Error returns a string representation of CopyError

func (*CopyError) Unwrap added in v0.4.0 

func (e *CopyError) Unwrap() error

Unwrap returns the underlying wrapped error

type FioMap added in v0.5.3 

type FioMap = xsync.MapOf[string, *Info]

FioMap is a concurrency safe map of relative path name and the corresponding Stat/Lstat info.

func NewFioMap added in v0.5.3 

func NewFioMap() *FioMap

type FioPairMap added in v0.5.3 

type FioPairMap = xsync.MapOf[string, Pair]

FioPairMap is a concurrency safe map of relative path name and the corresponding Stat/Lstat info of both the source and destination.

func NewFioPairMap added in v0.5.3 

func NewFioPairMap() *FioPairMap

type Info added in v0.2.0 

type Info struct {
	Ino  uint64
	Siz  int64
	Dev  uint64
	Rdev uint64

	Mod   fs.FileMode
	Uid   uint32
	Gid   uint32
	Nlink uint32

	Atim time.Time
	Mtim time.Time
	Ctim time.Time

	Nam   string
	Xattr Xattr
}

Info represents a file/dir metadata in a normalized form It satisfies the fs.FileInfo interface and notably supports extended file system attributes (`xattr(7)`). This type can also be safely marshaled and unmarshaled into a portable byte stream.

func Lstat added in v0.2.0 

func Lstat(nm string) (*Info, error)

Lstat is like os.Lstat() but also returns xattr

func Stat added in v0.2.0 

func Stat(nm string) (*Info, error)

Stat is like os.Stat() but also returns xattr

func (*Info) IsDir added in v0.2.0 

func (ii *Info) IsDir() bool

IsDir returns true if this Info represents a directory entry

func (*Info) IsRegular added in v0.2.0 

func (ii *Info) IsRegular() bool

IsRegular returns true if this Info represents a regular file

func (*Info) IsSameFS added in v0.3.2 

func (a *Info) IsSameFS(b *Info) bool

IsSameFs returns true if a and b represent file entries on the same file system

func (*Info) Marshal added in v0.5.4 

func (ii *Info) Marshal() ([]byte, error)

Marshal marshals 'ii' into a correctly sized buffer and returns it

func (*Info) MarshalSize added in v0.2.0 

func (ii *Info) MarshalSize() int

MarshalSize returns the marshaled size of _this_ instance of Info

func (*Info) MarshalTo added in v0.2.0 

func (ii *Info) MarshalTo(b []byte) (int, error)

MarshalTo marshals 'ii' into the provided buffer 'b'. The buffer 'b' is expected to be sufficiently big to hold the marshaled data. It returns the number of marshaled bytes (ie exactly the value returned by the corresponding MarshalSize()).

func (*Info) ModTime added in v0.2.0 

func (ii *Info) ModTime() time.Time

ModTime returns the file modification time

func (*Info) Mode added in v0.2.0 

func (ii *Info) Mode() fs.FileMode

Mode returns the file mode bits

func (*Info) Name added in v0.2.0 

func (ii *Info) Name() string

Name satisfies fs.FileInfo and returns the name of the fs entry.

func (*Info) Size added in v0.2.0 

func (ii *Info) Size() int64

Size returns the fs entry's size

func (*Info) String added in v0.2.0 

func (ii *Info) String() string

String is a string representation of Info

func (*Info) Sys added in v0.2.0 

func (ii *Info) Sys() any

Sys returns the platform specific info - in our case it returns a pointer to the underlying Info instance.

func (*Info) Unmarshal added in v0.2.0 

func (ii *Info) Unmarshal(b []byte) (int, error)

Unmarshal unmarshals the byte stream 'b' into a full rehydrated instance 'ii'. It returns the number of bytes consumed.

type Pair added in v0.5.3 

type Pair struct {
	Src, Dst *Info
}

Pair represents the Stat/Lstat info of a pair of related file system entries in the source and destination

type SafeFile 

type SafeFile struct {
	*os.File
	// contains filtered or unexported fields
}

SafeFile is an io.WriteCloser which uses a temporary file that will be atomically renamed when there are no errors and caller invokes Close(). The recommended usage is: 

sf, err := NewSafeFile(...)
... error handling

defer sf.Abort()

... write to sf ..
sf.Close()

It is safe to call Abort on a closed SafeFile; the first call to Close() or Abort() seals the outcome. Similarly, it is safe to call Close() after Abort() - the first call to either takes precedence.

func NewSafeFile 

func NewSafeFile(nm string, opts uint32, flag int, perm os.FileMode) (*SafeFile, error)

NewSafeFile creates a new temporary file that would either be aborted or safely renamed to the correct name. 'nm' is the name of the final file; if 'ovwrite' is true, then the file is overwritten if it exists.

func (*SafeFile) Abort 

func (sf *SafeFile) Abort()

Abort the file write and remove any temporary artifacts; it is safe to call Close() on a different code path; the first call to Abort() or Close() takes precedence.

func (*SafeFile) Close 

func (sf *SafeFile) Close() error

Close flushes all file data & metadata to disk, closes the file and atomically renames the temp file to the actual file - ONLY if there were no intervening errors.

func (*SafeFile) Write 

func (sf *SafeFile) Write(b []byte) (int, error)

Attempt to write everything in 'b' and don't proceed if there was a previous error or the file was already closed.

func (*SafeFile) WriteAt 

func (sf *SafeFile) WriteAt(b []byte, off int64) (int, error)

WriteAt writes 'b' at absolute offset 'off'

type WorkPool added in v0.4.0 

type WorkPool[Work any] struct {
	// contains filtered or unexported fields
}

func NewWorkPool added in v0.4.0 

func NewWorkPool[Work any](nworkers int, fp func(i int, w Work) error) *WorkPool[Work]

NewWorkPool creates a worker pool that invokes caller provided worker 'fp'. Each worker will process one unit of "work" submitted via Submit().

func (*WorkPool[Work]) Close added in v0.4.0 

func (wp *WorkPool[Work]) Close()

Close the work submission to workers and signal to them that there's no more work forthcoming.

func (*WorkPool[Work]) Err added in v0.4.0 

func (wp *WorkPool[Work]) Err(err error)

Submit an error to the pool - if the user provided worker does things asynchronously.

func (*WorkPool[Work]) Submit added in v0.4.0 

func (wp *WorkPool[Work]) Submit(w Work)

Submit submits one unit of work to the worker WorkPool must be active.

func (*WorkPool[Work]) Wait added in v0.4.0 

func (wp *WorkPool[Work]) Wait() error

Wait closes the work channel and waits for all workers to end. Returns any errors from the workers. It is an error to call this multiple times

type Xattr 

type Xattr map[string]string

Xattr is a collection of all the extended attributes of a given file

func GetXattr added in v0.2.0 

func GetXattr(nm string) (Xattr, error)

GetXattr returns all the extended attributes of a file. This function will traverse symlinks.

func LgetXattr added in v0.2.0 

func LgetXattr(nm string) (Xattr, error)

LGetXattr returns all the extended attributes of a file. If 'nm' points to a symlink, LGetXattr will return the extended attributes of the symlink and *not* the target.

func (Xattr) Equal added in v0.2.0 

func (x Xattr) Equal(y Xattr) bool

Equal returns true if all xattr of 'x' is the same as all the xattr of 'y' and returns false otherwise.

func (Xattr) String 

func (x Xattr) String() string

String returns the string representation of all the extended attributes

Source Files

View all Source files

Directories

Path Synopsis
Package walk does a concurrent file system traversal and returns each entry.
 Package walk does a concurrent file system traversal and returns each entry.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.