Documentation ¶
Overview ¶
Package fastwalk provides a faster version of filepath.Walk for file system scanning tools.
Index ¶
- Variables
- func DefaultNumWorkers() int
- func IgnoreDuplicateDirs(walkFn fs.WalkDirFunc) fs.WalkDirFunc
- func IgnoreDuplicateFiles(walkFn fs.WalkDirFunc) fs.WalkDirFunc
- func IgnorePermissionErrors(walkFn fs.WalkDirFunc) fs.WalkDirFunc
- func StatDirEntry(path string, de fs.DirEntry) (fs.FileInfo, error)
- func Walk(conf *Config, root string, walkFn fs.WalkDirFunc) error
- type Config
- type DirEntry
- type EntryFilter
Constants ¶
This section is empty.
Variables ¶
var DefaultConfig = Config{ Follow: false, NumWorkers: DefaultNumWorkers(), }
DefaultConfig is the default Config used when none is supplied.
var ErrSkipFiles = errors.New("fastwalk: skip remaining files in directory")
ErrSkipFiles is a used as a return value from WalkFuncs to indicate that the callback should not be called for any other files in the current directory. Child directories will still be traversed.
var ErrTraverseLink = errors.New("fastwalk: traverse symlink, assuming target is a directory")
ErrTraverseLink is used as a return value from WalkFuncs to indicate that the symlink named in the call may be traversed.
var SkipDir = fs.SkipDir
SkipDir is used as a return value from WalkDirFuncs to indicate that the directory named in the call is to be skipped. It is not returned as an error by any function.
Functions ¶
func DefaultNumWorkers ¶
func DefaultNumWorkers() int
DefaultNumWorkers returns the default number of worker goroutines to use in fastwalk.Walk and is the value of runtime.GOMAXPROCS(-1) clamped to a range of 4 to 32 except on Darwin where it is either 4 (8 cores or less) or 6 (more than 8 cores). This is because Walk / IO performance on Darwin degrades with more concurrency.
The optimal number for your workload may be lower or higher. The results of BenchmarkFastWalkNumWorkers benchmark may be informative.
func IgnoreDuplicateDirs ¶
func IgnoreDuplicateDirs(walkFn fs.WalkDirFunc) fs.WalkDirFunc
IgnoreDuplicateDirs wraps fs.WalkDirFunc walkFn to make it follow symbolic links and ignore duplicate directories (if a symlink points to a directory that has already been traversed it is skipped). The walkFn is called for for skipped directories, but the directory is not traversed (this is required for error handling).
The Config.Follow setting has no effect on the behavior of Walk when this wrapper is used.
In most use cases, the returned fs.WalkDirFunc should not be reused between in another call to Walk. If it is reused, any previously visited file will be skipped.
NOTE: The order of traversal is undefined. Given an "example" directory like the one below where "dir" is a directory and "smydir1" and "smydir2" are links to it, only one of "dir", "smydir1", or "smydir2" will be traversed, but which one is undefined.
example ├── dir ├── smydir1 -> dir └── smydir2 -> dir
func IgnoreDuplicateFiles ¶
func IgnoreDuplicateFiles(walkFn fs.WalkDirFunc) fs.WalkDirFunc
IgnoreDuplicateFiles wraps walkFn so that symlinks are followed and duplicate files are ignored. If a symlink resolves to a file that has already been visited it will be skipped.
In most use cases, the returned fs.WalkDirFunc should not be reused between in another call to Walk. If it is reused, any previously visited file will be skipped.
This can significantly slow Walk as os.Stat() is called for each path (on Windows, os.Stat() is only needed for symlinks).
func IgnorePermissionErrors ¶
func IgnorePermissionErrors(walkFn fs.WalkDirFunc) fs.WalkDirFunc
IgnorePermissionErrors wraps walkFn so that permission errors are ignored. The returned fs.WalkDirFunc may be reused.
func StatDirEntry ¶
StatDirEntry returns the fs.FileInfo for the file or subdirectory described by the entry. If the entry is a symbolic link, StatDirEntry returns the fs.FileInfo for the file the line references (os.Stat). If fs.DirEntry de is a fastwalk.DirEntry it's Stat() method is used and the returned fs.FileInfo may be a previously cached result.
func Walk ¶
func Walk(conf *Config, root string, walkFn fs.WalkDirFunc) error
Walk is a faster implementation of filepath.Walk.
filepath.Walk's design necessarily calls os.Lstat on each file, even if the caller needs less info. Many tools need only the type of each file. On some platforms, this information is provided directly by the readdir system call, avoiding the need to stat each file individually. fastwalk_unix.go contains a fork of the syscall routines.
See golang.org/issue/16399
Walk walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root.
If walkFn returns filepath.SkipDir, the directory is skipped.
Unlike filepath.WalkDir:
- File stat calls must be done by the user and should be done via the DirEntry argument to walkFn since it caches the results of Stat and Lstat.
- The fs.DirEntry argument is always a fastwalk.DirEntry, which has a Stat() method that returns the result of calling os.Stat() on the file. The result of Stat() may be cached.
- Multiple goroutines stat the filesystem concurrently. The provided walkFn must be safe for concurrent use.
- Walk can follow symlinks if walkFn returns the ErrTraverseLink sentinel error. It is the walkFn's responsibility to prevent Walk from going into symlink cycles.
Types ¶
type Config ¶
type Config struct { // Follow symbolic links ignoring directories that would lead // to infinite loops; that is, entering a previously visited // directory that is an ancestor of the last file encountered. // // The sentinel error ErrTraverseLink is ignored when Follow // is true (this to prevent users from defeating the loop // detection logic), but SkipDir and ErrSkipFiles are still // respected. Follow bool // Number of parallel workers to use. If NumWorkers if ≤ 0 then // the greater of runtime.NumCPU() or 4 is used. NumWorkers int }
type DirEntry ¶
type DirEntry interface { fs.DirEntry // Stat returns the FileInfo for the file or subdirectory described // by the entry. The returned FileInfo may be from the time of the // original directory read or from the time of the call to Stat. // If the entry denotes a symbolic link, Stat reports the information // about the target itself, not the link. Stat() (fs.FileInfo, error) }
A DirEntry extends the fs.DirEntry interface to add a Stat() method that returns the result of calling os.Stat() on the underlying file. The results of Info() and Stat() are cached.
The fs.DirEntry argument passed to the fs.WalkDirFunc by Walk is always a DirEntry. The only exception is the root directory with with Walk is called.
type EntryFilter ¶
type EntryFilter struct {
// contains filtered or unexported fields
}
An EntryFilter keeps track of visited directory entries and can be used to detect and avoid symlink loops or processing the same file twice.
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
examples
|
|
fwfind
fwfind is a an example program that is similar to POSIX find, but faster and worse (it's an example).
|
fwfind is a an example program that is similar to POSIX find, but faster and worse (it's an example). |
fwwc
fwwc is a an example program that recursively walks directories and prints the number of lines in each file it encounters.
|
fwwc is a an example program that recursively walks directories and prints the number of lines in each file it encounters. |
internal
|
|
dirent
Package dirent parses raw syscall dirents
|
Package dirent parses raw syscall dirents |