walk

README

walk

Fast parallel version of golang filepath.Walk()

Performs traversals in parallel so set GOMAXPROCS appropriately. Vaues of 8 to 16 seem to work best on my 4-CPU plus 4 SMT pseudo-CPU MacBookPro. The result is about 4x-6x the traversal rate of the standard Walk(). The two are not identical since we are walking the file system in a tumult of asynchronous walkFunc calls by a number of goroutines. So, take note of the following:

1. This walk honors all of the walkFunc error semantics but as multiple user-supplied walkFuncs may simultaneously encounter a traversal error or generate one to stop traversal, only the FIRST of these will be returned as the Walk() result.

2. Further, since there may be a few files in flight at the instant of error discovery, a few more walkFunc calls may happen after the first error-generating call has signaled its desire to stop. In general this is a non-issue but it could matter so pay attention when designing your walkFunc. (For example, if you accumulate results then you need to have your own means to know to stop accumulating once you signal an error.)

3. Because the walkFunc is called concurrently in multiple goroutines, it needs to be careful about what it does with external data to avoid collisions. Results may be printed using fmt, but generally the best plan is to send results over a channel or accumulate counts using a locked mutex.

These issues are illustrated/handled in the simple traversal programs supplied with walk. There is also a test file that is just the tests from filepath in the Go language's standard library. Walk passes these tests when run in single process mode, and passes most of them in concurrent mode (GOMAXPROCS > 1). The problem is not a real problem, but one of the test expecting a specific number of errors to be found based on presumed sequential traversals.

Copyright (c) 2016 Michael T Jones

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Documentation

Overview

Package filepath implements utility routines for manipulating filename paths in a way compatible with the target operating system-defined file paths.

Index

• Constants
• Variables
• func Clean(path string) string
• func EvalSymlinks(path string) (string, error)
• func FromSlash(path string) string
• func HasPrefix(p, prefix string) bool
• func IsAbs(path string) bool
• func Join(elem ...string) string
• func Rel(basepath, targpath string) (string, error)
• func ToSlash(path string) string
• func VolumeName(path string) (v string)
• func Walk(root string, walkFn WalkFunc) error
• type VisitData
• type WalkFunc
• type WalkState

Constants

const (
	Separator     = os.PathSeparator
	ListSeparator = os.PathListSeparator
)

Variables

var LstatP = &lstat

var SkipDir = errors.New("skip this directory")

SkipDir is used as a return value from WalkFuncs to indicate that the directory named in the call is to be skipped. It is not returned as an error by any function.

Functions

func Clean

func Clean(path string) string

Clean returns the shortest path name equivalent to path by purely lexical processing. It applies the following rules iteratively until no further processing can be done:

1. Replace multiple Separator elements with a single one.
2. Eliminate each . path name element (the current directory).
3. Eliminate each inner .. path name element (the parent directory) along with the non-.. element that precedes it.
4. Eliminate .. elements that begin a rooted path: that is, replace "/.." by "/" at the beginning of a path, assuming Separator is '/'.

The returned path ends in a slash only if it represents a root directory, such as "/" on Unix or `C:\` on Windows.

If the result of this process is an empty string, Clean returns the string ".".

See also Rob Pike, “Lexical File Names in Plan 9 or Getting Dot-Dot Right,” http://plan9.bell-labs.com/sys/doc/lexnames.html

func EvalSymlinks

func EvalSymlinks(path string) (string, error)

EvalSymlinks returns the path name after the evaluation of any symbolic links. If path is relative the result will be relative to the current directory, unless one of the components is an absolute symbolic link.

func FromSlash

func FromSlash(path string) string

FromSlash returns the result of replacing each slash ('/') character in path with a separator character. Multiple slashes are replaced by multiple separators.

func HasPrefix

func HasPrefix(p, prefix string) bool

HasPrefix exists for historical compatibility and should not be used.

func IsAbs

func IsAbs(path string) bool

IsAbs returns true if the path is absolute.

func Join

func Join(elem ...string) string

Join joins any number of path elements into a single path, adding a Separator if necessary. The result is Cleaned, in particular all empty strings are ignored.

func Rel

func Rel(basepath, targpath string) (string, error)

Rel returns a relative path that is lexically equivalent to targpath when joined to basepath with an intervening separator. That is, Join(basepath, Rel(basepath, targpath)) is equivalent to targpath itself. On success, the returned path will always be relative to basepath, even if basepath and targpath share no elements. An error is returned if targpath can't be made relative to basepath or if knowing the current working directory would be necessary to compute it.

func ToSlash

func ToSlash(path string) string

ToSlash returns the result of replacing each separator character in path with a slash ('/') character. Multiple separators are replaced by multiple slashes.

func VolumeName

func VolumeName(path string) (v string)

VolumeName returns leading volume name. Given "C:\foo\bar" it returns "C:" under windows. Given "\\host\share\foo" it returns "\\host\share". On other platforms it returns "".

func Walk

func Walk(root string, walkFn WalkFunc) error

Types

type VisitData

type VisitData struct {
	// contains filtered or unexported fields
}

type WalkFunc

type WalkFunc func(path string, info os.FileInfo, err error) error

WalkFunc is the type of the function called for each file or directory visited by Walk. The path argument contains the argument to Walk as a prefix; that is, if Walk is called with "dir", which is a directory containing the file "a", the walk function will be called with argument "dir/a". The info argument is the os.FileInfo for the named path.

If there was a problem walking to the file or directory named by path, the incoming error will describe the problem and the function can decide how to handle that error (and Walk will not descend into that directory). If an error is returned, processing stops. The sole exception is that if path is a directory and the function returns the special value SkipDir, the contents of the directory are skipped and processing continues as usual on the next file.

type WalkState

type WalkState struct {
	// contains filtered or unexported fields
}