pwalkdir

README

pwalkdir: parallel implementation of filepath.WalkDir

This is a wrapper for filepath.WalkDir which may speed it up by calling multiple callback functions (WalkDirFunc) in parallel, utilizing goroutines.

By default, it utilizes 2*runtime.NumCPU() goroutines for callbacks. This can be changed by using WalkN function which has the additional parameter, specifying the number of goroutines (concurrency).

pwalk vs pwalkdir

This package is very similar to pwalk, but utilizes filepath.WalkDir (added to Go 1.16), which does not call stat(2) on every entry and is therefore faster (up to 3x, depending on usage scenario).

Users who are OK with requiring Go 1.16+ should switch to this implementation.

Caveats

Please note the following limitations of this code:

• Unlike filepath.WalkDir, the order of calls is non-deterministic;

• Only primitive error handling is supported:

  • fs.SkipDir is not supported;

  • no errors are ever passed to WalkDirFunc;

  • once any error is returned from any walkDirFunc instance, no more calls to WalkDirFunc are made, and the error is returned to the caller of WalkDir;

  • if more than one WalkDirFunc instance will return an error, only one of such errors will be propagated to and returned by WalkDir, others will be silently discarded.

Documentation

For the official documentation, see https://pkg.go.dev/github.com/opencontainers/selinux/pkg/pwalkdir

Benchmarks

For a WalkDirFunc that consists solely of the return statement, this implementation is about 15% slower than the standard library's filepath.WalkDir.

Otherwise (if a WalkDirFunc is actually doing something) this is usually faster, except when the WalkDirN(..., 1) is used. Run go test -bench . to see how different operations can benefit from it, as well as how the level of paralellism affects the speed.

Documentation

Index

• func Walk(root string, walkFn fs.WalkDirFunc) error
• func WalkN(root string, walkFn fs.WalkDirFunc, num int) error

Functions

func Walk

func Walk(root string, walkFn fs.WalkDirFunc) error

Walk is a wrapper for filepath.WalkDir which can call multiple walkFn in parallel, allowing to handle each item concurrently. A maximum of twice the runtime.NumCPU() walkFn will be called at any one time. If you want to change the maximum, use WalkN instead.

The order of calls is non-deterministic.

Note that this implementation only supports primitive error handling:

- no errors are ever passed to walkFn;

- once a walkFn returns any error, all further processing stops and the error is returned to the caller of Walk;

- filepath.SkipDir is not supported;

- if more than one walkFn instance will return an error, only one of such errors will be propagated and returned by Walk, others will be silently discarded.

func WalkN

func WalkN(root string, walkFn fs.WalkDirFunc, num int) error

WalkN is a wrapper for filepath.WalkDir which can call multiple walkFn in parallel, allowing to handle each item concurrently. A maximum of num walkFn will be called at any one time.

Please see Walk documentation for caveats of using this function.