pwalk

README

pwalk: parallel implementation of filepath.Walk

This is a wrapper for filepath.Walk which may speed it up by calling multiple callback functions (WalkFunc) in parallel, utilizing goroutines.

By default, it utilizes 2*runtime.NumCPU() goroutines for callbacks. This can be changed by using WalkN function which has the additional parameter, specifying the number of goroutines (concurrency).

Caveats

Please note the following limitations of this code:

• Unlike filepath.Walk, the order of calls is non-deterministic;

• Only primitive error handling is supported:

  • filepath.SkipDir is not supported;

  • no errors are ever passed to WalkFunc;

  • once any error is returned from any WalkFunc instance, no more new calls to WalkFunc are made, and the error is returned to the caller of Walk;

  • if more than one walkFunc instance will return an error, only one of such errors will be propagated and returned by Walk, others will be silently discarded.

Documentation

For the official documentation, see https://pkg.go.dev/github.com/opencontainers/selinux/pkg/pwalk?tab=doc

Benchmarks

For a WalkFunc that consists solely of the return statement, this implementation is about 10% slower than the standard library's filepath.Walk.

Otherwise (if a WalkFunc is doing something) this is usually faster, except when the WalkN(..., 1) is used.

Documentation

Index

• func Walk(root string, walkFn WalkFunc) error
• func WalkN(root string, walkFn WalkFunc, num int) error
• type WalkFunc

Functions

func Walk

func Walk(root string, walkFn WalkFunc) error

Walk is a wrapper for filepath.Walk which can call multiple walkFn in parallel, allowing to handle each item concurrently. A maximum of twice the runtime.NumCPU() walkFn will be called at any one time. If you want to change the maximum, use WalkN instead.

The order of calls is non-deterministic.

Note that this implementation only supports primitive error handling:

- no errors are ever passed to WalkFn;

- once a walkFn returns any error, all further processing stops and the error is returned to the caller of Walk;

- filepath.SkipDir is not supported;

- if more than one walkFn instance will return an error, only one of such errors will be propagated and returned by Walk, others will be silently discarded.

func WalkN

func WalkN(root string, walkFn WalkFunc, num int) error

WalkN is a wrapper for filepath.Walk which can call multiple walkFn in parallel, allowing to handle each item concurrently. A maximum of num walkFn will be called at any one time.

Please see Walk documentation for caveats of using this function.

Types

type WalkFunc

type WalkFunc = filepath.WalkFunc