gocodewalker

README

gocodewalker

Library to help with walking of code directories in Go.

The problem. You want to walk the directories of a code repository. You want to respect .gitignore and .ignore files, and some are nested. This library is the answer.

• Designed to walk code repositories or find the root of them.
• By default, respects both .gitignore and .ignore files (can be disabled) and nested ones for accuracy
• Has configurable options for skipping files based on regex, extension or general match
• Uses readdir to provide as fast as possible file walking

NB this was moved from go-code-walker due to the name being annoying and to ensure it has a unique package name. Should still be drop in replaceable so long as you refer to the new package name.

https://pkg.go.dev/github.com/boyter/gocodewalker

Package provides file operations specific to code repositories such as walking the file tree obeying .ignore and .gitignore files or looking for the root directory assuming already in a git project.

Example of usage,

fileListQueue := make(chan *gocodewalker.File, 100)

fileWalker := gocodewalker.NewFileWalker(".", fileListQueue)

// restrict to only process files that have the .go extension
fileWalker.AllowListExtensions = append(fileWalker.AllowListExtensions, "go")

// handle the errors by printing them out and then ignore
errorHandler := func(e error) bool {
    fmt.Println("ERR", e.Error())
    return true
}
fileWalker.SetErrorHandler(errorHandler)

go fileWalker.Start()

for f := range fileListQueue {
    fmt.Println(f.Location)
}

The above by default will recursively add files to the fileListQueue respecting both .ignore and .gitignore files if found, and only adding files with the go extension into the queue.

You can also run the walker in parallel with the results intermixed if required,

fileListQueue := make(chan *gocodewalker.File, 100)

fileWalker := gocodewalker.NewParallelFileWalker([]string{".", "someotherdir"}, fileListQueue)
go fileWalker.Start()

for f := range fileListQueue {
    fmt.Println(f.Location)
}

All code is licenced as MIT.

Error Handler

You can supply your own error handler when walking. This allows you to perform an action when there is an error and decide if the walker should continue to process, or return.

The simplest handler is the below, which if set will swallow all errors and continue as best it can.

errorHandler := func(e error) bool {
    return true
}
fileWalker.SetErrorHandler(errorHandler)

If you wanted to return on errors you could use the following.

errorHandler := func(e error) bool {
    return false
}
fileWalker.SetErrorHandler(errorHandler)

If you wanted to terminate walking on an error you could use the following, which would cause it to return the error, and then terminate all walking. This might be desirable where any error indicates a total failure.

errorHandler := func(e error) bool {
    fileWalker.Terminate()
    return false
}
fileWalker.SetErrorHandler(errorHandler)

Testing

Done through unit/integration tests. Otherwise see https://github.com/svent/gitignore-test

See ./cmd/gocodewalker/main.go for an example of how to implement and validate

Info

Details on how gitignores work

https://stackoverflow.com/questions/71735516/proper-way-to-setup-multiple-gitignore-files-in-nested-folders-of-a-repository

Documentation

Index

• Constants
• Variables
• func FindRepositoryRoot(startDirectory string) string
• func GetExtension(name string) string
• func IsHidden(file os.FileInfo, directory string) (bool, error)
• func IsHiddenDirEntry(file fs.DirEntry, directory string) (bool, error)
• type File
• type FileWalker
  • func NewFileWalker(directory string, fileListQueue chan *File) *FileWalker
  • func NewParallelFileWalker(directories []string, fileListQueue chan *File) *FileWalker
  • func (f *FileWalker) SetConcurrency(i int)
  • func (f *FileWalker) SetErrorHandler(errors func(error) bool)
  • func (f *FileWalker) Start() error
  • func (f *FileWalker) Terminate()
  • func (f *FileWalker) Walking() bool

Constants

const (
	GitIgnore  = ".gitignore"
	Ignore     = ".ignore"
	GitModules = ".gitmodules"
)

Variables

var ErrTerminateWalk = errors.New("gocodewalker terminated")

ErrTerminateWalk error which indicates that the walker was terminated

Functions

func FindRepositoryRoot

func FindRepositoryRoot(startDirectory string) string

FindRepositoryRoot given the supplied directory backwards looking for .git or .hg directories indicating we should start our search from that location as it's the root. Returns the first directory below supplied with .git or .hg in it otherwise the supplied directory

func GetExtension

func GetExtension(name string) string

GetExtension is a custom version of extracting extensions for a file which deals with extensions specific to code such as .travis.yml and the like

func IsHidden

func IsHidden(file os.FileInfo, directory string) (bool, error)

IsHidden Returns true if file is hidden

func IsHiddenDirEntry

func IsHiddenDirEntry(file fs.DirEntry, directory string) (bool, error)

IsHiddenDirEntry is similar to IsHidden, excepts it accepts fs.DirEntry as its argument

Types

type File

type File struct {
	Location string
	Filename string
}

File is a struct returned which contains the location and the filename of the file that passed all exclusion rules

type FileWalker

type FileWalker struct {
	LocationExcludePattern []string // Case-sensitive patterns which exclude directory/file matches
	IncludeDirectory       []string
	ExcludeDirectory       []string // Paths to always ignore such as .git,.svn and .hg
	IncludeFilename        []string
	ExcludeFilename        []string
	IncludeDirectoryRegex  []*regexp.Regexp // Must match regex as logical OR IE can match any of them
	ExcludeDirectoryRegex  []*regexp.Regexp
	IncludeFilenameRegex   []*regexp.Regexp
	ExcludeFilenameRegex   []*regexp.Regexp
	AllowListExtensions    []string // Which extensions should be allowed case sensitive
	ExcludeListExtensions  []string // Which extensions should be excluded case sensitive

	IgnoreIgnoreFile bool     // Should .ignore files be respected?
	IgnoreGitIgnore  bool     // Should .gitignore files be respected?
	IgnoreGitModules bool     // Should .gitmodules files be respected?
	CustomIgnore     []string // Custom ignore files
	IncludeHidden    bool     // Should hidden files and directories be included/walked

	MaxDepth int
	// contains filtered or unexported fields
}

func NewFileWalker

func NewFileWalker(directory string, fileListQueue chan *File) *FileWalker

NewFileWalker constructs a filewalker, which will walk the supplied directory and output File results to the supplied queue as it finds them

func NewParallelFileWalker

func NewParallelFileWalker(directories []string, fileListQueue chan *File) *FileWalker

NewParallelFileWalker constructs a filewalker, which will walk the supplied directories in parallel and output File results to the supplied queue as it finds them

func (*FileWalker) SetConcurrency

func (f *FileWalker) SetConcurrency(i int)

SetConcurrency sets the concurrency when walking which controls the number of goroutines that walk directories concurrently by default it is set to 8 must be a whole integer greater than 0

func (*FileWalker) SetErrorHandler

func (f *FileWalker) SetErrorHandler(errors func(error) bool)

SetErrorHandler sets the function that is called on processing any error where if you return true it will attempt to continue processing, and if false will return the error instantly

func (*FileWalker) Start

func (f *FileWalker) Start() error

Start will start walking the supplied directory with the supplied settings and putting files that mach into the supplied channel. Returns usual ioutil errors if there is a file issue and a ErrTerminateWalk if terminate is called while walking

func (*FileWalker) Terminate

func (f *FileWalker) Terminate()

Terminate have the walker break out of walking and return as soon as it possibly can. This is needed because this walker needs to work in a TUI interactive mode and as such we need to be able to end old processes

func (*FileWalker) Walking

func (f *FileWalker) Walking() bool

Walking gets the state of the file walker and determine if we are walking or not