age

README

🌀 agenor: rx observable concurrent directory walker

🔰 Introduction

This project provides a directory walker in the same vein as the Walk in standard library filepath, but provides many features in addition to the basic facility of simply navigating. These include, but not limited to the following:

• Comprehensive filtering with regex/glob patterns
• Resume, from a previous navigation that was prematurely terminated (typically via a ctrl-c interrupt), or cancellation via a context; this is particularly useful if the client program runs heavy IO bound tasks resulting in relatively long batch runs
• Hibernation, this allows for client defined action to be invoked for eligible file/folders encountered during navigation, when a particular condition occurs; as opposed to invoking the client action for every file/folder from the root onwards. The navigator starts off in a hibernated state, then when the condition occurs, the navigator awakens and begins invoking the client action for eligible nodes.
• Concurrent navigation implemented with a reactive model, using rx observables
• Compatibility with os.fs file system
• Ability to hook many aspects of the traversal process

📚 Usage

🎀 Features

• unit testing with Ginkgo/Gomega
• implemented with 🐍 Cobra cli framework, assisted by 🐲 Cobrass
• i18n with go-i18n
• linting configuration and pre-commit hooks, (see: linting-golang).
• uses 💥 lo

🌐 l10n Translations

This template has been setup to support localisation. The default language is en-GB with support for en-US. There is a translation file for en-US defined as src/i18n/deploy/astrolib.active.en-US.json. This is the initial translation for en-US that should be deployed with the app.

Make sure that the go-i18n package has been installed so that it can be invoked as cli, see go-i18n for installation instructions.

To maintain localisation of the application, the user must take care to implement all steps to ensure translate-ability of all user facing messages. Whenever there is a need to add/change user facing messages including error messages, to maintain this state, the user must:

• define template struct (xxxTemplData) in src/i18n/messages.go and corresponding Message() method. All messages are defined here in the same location, simplifying the message extraction process as all extractable strings occur at the same place. Please see go-i18n for all translation/pluralisation options and other regional sensitive content.

For more detailed workflow instructions relating to i18n, please see i18n README

📨 Message Bus

• Contains an alternative version of bus. The requirement for a bus implementation is based upon the need to create loosely coupled internal code. The original bus was designed with concurrency in mind so it uses locks to achieve thread safety. This aspect is surplus to requirements as all we need it for are synchronous scenarios, so it has been striped out.

Documentation

Index

• Constants
• Variables
• type Accepter
• type Addon
• type Builders
  • func Prime(facade pref.Facade, settings ...pref.Option) *Builders
  • func Resume(facade pref.Facade, settings ...pref.Option) *Builders
• type Client
• type Director
• type ExistsInFS
• type Head
• type NavigatorFactory
  • func Run(wg pants.WaitGroup) NavigatorFactory
  • func Walk() NavigatorFactory
• type Node
• type Option
• type Options
• type Rel
• type Relic
• type RenameFS
• type ResumeStrategy
• type Servant
• type Subscription
• type TraversalFS
• type TraverseInput
• type TraverseJobStream
• type TraverseJobStreamR
• type TraverseJobStreamW
• type TraverseOutput
• type TraverseOutputStream
• type TraverseOutputStreamR
• type TraverseOutputStreamW
• type Using
• type WriteFileFS
• type WriterFS

Constants

const (
	OutputChSize       = 10
	CheckCloseInterval = time.Second / 10
	TimeoutOnSend      = time.Second * 2

	// 🌀 enum: ResumeStrategy
	ResumeStrategySpawn    = enums.ResumeStrategySpawn
	ResumeStrategyFastward = enums.ResumeStrategyFastward

	// 🌀 enum:Subscribe
	SubscribeFiles                = enums.SubscribeFiles
	SubscribeDirectories          = enums.SubscribeDirectories
	SubscribeDirectoriesWithFiles = enums.SubscribeDirectoriesWithFiles
	SubscribeUniversal            = enums.SubscribeUniversal
)

Variables

var (

	// NewReadDirFS creates a file system with read directory capability
	NewReadDirFS = nef.NewReadDirFS

	// NewReaderFS creates a read only file system
	NewReaderFS = nef.NewReaderFS

	// NewReadFileFS  creates a file system with read file capability
	NewReadFileFS = nef.NewReadFileFS

	// NewStatFS creates a file system with Stat method
	NewStatFS = nef.NewStatFS

	// NewWriteFileFS creates a file system with write file capability
	NewWriteFileFS = nef.NewWriteFileFS

	// NewWriterFS creates a file system with writer capabilities
	NewWriterFS = nef.NewWriterFS

	// NewCustomSampleFilter only needs to be called explicitly when defining
	// a custom sample filter.
	NewCustomSampleFilter = filtering.NewCustomSampleFilter

	// IfOption enables options to be conditional. IfOption condition evaluates to true
	// then the option is returned, otherwise nil.
	IfOption = pref.IfOption

	// IfOptionF allows the delaying of inception of the option until the condition
	// is known to be true. This is in contrast to IfOption where the Option is
	// pre-created, regardless of the condition.
	IfOptionF = pref.IfOptionF

	// IfElseOptionF is similar to IfOptionF except that it accepts 2 options, the
	// first represents the returned option if the condition true and the second
	// if false.
	// IfElseOptionF provides conditional option selection similar to IfOptionF but
	// handles both true and false cases. It accepts a condition and two
	// ConditionalOption functions:
	// tOption (executed when condition is true) and
	// fOption (executed when condition is false).
	IfElseOptionF = pref.IfElseOptionF

	// WithAdminPath defines the path for admin related files
	WithAdminPath = pref.WithAdminPath

	// WithCPU configures the worker pool used for concurrent traversal sessions
	// in the Run function to utilise a number of go-routines equal to the available
	// CPU count, optimising performance based on the system's processing capabilities.
	WithCPU = pref.WithCPU

	// WithDepth sets the maximum number of directories deep the navigator
	// will traverse to.
	WithDepth = pref.WithDepth

	// WithFaultHandler defines a custom handler to handle an error that occurs
	// when 'Stat'ing the tree root directory. When an error occurs, traversal terminates
	// immediately. The handler specified allows custom functionality to be invoked
	// when an error occurs here.
	WithFaultHandler = pref.WithFaultHandler

	// WithFilter used to determine which file system nodes (files or directories)
	// the client defined handler is invoked for. Note that the filter does not
	// determine navigation, it only determines wether the callback is invoked.
	WithFilter = pref.WithFilter

	// WithHibernationBehaviourExclusiveWake activates hibernation
	// with a wake condition. The wake condition should be defined
	// using WithHibernationFilterWake.
	WithHibernationBehaviourExclusiveWake = pref.WithHibernationBehaviourExclusiveWake

	// WithHibernationBehaviourInclusiveSleep activates hibernation
	// with a sleep condition. The sleep condition should be defined
	// using WithHibernationFilterSleep.
	WithHibernationBehaviourInclusiveSleep = pref.WithHibernationBehaviourInclusiveSleep

	// WithHibernationFilterSleep defines the sleep condition
	// for hibernation based traversal sessions.
	WithHibernationFilterSleep = pref.WithHibernationFilterSleep

	// WithHibernationFilterWake defines the wake condition
	// for hibernation based traversal sessions.
	WithHibernationFilterWake = pref.WithHibernationFilterWake

	// WithHibernationOptions defines options for a hibernation traversal
	// session.
	WithHibernationOptions = pref.WithHibernationOptions

	// WithHookCaseSensitiveSort specifies that a directory's contents
	// should be sorted with case sensitivity.
	WithHookCaseSensitiveSort = pref.WithHookCaseSensitiveSort

	// WithHookDirectorySubPath defines an custom hook to override the
	// default behaviour for obtaining the sub-path of a directory.
	WithHookDirectorySubPath = pref.WithHookDirectorySubPath

	// WithHookFileSubPath defines an custom hook to override the
	// default behaviour for obtaining the sub-path of a file.
	WithHookFileSubPath = pref.WithHookFileSubPath

	// WithHookQueryStatus defines an custom hook to override the
	// default behaviour for Stating a directory.
	WithHookQueryStatus = pref.WithHookQueryStatus

	// WithHookReadDirectory defines an custom hook to override the
	// default behaviour for reading a directory's contents.
	WithHookReadDirectory = pref.WithHookReadDirectory

	// WithHookSort defines an custom hook to override the
	// default behaviour for sorting a directory's contents.
	WithHookSort = pref.WithHookSort

	// WithLogger defines a structure logger
	WithLogger = pref.WithLogger

	// WithNavigationBehaviours defines all navigation behaviours
	WithNavigationBehaviours = pref.WithNavigationBehaviours

	// WithOnAscend sets ascend handler, invoked when navigator
	// traverses up a directory, ie after all children have been
	// visited.
	WithOnAscend = pref.WithOnAscend

	// WithOnBegin sets the begin handler, invoked before the start
	// of a traversal session.
	WithOnBegin = pref.WithOnBegin

	// WithOnDescend sets the descend handler, invoked when navigator
	// traverses down into a child directory.
	WithOnDescend = pref.WithOnDescend

	// WithOnEnd sets the end handler, invoked at the end of a traversal
	// session.
	WithOnEnd = pref.WithOnEnd

	// WithOnSleep sets the sleep handler, when hibernation is active
	// and the sleep condition has occurred, ie when a file system
	// node is encountered that matches the hibernation's sleep filter.
	WithOnSleep = pref.WithOnSleep

	// WithOnWake sets the wake handler, when hibernation is active
	// and the wake condition has occurred, ie when a file system
	// node is encountered that matches the hibernation's wake filter.
	WithOnWake = pref.WithOnWake

	// WithPanicHandler defines a custom handler to handle a panic.
	WithPanicHandler = pref.WithPanicHandler

	// WithNoRecurse sets the navigator to not descend sub-directories.
	WithNoRecurse = pref.WithNoRecurse

	// WithNoW sets the number of go-routines to use in the worker
	// pool used for concurrent traversal sessions requested by using
	// the Run function.
	WithNoW = pref.WithNoW

	// WithSamplingOptions specifies the sampling options.
	// SampleType: the type of sampling to use
	// SampleInReverse: determines the direction of iteration for the sampling
	// operation
	// NoOf: specifies number of items required in each sample (only applies
	// when not using Custom iterator options)
	// Iteration: allows the client to customise how a directory's contents are sampled.
	// The default way to sample is either by slicing the directory's contents or
	// by using the filter to select either the first/last n entries (using the
	// SamplingOptions). If the client requires an alternative way of creating a
	// sample, eg to take all files greater than a certain size, then this
	// can be achieved by specifying Each and While inside Iteration.
	WithSamplingOptions = pref.WithSamplingOptions

	// WithSkipHandler defines a handler that will be invoked if the
	// client callback returns an error during traversal. The client
	// can control if traversal is either terminated early (fs.SkipAll)
	// or the remaining items in a directory are skipped (fs.SkipDir).
	WithSkipHandler = pref.WithSkipHandler

	// WithSortBehaviour enabling setting of all sorting behaviours.
	WithSortBehaviour = pref.WithSortBehaviour

	// WithSubPathBehaviour defines all sub-path behaviours.
	WithSubPathBehaviour = pref.WithSubPathBehaviour
)

Types

type Accepter

type Accepter = pref.Accepter

🌀 pref

type Addon

type Addon interface{}

type Builders

type Builders struct {
	// contains filtered or unexported fields
}

Builders performs build orchestration via its buildAll method. Builders is instructed by the factories (via Configure) of which there are 2; one for Walk and one for Run. The Prime/Resume extents create the Builders instance.

func Prime

func Prime(facade pref.Facade, settings ...pref.Option) *Builders

Prime extent requests that the navigator performs a full traversal from the root path specified.

func Resume

func Resume(facade pref.Facade, settings ...pref.Option) *Builders

Resume extent requests that the navigator performs a resume traversal, loading state from a previously saved session as a result of it being terminated prematurely via a ctrl-c interrupt.

type Client

type Client = core.Client

🌀 core

type Director

type Director interface {
	// Extent represents the magnitude of the traversal; ie we can
	// perform a full 'Prime' traversal, or 'Resume' from a previously
	// cancelled run.
	//
	Extent(bs *Builders) core.Navigator
}

Director

type ExistsInFS

type ExistsInFS = nef.ExistsInFS

🌀 nef

type Head

type Head = pref.Head

type NavigatorFactory

type NavigatorFactory interface {
	// Configure is a factory function that creates a navigator.
	// We don't return an error here as that would make using the factory
	// awkward. Instead, if there is an error during the build process,
	// we return a fake navigator that when invoked immediately returns
	// a traverse error indicating the build issue.
	//
	Configure(addons ...Addon) Director
}

NavigatorFactory

func Run

func Run(wg pants.WaitGroup) NavigatorFactory

Run requests a concurrent traversal of a directory tree.

func Walk

func Walk() NavigatorFactory

Walk requests a sequential traversal of a directory tree.

type Node

type Node = core.Node

type Option

type Option = pref.Option

type Options

type Options = pref.Options

type Rel

type Rel = nef.Rel

type Relic

type Relic = pref.Relic

type RenameFS

type RenameFS = nef.RenameFS

type ResumeStrategy

type ResumeStrategy = enums.ResumeStrategy

type Servant

type Servant = core.Servant

type Subscription

type Subscription = enums.Subscription

🌀 enums

type TraversalFS

type TraversalFS = tfs.TraversalFS

🌀 tfs

type TraverseInput

type TraverseInput struct {
	// Servant represents the file system entity (file or directory) for which
	// a job will execute.
	Servant Servant

	// Handler is the client defined callback function that should be
	// invoked for all eligible Nodes.
	Handler core.Client
}

TraverseInput represents the type of inputs accepted by the worker pool

type TraverseJobStream

type TraverseJobStream = pants.JobStream[TraverseInput]

TraverseJobStream represents the core channel type of the worker pool's input stream. The client owns this channel and is responsible for closing it when done or invoking Conclude directly on the pool (See boost for more details).

type TraverseJobStreamR

type TraverseJobStreamR = pants.JobStreamR[TraverseInput]

TraverseJobStreamR worker pool's read stream, pool reads from this channel.

type TraverseJobStreamW

type TraverseJobStreamW = pants.JobStreamW[TraverseInput]

TraverseJobStreamW worker pool's write stream, client writes to this channel.

type TraverseOutput

type TraverseOutput struct {
	// Servant represents the file system entity (file or directory) from
	// which this output was generated via the client defined handler.
	Servant Servant

	// Error error result of client's handler.
	Error error

	// Data is a custom field reserved for the client
	Data any
}

TraverseOutput represents the output of a single job executed by the pool.

type TraverseOutputStream

type TraverseOutputStream = pants.JobOutputStream[TraverseOutput]

TraverseOutputStream represents the core channel type of the worker pool's output stream. The pool owns this stream and will be closed only when safe to do so, which will be anytime after navigation is complete. The channel is only closed when there are no remaining outstanding jobs and all workers are idle.

type TraverseOutputStreamR

type TraverseOutputStreamR = pants.JobOutputStreamR[TraverseOutput]

TraverseOutputStreamR worker pool's output stream read by the client.

type TraverseOutputStreamW

type TraverseOutputStreamW = pants.JobOutputStreamW[TraverseOutput]

TraverseOutputStreamW worker pool's output stream written to by the pool.

type Using

type Using = pref.Using

type WriteFileFS

type WriteFileFS = nef.WriteFileFS

type WriterFS

type WriterFS = nef.WriterFS