rake

Documentation

Index

• type Any
• type Rake
  • func New(rake func(a item), attr func(a item) interface{}, somany int) (my *Rake)
  • func (my *Rake) Done() (done <-chan struct{})
  • func (my *Rake) Feed(items ...item) *Rake

Types

type Any

type Any generic.Type

Any is the generic type flowing thru the pipe network.

type Rake

type Rake struct {
	// contains filtered or unexported fields
}

Rake represents a fanned out circular pipe network with a flexibly adjusting buffer. Any item is processed once only - items seen before are filtered out.

A Rake may be used e.g. as a crawling Crawler where every link shall be visited only once.

func New

func New(
	rake func(a item),
	attr func(a item) interface{},
	somany int,
) (
	my *Rake,
)

New returns a (pointer to a) new operational Rake.

`rake` is the operation to be executed in parallel on any item which has not been seen before. Have it use `myrake.Feed(items...)` in order to provide feed-back.

`attr` allows to specify an attribute for the seen filter. Pass `nil` to filter on any item itself.

`somany` is the # of parallel processes - the parallelism of the network built by Rake, the # of parallel raking endpoints of the Rake.

func (*Rake) Done

func (my *Rake) Done() (done <-chan struct{})

Done returns a channel which will be signalled and closed when traffic has subsided, nothing is left to be processed and consequently all goroutines have terminated.

func (*Rake) Feed

func (my *Rake) Feed(items ...item) *Rake

Feed registers new items on the network.