rake

Documentation

Index

• type Any
• type Rake
  • func New(rake func(a item), attr func(a item) interface{}, somany int) (my *Rake)
  • func (my *Rake) Attr(attr func(a item) interface{}) *Rake
  • func (my *Rake) Done() (done <-chan struct{})
  • func (my *Rake) Feed(items ...item) *Rake
  • func (my *Rake) Rake(rake func(a item)) *Rake

Examples

• Rake (Chained)
• Rake (Closure)

Types

type Any

type Any generic.Type

Any is the generic type flowing thru the pipe network.

type Rake

type Rake struct {
	// contains filtered or unexported fields
}

Rake represents a fanned out circular pipe network with a flexibly adjusting buffer. Any item is processed once only - items seen before are filtered out.

A Rake may be used e.g. as a crawling Crawler where every link shall be visited only once.

Example (Chained)

var r *Rake
crawl := func(item Any) {
	if false {
		log.Println("have:", item)
	}
	for i := 0; i < rand.Intn(9)+2; i++ {
		r.Feed(rand.Intn(2000)) // up to 10 new numbers < 2.000
	}
	time.Sleep(time.Millisecond * 10)
}

r = New(nil, nil, 80).Rake(crawl).Feed(1)

<-r.Done()

fmt.Println("Done")

Output:

Done

Example (Closure)

var r *Rake
crawl := func(item Any) {
	if false {
		log.Println("have:", item)
	}
	for i := 0; i < rand.Intn(9)+2; i++ {
		r.Feed(rand.Intn(2000)) // up to 10 new numbers < 2.000
	}
	time.Sleep(time.Millisecond * 10)
}

r = New(crawl, nil, 80)

r.Feed(1)

<-r.Done()

fmt.Println("Done")

Output:

Done

func New

func New(
	rake func(a item),
	attr func(a item) interface{},
	somany int,
) (
	my *Rake,
)

New returns a (pointer to a) new operational Rake.

`rake` is the operation to be executed in parallel on any item which has not been seen before. Have it use `myrake.Feed(items...)` in order to provide feed-back.

`attr` allows to specify an attribute for the seen filter. Pass `nil` to filter on any item itself.

`somany` is the # of parallel processes - the parallelism of the network built by Rake, the # of parallel raking endpoints of the Rake.

func (*Rake) Attr

func (my *Rake) Attr(attr func(a item) interface{}) *Rake

Attr sets the (optional) attribute to discriminate 'seen'.

`attr` allows to specify an attribute for the 'seen' filter. If not set 'seen' will discriminate any item by itself.

Seen panics iff called after first nonempty `Feed(...)`

func (*Rake) Done

func (my *Rake) Done() (done <-chan struct{})

Done returns a channel which will be signalled and closed when traffic has subsided, nothing is left to be processed and consequently all goroutines have terminated.

func (*Rake) Feed

func (my *Rake) Feed(items ...item) *Rake

Feed registers new items on the network.

func (*Rake) Rake

func (my *Rake) Rake(rake func(a item)) *Rake

Rake sets the rake function to be applied (in parallel).

`rake` is the operation to be executed in parallel on any item which has not been seen before.

You may provide `nil` here and call `Rake(..)` later to provide it. Or have it use `myrake.Feed(items...)` in order to provide feed-back.

Rake panics iff called after first nonempty `Feed(...)`