goccer

README

Goccer

Go concurrent crawler(s) library

Usage

package main

import (
	"log"

	"github.com/oglinuk/goccer"
)

func main() {
	wp := goccer.NewWorkerpool()

	collected := wp.Queue([]string{"https://fourohfournotfound.com"})

	for _, c := range collected {
		log.Println(c)
	}
}

Examples

See examples directory.

Documentation

Index

• type Workerpool
  • func NewWorkerpool() *Workerpool
  • func (wp *Workerpool) Queue(ps []string) []string

Types

type Workerpool

type Workerpool struct {
	// contains filtered or unexported fields
}

Workerpool is a wrapper for the jobs chan to send seeds, the wg sync.WaitGroup to wait for goroutines, the w(riter) which is an in-memory implementation (only at the moment, see TODO above crawler), the (c)rawler, and a mu(tex) to prevent concurrent map writes

func NewWorkerpool

func NewWorkerpool() *Workerpool

NewWorkerpool constructor

func (*Workerpool) Queue

func (wp *Workerpool) Queue(ps []string) []string

Queue (p)ath(s) to be crawled