crawler

Documentation

Index

• type CrawlerDatabase
• type CrawlerService

Types

type CrawlerDatabase

type CrawlerDatabase interface {
	// Insert is a method to insert new page crawled on database.
	Insert(ctx context.Context, uri string, depth uint, uris []string) error

	// Find is a method to fetch links crawled from database.
	Find(ctx context.Context, uri string, depth uint) ([]string, error)
}

CrawlerDatabase is a repository to handle with persistence layer.

type CrawlerService

type CrawlerService interface {
	// Craw execute the call to craw pages concurrently and will respect depth param.
	Craw(ctx context.Context, uri string, depth uint) ([]string, error)
}

CrawlerService is a abstraction to handle with web crawler.