scrape

Documentation

Index

• type Statistic
• type Task
  • func URLtoTask(urls []string) []Task
• type TaskDocumentResponse
  • func URLToDocument(c *http.Client, tasks []Task) ([]TaskDocumentResponse, error)

Types

type Statistic

type Statistic struct {
	Items int
}

Statistic contains statistics about a scrape

type Task

type Task struct {
	URL     string
	Payload string
}

Task is a scrape task

func URLtoTask

func URLtoTask(urls []string) []Task

URLtoTask is a helper function to convert a list of URLs to a list of tasks

type TaskDocumentResponse

type TaskDocumentResponse struct {
	Document goquery.Document
	URL      string
}

TaskDocumentResponse is the result of a fetched task and it's originating URL

func URLToDocument

func URLToDocument(c *http.Client, tasks []Task) ([]TaskDocumentResponse, error)

URLToDocument is a convinience function to directly get a goquery document(s) from a list of URLs