jobs

Documentation

Index

• func CreateDNSJob(url *url.URL, resolverAddress net.IP, resolverPort uint16) (*pipeline.Job, error)
• func CreateHTTPJob(url *url.URL, userAgent string) *pipeline.Job
• func CreateRenderJob(url *url.URL, quality int) *pipeline.Job
• func CreateRobotsJob(target *url.URL, userAgent string) (*pipeline.Job, error)
• func CreateScrapeJob(payload *HTTPResponsePayload) *pipeline.Job
• type HTTPRequestPayload
• type HTTPResponsePayload

Functions

func CreateDNSJob

func CreateDNSJob(url *url.URL, resolverAddress net.IP, resolverPort uint16) (*pipeline.Job, error)

CreateDNSJob fetches a hostname's A record and creates the record.

func CreateHTTPJob

func CreateHTTPJob(url *url.URL, userAgent string) *pipeline.Job

CreateHTTPJob performs a HTTP GET request and returns the corresponding records.

func CreateRenderJob

func CreateRenderJob(url *url.URL, quality int) *pipeline.Job

CreateRenderJob creates a render of the site.

func CreateRobotsJob

func CreateRobotsJob(target *url.URL, userAgent string) (*pipeline.Job, error)

CreateRobotsJob fetches the Robots TXT file.

func CreateScrapeJob

func CreateScrapeJob(payload *HTTPResponsePayload) *pipeline.Job

CreateScrapeJob extracts URLs in use by the target.

Types

type HTTPRequestPayload

type HTTPRequestPayload struct {
	warc.RawPayload
	Request *http.Request
}

HTTPRequestPayload is the payload of a HTTP request.

type HTTPResponsePayload

type HTTPResponsePayload struct {
	warc.RawPayload
	Response *http.Response
}

HTTPResponsePayload is the payload of a HTTP response.