extractor

Documentation

Index

• Variables
• func ExtractSources(input io.Reader, extractionPoints ...map[string][]string) (<-chan url.URL, error)
• func FetchResponse(u string, method string, headers http.Header) (*http.Response, error)
• func Filter(input <-chan url.URL, options ...func([]url.URL) []url.URL) (<-chan url.URL, error)
• func WithComplete(base *url.URL) func([]url.URL) []url.URL
• func WithResolve() func([]url.URL) []url.URL

Variables

var ExtractionPoints = map[string][]string{
	"script": {"src", "data-src"},
}

ExtractionPoints defines the default HTML tags and their attributes from which JavaScript sources are extracted.

Functions

func ExtractSources

func ExtractSources(input io.Reader, extractionPoints ...map[string][]string) (<-chan url.URL, error)

ExtractSources extracts all JavaScript sources found in the provided HTTP response reader. The optional extractionPoints can be used to overwrite the default extraction points map with a set of HTML tag names, together with a list of what attributes to extract from.

func FetchResponse

func FetchResponse(u string, method string, headers http.Header) (*http.Response, error)

FetchResponse fetches the HTTP response for the given URL.

func Filter

func Filter(input <-chan url.URL, options ...func([]url.URL) []url.URL) (<-chan url.URL, error)

Filter applies options to filter URLs from the input channel.

func WithComplete

func WithComplete(base *url.URL) func([]url.URL) []url.URL

WithComplete is an option to complete relative URLs.

func WithResolve

func WithResolve() func([]url.URL) []url.URL

WithResolve is an option to filter URLs that resolve successfully.