filter

package
v0.0.0-...-c20d51c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 2, 2020 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package filter provides the ability to filter and modify a channel of URLs. This includes filtering exclusions, duplicates, and expanding the channel based on the wordlist.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ExtendURL

func ExtendURL(u *url.URL, tail string) *url.URL

Types

type DotProductExpander

type DotProductExpander struct {
	Hostlist []string
	// contains filtered or unexported fields
}

func NewDotProductExpander

func NewDotProductExpander(hostlist []string) *DotProductExpander

func (*DotProductExpander) Expand

func (dp *DotProductExpander) Expand(inchan <-chan *task.Task) <-chan *task.Task

func (*DotProductExpander) SetAddCount

func (dp *DotProductExpander) SetAddCount(adder workqueue.QueueAddCount)

type Expander

type Expander interface {
	Expand(in <-chan *task.Task) <-chan *task.Task
	SetAddCount(workqueue.QueueAddCount)
}

An Expander is responsible for taking input URLs and expanding them to the various mutations to be processed.

type ExtensionExpander

type ExtensionExpander struct {
	// contains filtered or unexported fields
}

Expand extensions when none are given

func NewExtensionExpander

func NewExtensionExpander(extensions []string) *ExtensionExpander

func (*ExtensionExpander) Expand

func (e *ExtensionExpander) Expand(in <-chan *task.Task) <-chan *task.Task

func (*ExtensionExpander) SetAddCount

func (e *ExtensionExpander) SetAddCount(adder workqueue.QueueAddCount)

type HeaderExpander

type HeaderExpander struct {
	Header http.Header
	// contains filtered or unexported fields
}

Header expander tries each of the headers known to it in turn

func NewHeaderExpander

func NewHeaderExpander(header http.Header) *HeaderExpander

func (*HeaderExpander) Expand

func (e *HeaderExpander) Expand(in <-chan *task.Task) <-chan *task.Task

func (*HeaderExpander) SetAddCount

func (e *HeaderExpander) SetAddCount(adder workqueue.QueueAddCount)

type WordMangler

type WordMangler func(string) string

A WordMangler is responsible for modifying a wordlist entry to produce alternatives.

type WordlistExpander

type WordlistExpander struct {
	// List of words to expand
	Wordlist []string
	// contains filtered or unexported fields
}

An Expander is responsible for taking input URLs and expanding them to include all of the words in the wordlist.

func NewWordlistExpander

func NewWordlistExpander(Wordlist []string, addSlashes, mangleCases bool) *WordlistExpander

NewWordlistExpander creates a new Expander for a list

func (*WordlistExpander) Expand

func (e *WordlistExpander) Expand(in <-chan *task.Task) <-chan *task.Task

func (*WordlistExpander) ProcessWordlist

func (e *WordlistExpander) ProcessWordlist()

Update the wordlist to contain directory & non-directory entries

func (*WordlistExpander) SetAddCount

func (e *WordlistExpander) SetAddCount(adder workqueue.QueueAddCount)

type WorkFilter

type WorkFilter struct {
	// contains filtered or unexported fields
}

WorkFilter is responsible for making sure that a given URL is only tested once, and also for applying any exclusion rules to prevent URLs from being scanned.

func NewWorkFilter

func NewWorkFilter(settings *ss.ScanSettings, counter workqueue.QueueDoneFunc) *WorkFilter

func (*WorkFilter) AddRobotsFilter

func (f *WorkFilter) AddRobotsFilter(scope []*url.URL, clientFactory client.ClientFactory)

Filter data from robots.txt

func (*WorkFilter) FilterURL

func (f *WorkFilter) FilterURL(u *url.URL)

Add another URL to filter

func (*WorkFilter) RunFilter

func (f *WorkFilter) RunFilter(src <-chan *task.Task) <-chan *task.Task

Apply a filter to a channel of URLs. Runs asynchronously.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL