Documentation ¶
Overview ¶
Package subscraping contains the logic of scraping agents
Index ¶
- func Exists(values []string, key string) bool
- func NewSubdomainExtractor(domain string) (*regexp.Regexp, error)
- type BasicAuth
- type Keys
- type Result
- type ResultType
- type Session
- func (s *Session) DiscardHTTPResponse(response *http.Response)
- func (s *Session) Get(ctx context.Context, getURL, cookies string, headers map[string]string) (*http.Response, error)
- func (s *Session) HTTPRequest(ctx context.Context, method, requestURL, cookies string, ...) (*http.Response, error)
- func (s *Session) Post(ctx context.Context, postURL, cookies string, headers map[string]string, ...) (*http.Response, error)
- func (s *Session) SimpleGet(ctx context.Context, getURL string) (*http.Response, error)
- func (s *Session) SimplePost(ctx context.Context, postURL, contentType string, body io.Reader) (*http.Response, error)
- type Source
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Keys ¶
type Keys struct { Binaryedge string `json:"binaryedge"` Bufferover string `json:"bufferover"` C99 string `json:"c99"` CensysToken string `json:"censysUsername"` CensysSecret string `json:"censysPassword"` Certspotter string `json:"certspotter"` Chaos string `json:"chaos"` Chinaz string `json:"chinaz"` DNSDB string `json:"dnsdb"` GitHub []string `json:"github"` IntelXHost string `json:"intelXHost"` IntelXKey string `json:"intelXKey"` PassiveTotalUsername string `json:"passivetotal_username"` PassiveTotalPassword string `json:"passivetotal_password"` Robtex string `json:"robtex"` Securitytrails string `json:"securitytrails"` Shodan string `json:"shodan"` Spyse string `json:"spyse"` ThreatBook string `json:"threatbook"` URLScan string `json:"urlscan"` Virustotal string `json:"virustotal"` ZoomEyeUsername string `json:"zoomeye_username"` ZoomEyePassword string `json:"zoomeye_password"` ZoomEyeKey string `json:"zoomeye_key"` FofaUsername string `json:"fofa_username"` FofaSecret string `json:"fofa_secret"` FullHunt string `json:"fullhunt"` Quake string `json:"quake"` Hunter string `json:"hunter"` }
Keys contains the current API Keys we have in store
type Result ¶
type Result struct { Type ResultType Source string Value string Error error IpPorts map[string][]int }
Result is a result structure returned by a source
type ResultType ¶
type ResultType int
ResultType is the type of result returned by the source
const ( Subdomain ResultType = iota Error )
Types of results returned by the source
type Session ¶
type Session struct { // Extractor is the regex for subdomains created for each domain Extractor *regexp.Regexp // Keys is the API keys for the application Keys *Keys // Client is the current http client Client *http.Client // Rate limit instance RateLimiter ratelimit.Limiter }
Session is the option passed to the source, an option is created uniquely for each source.
func NewSession ¶
NewSession creates a new session object for a domain
func (*Session) DiscardHTTPResponse ¶
DiscardHTTPResponse discards the response content by demand
func (*Session) Get ¶
func (s *Session) Get(ctx context.Context, getURL, cookies string, headers map[string]string) (*http.Response, error)
Get makes a GET request to a URL with extended parameters
func (*Session) HTTPRequest ¶
func (s *Session) HTTPRequest(ctx context.Context, method, requestURL, cookies string, headers map[string]string, body io.Reader, basicAuth BasicAuth) (*http.Response, error)
HTTPRequest makes any HTTP request to a URL with extended parameters
func (*Session) Post ¶
func (s *Session) Post(ctx context.Context, postURL, cookies string, headers map[string]string, body io.Reader) (*http.Response, error)
Post makes a POST request to a URL with extended parameters
type Source ¶
type Source interface { // Run takes a domain as argument and a session object // which contains the extractor for subdomains, http client // and other stuff. Run(context.Context, string, *Session) <-chan Result // Name returns the name of the source Name() string }
Source is an interface inherited by each passive source
Directories ¶
Path | Synopsis |
---|---|
sources
|
|
alienvault
Package alienvault logic
|
Package alienvault logic |
anubis
Package anubis logic
|
Package anubis logic |
archiveis
Package archiveis is a Archiveis Scraping Engine in Golang
|
Package archiveis is a Archiveis Scraping Engine in Golang |
binaryedge
Package binaryedge logic
|
Package binaryedge logic |
bufferover
Package bufferover is a bufferover Scraping Engine in Golang
|
Package bufferover is a bufferover Scraping Engine in Golang |
c99
Package c99 logic
|
Package c99 logic |
censys
Package censys logic
|
Package censys logic |
certspotter
Package certspotter logic
|
Package certspotter logic |
chaos
Package chaos logic
|
Package chaos logic |
commoncrawl
Package commoncrawl logic
|
Package commoncrawl logic |
crtsh
Package crtsh logic
|
Package crtsh logic |
dnsdb
Package dnsdb logic
|
Package dnsdb logic |
dnsdumpster
Package dnsdumpster logic
|
Package dnsdumpster logic |
fofa
Package fofa logic
|
Package fofa logic |
github
Package github GitHub search package Based on gwen001's https://github.com/gwen001/github-search github-subdomains
|
Package github GitHub search package Based on gwen001's https://github.com/gwen001/github-search github-subdomains |
hackertarget
Package hackertarget logic
|
Package hackertarget logic |
intelx
Package intelx logic
|
Package intelx logic |
passivetotal
Package passivetotal logic
|
Package passivetotal logic |
rapiddns
Package rapiddns is a RapidDNS Scraping Engine in Golang
|
Package rapiddns is a RapidDNS Scraping Engine in Golang |
riddler
Package riddler logic
|
Package riddler logic |
robtex
Package robtex logic
|
Package robtex logic |
securitytrails
Package securitytrails logic
|
Package securitytrails logic |
shodan
Package shodan logic
|
Package shodan logic |
sitedossier
Package sitedossier logic
|
Package sitedossier logic |
sonarsearch
Package sonarsearch logic
|
Package sonarsearch logic |
spyse
Package spyse logic
|
Package spyse logic |
sublist3r
Package sublist3r logic
|
Package sublist3r logic |
threatbook
Package threatbook logic
|
Package threatbook logic |
threatcrowd
Package threatcrowd logic
|
Package threatcrowd logic |
threatminer
Package threatminer logic
|
Package threatminer logic |
virustotal
Package virustotal logic
|
Package virustotal logic |
waybackarchive
Package waybackarchive logic
|
Package waybackarchive logic |
zoomeye
Package zoomeye logic
|
Package zoomeye logic |