webanalyze

README

webanalyze

This is a port of Wappalyzer in Go. This tool is designed to be performant and allows to test huge lists of hosts.

Installation and usage

$ go get -u github.com/rverton/webanalyze/...
$ webanalyze -update # loads new apps.json file from wappalyzer project
$ webanalyze -h
Usage of webanalyze:
  -apps string
        app definition file. (default "apps.json")
  -crawl int
        links to follow from the root page (default 0)
  -host string
        single host to test
  -hosts string
        filename with hosts, one host per line.
  -output string
        output format (stdout|csv|json) (default "stdout")
  -search
        searches all urls with same base domain (i.e. example.com and sub.example.com) (default true)
  -update
        update apps file
  -worker int
        number of worker (default 4)

The -update flags downloads a current version of apps.json from the wappalyzer repository to the current folder.

Display

Run cmd/webanalyze/index.html (on sth. like SimpleHTTPServer) to display results in a searchable dashboard.

Development / Usage as a lib

See cmd/webanalyze/main.go for an example.

Example

$ webanalyze -host https://stackshare.io
2019/01/05 23:41:45 Scanning with 4 workers.
2019/01/05 23:41:46 [+] https://stackshare.io (1.025640074s):
2019/01/05 23:41:46 	- jQuery,  (JavaScript Libraries)
2019/01/05 23:41:46 	- Cowboy,  (Web Frameworks, Web Servers)
2019/01/05 23:41:46 	- Erlang,  (Programming Languages)
2019/01/05 23:41:46 	- Ruby on Rails,  (Web Frameworks)
2019/01/05 23:41:46 	- Ruby,  (Programming Languages)

$ webanalyze -host https://stackshare.io -output csv
2019/01/05 23:45:04 Scanning with 4 workers.
Host,Category,App,Version
https://stackshare.io,"Web Frameworks,Web Servers",Cowboy,
https://stackshare.io,Programming Languages,Erlang,
https://stackshare.io,Web Frameworks,Ruby on Rails,
https://stackshare.io,Programming Languages,Ruby,
https://stackshare.io,JavaScript Libraries,jQuery,

Documentation

Index

• Constants
• func DownloadFile(from, to string) error
• func Init(host string, appsFile string, body string, headers map[string][]string) (chan Result, error)
• type App
  • func (app *App) FindInHeaders(headers http.Header) (matches [][]string, version string)
• type AppRegexp
• type AppsDefinition
• type Category
• type Job
  • func NewOfflineJob(url, body string, headers map[string][]string) *Job
  • func NewOnlineJob(url, body string, headers map[string][]string, crawlCount int, ...) *Job
• type Match
• type Result
• type StringArray
  • func (t *StringArray) UnmarshalJSON(data []byte) error
• type WebAnalyzer
  • func NewWebAnalyzer(workers int, appsFile string) (*WebAnalyzer, error)

Constants

const WappalyzerURL = "https://raw.githubusercontent.com/AliasIO/Wappalyzer/master/src/apps.json"

WappalyzerURL is the link to the latest apps.json file in the Wappalyzer repo

Functions

func DownloadFile

func DownloadFile(from, to string) error

DownloadFile pulls the latest apps.json file from the Wappalyzer github

func Init

func Init(host string, appsFile string, body string, headers map[string][]string) (chan Result, error)

Init sets up all the workders, reads in the host data and returns the results channel or an error

Types

type App

type App struct {
	Cats     StringArray       `json:"cats"`
	CatNames []string          `json:"category_names"`
	Cookies  map[string]string `json:"cookies"`
	Headers  map[string]string `json:"headers"`
	Meta     map[string]string `json:"meta"`
	HTML     StringArray       `json:"html"`
	Script   StringArray       `json:"script"`
	URL      StringArray       `json:"url"`
	Website  string            `json:"website"`
	Implies  StringArray       `json:"implies"`

	HTMLRegex   []AppRegexp `json:"-"`
	ScriptRegex []AppRegexp `json:"-"`
	URLRegex    []AppRegexp `json:"-"`
	HeaderRegex []AppRegexp `json:"-"`
	MetaRegex   []AppRegexp `json:"-"`
	CookieRegex []AppRegexp `json:"-"`
}

App type encapsulates all the data about an App from apps.json

func (*App) FindInHeaders

func (app *App) FindInHeaders(headers http.Header) (matches [][]string, version string)

type AppRegexp

type AppRegexp struct {
	Name    string
	Regexp  *regexp.Regexp
	Version string
}

type AppsDefinition

type AppsDefinition struct {
	Apps map[string]App      `json:"apps"`
	Cats map[string]Category `json:"categories"`
}

AppsDefinition type encapsulates the json encoding of the whole apps.json file

var (
	// AppDefs provides access to the unmarshalled apps.json file
	AppDefs *AppsDefinition
)

type Category

type Category struct {
	Name string `json:"name"`
}

Category names defined by wappalyzer

type Job

type Job struct {
	URL             string
	Body            []byte
	Headers         http.Header //map[string][]string
	Cookies         []*http.Cookie
	Crawl           int
	SearchSubdomain bool
	// contains filtered or unexported fields
}

Job may consist only of a URL, in which case webanalyse will proceed to download from that URL, or it may consist of the Body and Headers of a request to a URL and the URL itself, in which case these fields will be trusted and used for analysis without further network traffic. If a Job is constructed using the OfflineJob constructor then a flag will be set to prevent downloading regardless of the contents (or absence) of the Body or Headers fields.

func NewOfflineJob

func NewOfflineJob(url, body string, headers map[string][]string) *Job

NewOfflineJob constructs a job out of the constituents of a webanalyzer analysis; a URL, a body, and response headers. This constructor also sets a flag to explicitly prevent fetching from the URL even if the body and headers are nil or empty. Use this for...offline jobs.

func NewOnlineJob

func NewOnlineJob(url, body string, headers map[string][]string, crawlCount int, searchSubdomain bool) *Job

NewOnlineJob constructs a job that may either have a URL only, or a URL, Body and Headers. If it contains at least a URL and Body, then webanalyzer will not re-download the data, but if a Body is absent then downloading will be attempted.

type Match

type Match struct {
	App     `json:"app"`
	AppName string     `json:"app_name"`
	Matches [][]string `json:"matches"`
	Version string     `json:"version"`
}

Match type encapsulates the App information from a match on a document

type Result

type Result struct {
	Host     string        `json:"host"`
	Matches  []Match       `json:"matches"`
	Duration time.Duration `json:"duration"`
	Error    error         `json:"error"`
}

Result type encapsulates the result information from a given host

type StringArray

type StringArray []string

StringArray type is a wrapper for []string for use in unmarshalling the apps.json

func (*StringArray) UnmarshalJSON

func (t *StringArray) UnmarshalJSON(data []byte) error

UnmarshalJSON is a custom unmarshaler for handling bogus apps.json types from wappalyzer

type WebAnalyzer

type WebAnalyzer struct {
	Results chan Result
	// contains filtered or unexported fields
}

WebAnalyzer types holds an analyzation job

func NewWebAnalyzer

func NewWebAnalyzer(workers int, appsFile string) (*WebAnalyzer, error)

NewWebAnalyzer returns an analyzer struct for an ongoing job, which may be "fed" jobs via a method and returns them via a channel when complete.