gofofa

package module
v0.2.27 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 18, 2024 License: MIT Imports: 33 Imported by: 0

README

GoFOFA

Latest releaseGitHub Release DateGitHub All ReleasesGitHub issues

中文 README | User Guide | Download

Project Background

GoFOFA is a command-line FOFA query tool written in Go. In addition to basic FOFA API calls, it can also process data further, enabling the transformation from raw data to business data through a modular invocation mechanism.

For any questions about GoFOFA, feel free to join our FOFA community via WeChat Group or Telegram for technical discussions.

Installation

Run the following command in the terminal to install the latest version of GoFOFA:

go install github.com/FofaInfo/GoFOFA/cmd/fofa@latest

Configure the environment variable with your FOFA API key:

export FOFA_KEY='your_key'

Execute the test command fofa search -s 1 ip=1.1.1.1. If results are returned, the installation and configuration are successful.

Output example:

fofa search -s 1 ip=1.1.1.1
query fofa of: ip=1.1.1.1
1.1.1.1,8880

GoFOFA offers rich functionalities. For a complete user guide and installation instructions, click here.

Key Features

Batch Search and Download Module

In addition to basic API calls, the query module introduces many practical features, such as batch search, URL concatenation, obtaining domains from certificate extension, diverse icon-based queries, and large-scale data downloads.

Batch search addresses issues with querying large volumes of homogeneous data. Currently, it supports batch queries for IPs or domains.

The program can automatically process IPs in a file, concatenate statements for batch queries, and complete data downloads. For example, if a task involves 1,000 IPs, the program will concatenate them into groups of 100 queries (10 groups in total) and perform automated downloads, significantly reducing data retrieval time.

Example case:

  1. A task file ip.txt containing 1,000 IPs;
  2. Query results from the last year;
  3. Retrieved fields include ip, port, host, link, title, and domain;
  4. Automatically saved as a data.csv file.
fofa dump -i ip.txt -bt ip -bs 50000 -f ip,port,host,protocol,lastupdatetime -o data.csv
Web Liveness Detection Module

Liveness detection is a common need after retrieving data. This module supports simultaneous data retrieval and liveness detection, returning results with an added isActive field and updating the status_code field.

Command parameter: checkActive, with a numerical value representing the number of retries in case of timeout.

Example case:

  1. Requesting 100 results;
  2. Selecting json format for output;
  3. Setting 3 retries for timeouts.
fofa --checkActive 3 -s 100 --format=json port=80 && type=subdomain
JS Rendering Module

The JS Rendering Module processes URLs in the retrieved data for JavaScript rendering, supporting the extraction of rendered HTML tags such as title and body.

Command parameters:

  • jsRender to activate the rendering module;
  • -url to specify a single target;
  • -tags to select the tags to retrieve after rendering.

🔴 Note: The jsRender module is resource-intensive; it's not recommended to arbitrarily modify the workers parameter.

Example case:

  1. Requesting 10 results;
  2. Rendering and recognizing URLs from the pipeline;
  3. Adding the newly extracted title field.
fofa jsRender -tags title --format=json -i link.txt -o data.txt
Data Classification Module

In the result output phase, GoFOFA supports categorizing CSV files based on key characteristics. This operation can be configured through the config.yaml file.

Users can set filtering rules in the config.yaml file beforehand. Using a built-in contain method, data can be processed and categorized.

Usage example:

fofa category -input input.csv [-o category.csv]

Configuration example:

  1. sheet1: Websites with asset titles containing keywords such as "backend", "system", "login" , "management";
  2. sheet2: Assets with titles containing "nginx," "tomcat," "IIS," or "Welcome to OpenResty";
  3. sheet3: Assets with protocols other than HTTP/HTTPS;
  4. sheet4: Assets whose category field includes labels like "router," "video surveillance," "network storage," or "firewall."
categories:  
  - name: "sheet1"    
    filters:    
      - "CONTAIN(title, 'backend') && CONTAIN(title, 'login')"
      - "CONTAIN(title, 'management') && CONTAIN(title, 'system')"

  - name: "sheet2"    
    filters:      
      - "CONTAIN(title, 'nginx') && CONTAIN(title, 'tomcat')"
      - "CONTAIN(title, 'IIS') && CONTAIN(title, 'Welcome to OpenResty')"

  - name: "sheet3"
    filters:
      - "protocol!='http' || protocol!='https'"

  - name: "sheet4"
    filters:
      - "CONTAIN(product_category, 'router') && CONTAIN(product_category, 'video surveillance')"
      - "CONTAIN(product_category, 'network storage') && CONTAIN(product_category, 'firewall')"

GoFOFA All Parameters list

Parameter Abbreviation Default Value Description
fields f ip,port Fields returned by FOFA. Learn More
format csv Output format: csv/json/xml
outFile o Output file. If not set, prints to terminal
size s 100 Query size. Maximum is 10,000, subject to deductMode
deductMode Consumption of f-points. If not set, uses max free query limit
fixUrl false Combines URLs (e.g., 1.1.1.1,80 becomes http://1.1.1.1)
urlPrefix http:// URL prefix
full false Retrieves full data
uniqByIP false Removes duplicates based on IP
workers 10 Number of threads
rate 2 Query rate per second
template ip={} Replaces {} with content from pipeline input
inFile i Input file. If not set, reads from pipeline input
checkActive -1 Number of retries for liveness checks. -1 disables it
deWildcard -1 Removes wildcard domains. -1 disables this feature
filter Data filtering rules (e.g., `port<100
dedupHost false Removes duplicates for subdomains
headline false Outputs CSV headers. Available only when format is CSV
help h false Displays usage information
dump
Parameter Abbreviation Default Value Description
fields f ip,port Fields returned by FOFA. Learn More
format csv Output format: csv/json/xml
outFile o Output file. If not set, prints to terminal
inFile i Input file. If not set, reads from pipeline input
size s 100 Query size. No upper limit but consumes f-points or free query quota
fixUrl false Combines URLs (e.g., 1.1.1.1,80 becomes http://1.1.1.1)
urlPrefix http:// URL prefix
full false Retrieves full data
batchSize bs 1000 Number of records to fetch per batch
batchType bt Batch query type: ip/domain
help h false Displays usage information
jsRender
Parameter Abbreviation Default Value Description
url u Single URL rendering
tags t Tags to extract. Options: title/body
format csv Output format: csv/json/xml
outFile o Output file. If not set, prints to terminal
inFile i Input file. If not set, reads from pipeline input
workers 2 Number of threads
retry 3 Number of timeout retries
help h false Displays usage information
domains
Parameter Abbreviation Default Value Description
outFile o Output file. If not set, prints to terminal
size s 100 Query size. Maximum is 10,000, subject to deductMode
deductMode Consumption of f-points. If not set, uses max free query limit
full false Retrieves full data
withCount false Outputs domain count
clue false Outputs clue statements
help h false Displays usage information
active
Parameter Abbreviation Default Value Description
url u Single URL liveness check
format csv Output format: csv/json/xml
outFile o Output file. If not set, prints to terminal
inFile i Input file. If not set, reads from pipeline input
workers 2 Number of threads
retry 3 Number of timeout retries
help h false Displays usage information
category
Parameter Abbreviation Default Value Description
inFile i Input classification file (CSV)
unique Ensures unique classification data
help h false Displays usage information
dedup
Parameter Abbreviation Default Value Description
dedup d Field to deduplicate
inFile i Input file for deduplication (CSV)
outFile o duplicate.csv Output file
help h false Displays usage information
host
Parameter Abbreviation Default Value Description
help h false Displays usage information
icon
Parameter Abbreviation Default Value Description
open false Opens FOFA search page based on icon results
help h false Displays usage information
stats
Parameter Abbreviation Default Value Description
fields f title,country Fields returned by FOFA. Learn More
size s 5 Query count. -1 for infinite queries
help h false Displays usage information
random
Parameter Abbreviation Default Value Description
fields f ip,port,host,header,title,server,lastupdatetime Fields returned by FOFA. Learn More
format json Output format: csv/json/xml
size s 1 Query count. -1 for infinite queries
sleep 1000 Interval between queries in milliseconds
fixUrl false Combines URLs (e.g., 1.1.1.1,80 becomes http://1.1.1.1)
urlPrefix http:// URL prefix
full false Retrieves full data
help h false Displays usage information
count
Parameter Abbreviation Default Value Description
help h false Displays usage information
account
Parameter Abbreviation Default Value Description
help h false Displays usage information

Final Thoughts

Before deciding to open-source GoFOFA, we were well aware that the community already had several excellent FOFA API tools. Examples include fofa_viewer by WgpSec, fofax by Xiecat, and FofaMap by the HxO team. These tools are powerful and beloved by white-hat researchers. Initially, we believed that with so many outstanding FOFA API tools already available, creating another similar tool as an official project might seem unnecessary.

However, through a deeper understanding of user needs, we discovered that merely retrieving API data is far from sufficient. The real challenge lies in efficient post-processing of the data to meet various practical scenarios. This pain point resonated with us deeply. “There’s nothing beyond the capabilities of our users,” we thought, as the diverse requirements for data processing continued to emerge. Even within our team, while handling data, we often found ourselves developing various solutions for secondary processing.

This led us to a thought: why not consolidate common needs into an open-source tool? A tool that not only retrieves data efficiently but also provides powerful secondary processing capabilities, while serving as a platform for community co-creation. Users can experience FOFA's features, propose additional requirements, and even co-create new functionalities with us.

We hope that by open-sourcing GoFOFA, more users can enjoy this powerful and flexible tool. We also look forward to collaborating with everyone to foster innovation and co-creation within the FOFA community.

If you think this tool is useful, give us a ⭐ on GitHub!

Documentation

Overview

Package gofofa fofa client in Go

env settings: - FOFA_CLIENT_URL full fofa connnection string, format: <url>/?email=<email>&key=<key>&version=<v2> - FOFA_SERVER fofa server - FOFA_EMAIL fofa account email - FOFA_KEY fofa account key

Index

Constants

View Source 
const (
	NoHostWithFixURL = "host field must included when fixUrl option set"
)

Variables

This section is empty.

Functions

func Category 

func Category(configFile, inputFile string, options ...CategoryOptions) (string, error)

func ExtractIconFromHtml 

func ExtractIconFromHtml(data []byte) string

ExtractIconFromHtml extract link icon from html

func FofaURLFromEnv 

func FofaURLFromEnv() string

FofaURLFromEnv parse fofa connection url from env, then generate url string

func IconHash 

func IconHash(iconUrl string) (hash string, err error)

IconHash if url is a local icon file, then calc the hash if url is remote icon url, the download and calc the hash if url is web homepage, then try to parse favicon url and download it, then calc the hash

func NewFixUrl 

func NewFixUrl(rowURL string) string

func NewRequestConfig 

func NewRequestConfig(fullURL string) *http.Client

Types

type AccountInfo 

type AccountInfo struct {
	Error          bool     `json:"error"`            // error or not
	ErrMsg         string   `json:"errmsg,omitempty"` // error string message
	FCoin          int      `json:"fcoin"`            // fcoin count
	FofaPoint      int64    `json:"fofa_point"`       // fofa point
	IsVIP          bool     `json:"isvip"`            // is vip
	VIPLevel       VipLevel `json:"vip_level"`        // vip level
	RemainApiQuery int      `json:"remain_api_query"` // available query
	RemainApiData  int      `json:"remain_api_data"`  // available data amount
}

AccountInfo fofa account info

func (AccountInfo) String 

func (ai AccountInfo) String() string

type Cate added in v0.2.27 

type Cate struct {
	Name    string   `yaml:"name"`
	Filters []string `yaml:"filters"`
}

type CategoryOptions 

type CategoryOptions struct {
	Unique       bool   // is the classification unique
	RelationFile string // relation file
	SourceField  string // source field
	TargetField  string // target field
}

type Client 

type Client struct {
	Server     string // can set local server for debugging, format: <scheme>://<host>
	APIVersion string // api version
	Email      string // fofa email
	Key        string // fofa key

	Account    AccountInfo // fofa account info
	DeductMode DeductMode  // Deduct Mode
	// contains filtered or unexported fields
}

Client of fofa connection

func NewClient 

func NewClient(options ...ClientOption) (*Client, error)

NewClient from fofa connection string to config and with env config merge

func (*Client) AccountInfo 

func (c *Client) AccountInfo() (ac AccountInfo, err error)

AccountInfo fetch account info from fofa

func (*Client) DumpSearch 

func (c *Client) DumpSearch(query string, allSize int, batchSize int, fields []string, onResults func([][]string, int) error, options ...SearchOptions) (err error)

DumpSearch search fofa host data query fofa query string size data size: -1 means all，0 means just data total info, >0 means actual size fields of fofa host search options for search

func (*Client) Fetch 

func (c *Client) Fetch(apiURI string, params map[string]string, v interface{}) (err error)

Fetch http request and parse as json return to v

func (*Client) GetContext 

func (c *Client) GetContext() context.Context

GetContext 获取context，用于中止任务

func (*Client) HostSearch 

func (c *Client) HostSearch(query string, size int, fields []string, options ...SearchOptions) (res [][]string, err error)

HostSearch search fofa host data query fofa query string size data size: -1 means all，0 means just data total info, >0 means actual size fields of fofa host search options for search

func (*Client) HostSize 

func (c *Client) HostSize(query string) (count int, err error)

HostSize fetch query matched host count

func (*Client) HostStats 

func (c *Client) HostStats(host string) (data HostStatsData, err error)

HostStats fetch query matched host count

func (*Client) SetContext 

func (c *Client) SetContext(ctx context.Context)

SetContext 设置context，用于中止任务

func (*Client) Stats 

func (c *Client) Stats(query string, size int, fields []string) (res []StatsObject, err error)

Stats aggs fofa host data query fofa query string size data size fields' field of fofa host struct

func (*Client) URL 

func (c *Client) URL() string

URL generate fofa connection url string

func (*Client) Update 

func (c *Client) Update(configURL string) error

Update merge config from config url

type ClientOption 

type ClientOption func(c *Client) error

func WithAccountDebug 

func WithAccountDebug(v bool) ClientOption

WithAccountDebug 是否错误里面显示账号密码原始信息

func WithLogger 

func WithLogger(logger *logrus.Logger) ClientOption

WithLogger set logger

func WithOnResults 

func WithOnResults(onResults func(results [][]string)) ClientOption

WithOnResults set on results callback

func WithURL 

func WithURL(configURL string) ClientOption

WithURL configURL format: <url>/?email=<email>&key=<key>&version=<v2>&tlsdisabled=false&debuglevel=0

type Config added in v0.2.27 

type Config struct {
	Categories   []Cate      `yaml:"categories"`
	CustomFields []CusFields `yaml:"custom_fields"`
}

func LoadConfig added in v0.2.27 

func LoadConfig(configFile string) (*Config, error)

type CusFields added in v0.2.27 

type CusFields struct {
	Name   string `yaml:"name"`
	Fields string `yaml:"fields"`
}

type DeductMode 

type DeductMode int

DeductMode should deduct fcoin automatically or just use free limit 

const (
	// DeductModeFree only use free limit size
	DeductModeFree DeductMode = 0
	// DeductModeFCoin deduct fcoin automatically if account has fcoin
	DeductModeFCoin DeductMode = 1
)

func ParseDeductMode 

func ParseDeductMode(v string) DeductMode

ParseDeductMode parse string to DeductMode

type HostResults 

type HostResults struct {
	Mode    string      `json:"mode"`
	Error   bool        `json:"error"`
	Errmsg  string      `json:"errmsg"`
	Query   string      `json:"query"`
	Page    int         `json:"page"`
	Size    int         `json:"size"` // 总数
	Results interface{} `json:"results"`
	Next    string      `json:"next"`
}

HostResults /search/all api results

type HostStatsData 

type HostStatsData struct {
	Error       bool     `json:"error"`
	Errmsg      string   `json:"errmsg"`
	Host        string   `json:"host"`
	IP          string   `json:"ip"`
	ASN         int      `json:"asn"`
	ORG         string   `json:"org"`
	Country     string   `json:"country_name"`
	CountryCode string   `json:"country_code"`
	Protocols   []string `json:"protocol"`
	Ports       []int    `json:"port"`
	Categories  []string `json:"category"`
	Products    []string `json:"product"`
	UpdateTime  string   `json:"update_time"`
}

HostStatsData /host api results

type HttpResponse 

type HttpResponse struct {
	IsActive   bool
	StatusCode string
}

func DoHttpCheck 

func DoHttpCheck(rowURL string, retry int) HttpResponse

type SearchOptions 

type SearchOptions struct {
	FixUrl      bool   // each host fix as url, like 1.1.1.1,80 will change to http://1.1.1.1, https://1.1.1.1:8443 will no change
	UrlPrefix   string // default is http://
	Full        bool   // search result for over a year
	UniqByIP    bool   // uniq by ip
	CheckActive int    // probe website is existed, add isActive field
	DeWildcard  int    // number of wildcard domains retained
	Filter      string // filter data by rules
	DedupHost   bool   // prioritize subdomain data retention
}

SearchOptions options of search, for post processors

type StatsItem 

type StatsItem struct {
	Name  string
	Count int
}

StatsItem one stats item

type StatsObject 

type StatsObject struct {
	Name  string
	Items []StatsItem
}

StatsObject one stats object

type StatsResults 

type StatsResults struct {
	Error          bool                   `json:"error"`
	Errmsg         string                 `json:"errmsg"`
	Distinct       map[string]interface{} `json:"distinct"`
	Aggs           map[string]interface{} `json:"aggs"`
	LastUpdateTime string                 `json:"lastupdatetime"`
}

StatsResults /search/stats api results

type VipLevel 

type VipLevel int
const (
	VipLevelNone        VipLevel = 0 // 注册用户
	VipLevelNormal      VipLevel = 1 // 普通会员
	VipLevelAdvanced    VipLevel = 2 // 高级会员
	VipLevelEnterprise  VipLevel = 3 // 企业版
	VipLevelEnterprise2 VipLevel = 5 // 企业版
)
const (
	VipLevelSubPersonal VipLevel = 11 // 订阅个人
	VipLevelSubPro      VipLevel = 12 // 订阅专业
	VipLevelSubBuss     VipLevel = 13 // 订阅商业版
)
const (
	VipLevelRed     VipLevel = 20  // 红队版
	VipLevelStudent VipLevel = 22  // 教育账户
	VipLevelNever   VipLevel = 100 // 不可能的等级
)

type WorkerBrowser 

type WorkerBrowser struct {
	Url   string
	Retry int
}

func NewWorkerBrowser 

func NewWorkerBrowser(url string, retry int) *WorkerBrowser

func (*WorkerBrowser) ParseHTML 

func (wp *WorkerBrowser) ParseHTML(htmlStr string, tag string) string

func (*WorkerBrowser) Run 

func (wp *WorkerBrowser) Run() (response map[string]interface{}, err error)

Source Files

View all Source files

Directories

Path Synopsis
cmd
fofa
fofa/cmd
pkg
outformats
readformats

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.