googlesearch

README

⭐   the project to show your appreciation. :arrow_upper_right:

Quickly scrape Google Search Results.

Example

import "fmt"
import "github.com/rocketlaunchr/google-search"

func main() {
	fmt.Println(googlesearch.Search(nil, "cars for sale in Toronto, Canada"))
}

Results:

([]googlesearch.Result) (len=11 cap=16) {
 (googlesearch.Result) {
  Rank: (int) 1,
  URL: (string) (len=42) "https://www.autotrader.ca/cars/on/toronto/",
  Title: (string) (len=51) "New & Used Cars for sale in Toronto | autoTRADER.ca",
  Description: (string) ""
 },
 (googlesearch.Result) {
  Rank: (int) 2,
  URL: (string) (len=42) "https://www.autotrader.ca/cars/on/toronto/",
  Title: (string) (len=51) "New & Used Cars for sale in Toronto | autoTRADER.ca",
  Description: (string) ""
 },
 (googlesearch.Result) {
  Rank: (int) 3,
  URL: (string) (len=50) "https://www.carpages.ca/ontario/toronto/used-cars/",
  Title: (string) (len=31) "Used Cars Toronto | Carpages.ca",
  Description: (string) (len=337) "13518 results - Used Cars, Trucks and SUVs for Sale in Toronto, ON. 2009 Acura TL. AWD, Navi, Camera, Leather, 3/Y warranty availabl. 2010 Chevrolet Traverse. 2LT. 2017 Jaguar F-PACE. 35T-AWD-NAVI-CAMERA-PANO ROOF-CPO WARRANTY. 2005 Audi A6. $2,495. 2007 Audi A4. 2.0T. 2012 Audi Q7. 3.0L Premium Plus. 2005 Ford F-250. 2010 Nissan Cube."
 }
}

⚠ Warning

The implementation relies on Google's search page DOM being constant. From time to time, Google changes their DOM and thus breaks the implementation.

In the event it changes, this package will be updated as soon as possible.

Also note, that if you call this function too quickly, Google detects that it is being scraped and produces a recaptcha which interferes with the scraping. Don't call it in quick succession. It may take some time before Google unblocks you.

Always use the built-in rate-limiter set to reasonable settings.

  HTTP STATUS CODE: 429 — Too Many Requests

Credits

Special thanks to Edmund Martin.

Other useful packages

• awesome-svelte - Resources for killing react.js
• dataframe-go - Statistics and data manipulation
• dbq - Zero boilerplate database operations for Go
• electron-alert - SweetAlert2 for Electron Applications
• igo - A Go transpiler with cool new syntax such as fordefer (defer for for-loops)
• mysql-go - Properly cancel slow MySQL queries
• react - Build front end applications using Go
• remember-go - Cache slow database queries
• showerglass - A soothing face filter for privacy
• testing-go - Testing framework for unit testing

Documentation

Index

• Variables
• type Result
  • func Search(ctx context.Context, searchTerm string, opts ...SearchOptions) ([]Result, error)
• type SearchOptions

Examples

• Search

Variables

var ErrBlocked = errors.New("google block")

ErrBlocked indicates that Google has detected that you were scraping and temporarily blocked you. The duration of the block is unspecified.

See: https://github.com/rocketlaunchr/google-search#warning-warning

var GoogleDomains = map[string]string{}/* 198 elements not displayed */

GoogleDomains represents localized Google homepages. The 2 letter country code is based on ISO 3166-1 alpha-2.

See: https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2

var RateLimit = rate.NewLimiter(rate.Inf, 0)

RateLimit sets a global limit to how many requests to Google Search can be made in a given time interval. The default is unlimited (but obviously Google Search will block you temporarily if you do too many calls too quickly).

See: https://godoc.org/golang.org/x/time/rate#NewLimiter

Types

type Result

type Result struct {

	// Rank is the order number of the search result.
	Rank int `json:"rank"`

	// URL of result.
	URL string `json:"url"`

	// Title of result.
	Title string `json:"title"`

	// Description of the result.
	Description string `json:"description"`
}

Result represents a single result from Google Search.

func Search

func Search(ctx context.Context, searchTerm string, opts ...SearchOptions) ([]Result, error)

Search returns a list of search results from Google.

Example

opt := SearchOptions{
	CountryCode: "au",
}

//lint:ignore SA1012 ignore this bare essentials by passing nil for context and removing context package (despite not being idiomatic go).
serp, err := Search(nil, "First Aid Course Australia Wide First Aid", opt)

if err != nil {
	fmt.Print(err.Error())
}

for _, result := range serp {
	if strings.Contains(result.URL, "australiawidefirstaid.com.au") {
		fmt.Println("Australia Wide First Aid (https://www.australiawidefirstaid.com.au/) found in the serp")
		break
	}
}

Output:

Australia Wide First Aid (https://www.australiawidefirstaid.com.au/) found in the serp

type SearchOptions

type SearchOptions struct {

	// CountryCode sets the ISO 3166-1 alpha-2 code of the localized Google Search homepage to use.
	// The default is "us", which will return results from https://www.google.com.
	CountryCode string

	// LanguageCode sets the language code.
	// Default: en
	LanguageCode string

	// Limit sets how many results to fetch (at maximum).
	Limit int

	// Start sets from what rank the new result set should return.
	Start int

	// UserAgent sets the UserAgent of the http request.
	// Default: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
	UserAgent string

	// OverLimit searches for more results than that specified by Limit.
	// It then reduces the returned results to match Limit.
	OverLimit bool

	// ProxyAddr sets a proxy address to avoid IP blocking.
	ProxyAddr string

	// FollowNextPage, when set, scrapes subsequent result pages.
	FollowNextPage bool

	Safe bool
}

SearchOptions modifies how the Search function behaves.