wee

package module
v0.1.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 19, 2024 License: GPL-3.0 Imports: 39 Imported by: 3

README

wee

Wee is a powerful and flexible web crawling library based on go-rod, providing a clean interface for writing web crawlers in Go. It offers advanced browser control and customization options.

Features

  • Easy-to-use API for web crawling tasks
  • Support for headless and headed browser modes
  • User mode for connecting to system Chrome browser
  • Cookie management and persistence
  • Customizable timeouts and error handling
  • Humanized and stealth mode options
  • Popover handling
  • Advanced browser configuration options
  • Support for Chrome extensions
  • Proxy configuration

Installation

go get github.com/coghost/wee

Usage

Here's a basic example of using wee to crawl search results from Baidu:

package main

import (
	"fmt"
	"github.com/coghost/wee"
)

func main() {
	bot := wee.NewBotDefault()
	defer bot.BlockInCleanUp()

	const (
		url     = `https://www.baidu.com`
		input   = `input[id="kw"]`
		keyword = `golang`
		items   = `div.result[id] h3>a`
		noItems = `div.this_should_not_existed`
	)

	bot.MustOpen(url)
	bot.MustInput(input, keyword, wee.WithSubmit(true))

	got := bot.MustAnyElem([]string{items, noItems})
	if got == noItems {
		fmt.Printf("%s has no results\n", keyword)
		return
	}

	elems, err := bot.Elems(items)
	if err != nil {
		fmt.Printf("get items failed: %v\n", err)
		return
	}

	for _, elem := range elems {
		fmt.Printf("%s - %s\n", elem.MustText(), *elem.MustAttribute("href"))
	}
}

Advanced Usage

Creating a Bot

Wee provides several ways to create a bot:

// Default bot
bot := wee.NewBotDefault()

// Headless bot
bot := wee.NewBotHeadless()

// Bot for debugging
bot := wee.NewBotForDebug()

// User mode bot (connects to system Chrome)
bot := wee.NewBotUserMode()
Customizing Bot Behavior

You can customize bot behavior using options:

bot := wee.NewBot(
    wee.UserAgent("Custom User Agent"),
    wee.AcceptLanguage("en-US"),
    wee.WithCookies(true),
    wee.Humanized(true),
    wee.StealthMode(true),
    wee.WithPopovers("popover-selector"),
)

Wee supports various ways to manage cookies:

// Use cookies from default location
wee.WithCookies(true)

// Specify cookie folder
wee.WithCookieFolder("/path/to/cookies")

// Use specific cookie file
wee.WithCookieFile("/path/to/cookiefile.json")

// Use cookies from clipboard (Copy as cURL)
wee.CopyAsCURLCookies(clipboardContent)
Error Handling

You can customize error handling behavior:

bot.SetPanicWith(wee.PanicByLogError)

Advanced Browser Configuration

Wee provides powerful options to configure the browser:

Creating a Browser
launcher, browser := wee.NewBrowser(
    wee.BrowserHeadless(false),
    wee.BrowserSlowMotion(1000),
    wee.BrowserUserDataDir("/path/to/user/data"),
    wee.BrowserPaintRects(true),
    wee.BrowserProxy("http://proxy.example.com:8080"),
)
User Mode Browser
launcher, browser := wee.NewUserMode(
    wee.BrowserUserDataDir("/path/to/user/data"),
    wee.LaunchLeakless(true),
)
Customizing Launcher
launcher := wee.NewLauncher(
    wee.BrowserHeadless(true),
    wee.BrowserExtensions("/path/to/extension1", "/path/to/extension2"),
    wee.BrowserFlags("--disable-gpu", "--no-sandbox"),
)

Browser Options

Wee supports various browser options:

  • BrowserHeadless(bool): Run browser in headless mode
  • BrowserSlowMotion(int): Set slow motion delay in milliseconds
  • BrowserUserDataDir(string): Set user data directory
  • BrowserPaintRects(bool): Show paint rectangles (useful for debugging)
  • BrowserProxy(string): Set proxy server
  • BrowserExtensions(...string): Load Chrome extensions
  • BrowserFlags(...string): Set additional Chrome flags
  • LaunchLeakless(bool): Use leakless mode when launching browser
  • BrowserIncognito(bool): Launch browser in incognito mode
  • BrowserIgnoreCertErrors(bool): Ignore certificate errors

More Examples

Check out the examples folder for more detailed examples.

Common Issues

panic: context canceled

The "context canceled" issue usually occurs when an element is bound with a timeout on bot.Elem(xxx). To resolve this, you can use:

elem.CancelTimeout().Timeout(xxx)

This will rewrite the previous timeout value.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License

Documentation

Index

Constants

View Source 
const (
	SEP       = "@@@"
	IFrameSep = "$$$"
)
View Source 
const (
	// ZeroToSec no timeout in second
	ZeroToSec = 0
	// NapToSec a nap timeout in second
	NapToSec = 2
	// ShortToSec short timeout in second
	ShortToSec = 10
	// MediumToSec medium timeout in second
	MediumToSec = 20
	// LongToSec long timeout in second
	LongToSec = 60
)
View Source 
const (
	PT1Sec  = 1
	PT10Sec = 10
	PT20Sec = 20
	PT60Sec = 60
	// PT10MilliSec a very short timeout in millisecond
	PT10MilliSec = 0.01
)
View Source 
const (
	LongScrollStep     = 32
	MediumScrollStep   = 16
	ShortScrollStep    = 8
	QuickScrollStep    = 4
	DirectlyScrollStep = 1
)
View Source 
const (
	ActivatePageRetry = 10
)
View Source 
const PaintRects = "--show-paint-rects"

PaintRects is a flag to show paint rectangles in the browser

View Source 
const (
	// SlowMotionMillis is the default slow motion duration in milliseconds
	SlowMotionMillis = 500
)

Variables

View Source 
var (
	// ErrNoSelectorClicked is returned when no cookie acceptance selectors could be clicked successfully.
	ErrNoSelectorClicked = errors.New("failed to click any cookie acceptance selectors")

	// ErrPageLoadAfterCookies is returned when the page fails to load after attempting to accept cookies.
	ErrPageLoadAfterCookies = errors.New("page failed to load after accepting cookies")
)
View Source 
var (
	ErrSelectorEmpty   = errors.New("selector is empty")
	ErrNotInteractable = errors.New("elem not interactable")
	ErrNotVisible      = errors.New("elem is not visible")

	ErrInvalidByTextFormat = fmt.Errorf("invalid selector format")
)
View Source 
var (
	ErrCannotActivateOpenedPage = errors.New("cannot activate latest opened page")
	ErrMissingCookieFile        = errors.New("missing cookie file")
)
View Source 
var (
	ErrEmptySelector  = errors.New("empty selector")
	ErrGetElemsByText = errors.New("get elems not support by text")
	ErrCannotFindElem = errors.New("cannot find elem of selector")
)
View Source 
var ErrElemShapeBox = errors.New("cannot get element box by shape.Box()")
View Source 
var ErrEmptyCookieStr = errors.New("no cookie str found")
View Source 
var ErrGetTextAfterInput = errors.New("cannot get text after input")

Functions

func BindBotLanucher 

func BindBotLanucher(bot *Bot, options ...BotOption)

BindBotLanucher launches the browser and page for the bot. This is used when we create a bot first and launch the browser elsewhere.

func Blocked 

func Blocked()

Blocked prompts the user to decide whether to quit or continue indefinitely. It calls QuitOnTimeout with a negative value to trigger the interactive prompt.

func Confirm 

func Confirm(msg ...string) bool

Confirm displays an interactive confirmation prompt and returns the user's choice as a boolean. It uses pterm's DefaultInteractiveConfirm to show the prompt.

Parameters:

  • msg: Optional. A custom message for the confirmation prompt. If not provided, a default message is used.

Returns:

  • bool: true if the user confirms, false otherwise.

func ErrCannotFindSelector 

func ErrCannotFindSelector(sel string) error

func Filenamify 

func Filenamify(name string) string

Filenamify converts a string into a valid filename by replacing illegal characters.

Parameters:

  • name: The input string to be converted into a valid filename.

Returns:

  • A string with illegal filename characters replaced by underscores.

This function replaces common illegal filename characters (such as /, \, ", :, *, ?, <, >, |) with underscores. It's useful for sanitizing strings that will be used as filenames, ensuring they are compatible with most file systems. The function uses strutil.Replaces to perform the replacements based on a predefined map of illegal characters to underscores.

func FirstOrDefault 

func FirstOrDefault[T any](dft T, args ...T) T

FirstOrDefault returns the first value from the variadic args if provided, otherwise returns the default value. It uses a generic type T to work with any data type.

Parameters:

  • dft: The default value to return if no args are provided.
  • args: A variadic parameter of type T.

Returns:

  • The first value from args if provided, otherwise the default value dft.

func FloatAorB added in v0.1.4 

func FloatAorB(a, b float64) float64

func ForceQuitBrowser added in v0.1.4 

func ForceQuitBrowser(browserName string, retry int) error

ForceQuitBrowser attempts to close a browser by killing its process.

Parameters:

  • browserName: The name of the browser process to be killed.
  • retry: The number of times to retry killing the process if unsuccessful.

Returns:

  • An error if the process couldn't be killed after all retry attempts, nil otherwise.

This function repeatedly tries to kill the specified browser process using the KillProcess function. If unsuccessful, it will retry up to the specified number of times, with a random sleep interval between attempts (using RandSleepNap). If all attempts fail, it returns the last error encountered.

func IntAorB added in v0.1.3 

func IntAorB(a, b int) int

func IsJSON 

func IsJSON(str string) bool

IsJSON checks if a string is valid JSON.

Parameters:

  • str: The input string to be checked.

Returns:

  • A boolean indicating whether the input string is valid JSON.

This function attempts to unmarshal the input string into a json.RawMessage. If the unmarshal operation succeeds (returns nil error), the function returns true, indicating that the input is valid JSON. Otherwise, it returns false.

func IsZeroVal 

func IsZeroVal(x interface{}) bool

IsZeroVal checks if a value of any type is its zero value.

Parameters:

  • x: The interface{} to be checked.

Returns:

  • A boolean indicating whether x is its zero value.

This function checks if x is nil or if it's equal to its type's zero value. It uses reflect.DeepEqual to compare x with the zero value of its type. The function works for all types, including primitives, structs, and pointers.

func KillProcess added in v0.1.4 

func KillProcess(name string) error

KillProcess terminates a process with the given name.

Parameters:

  • name: The name of the process to be terminated.

Returns:

  • An error if the process list couldn't be retrieved or if the process couldn't be killed. Returns nil if the process was successfully terminated or if no matching process was found.

This function iterates through all running processes, compares their names with the provided name, and attempts to kill the first matching process. If multiple processes have the same name, only the first encountered will be terminated. The function stops searching after killing one process.

func MustStrToFloat 

func MustStrToFloat(raw string, keptChars string) float64

MustStrToFloat converts a string to a float64, keeping only specified characters. It panics if the conversion fails.

Parameters:

  • raw: The input string to be converted.
  • keptChars: A string containing characters to be kept along with digits.

Returns:

  • A float64 representation of the filtered string.

This function first calls StrToNumChars to filter the input string, then uses cast.ToFloat64 to convert the result to a float64. If the conversion fails, it will panic.

func MustStringify 

func MustStringify(data interface{}) string

MustStringify converts an interface{} to its JSON string representation. It's similar to Stringify but panics if an error occurs during the conversion.

Parameters:

  • data: The interface{} to be converted to a JSON string.

Returns:

  • A string containing the JSON representation of the input data.

This function calls Stringify internally and panics if an error is returned. It should be used only when you're certain that the conversion will succeed, or if you want to halt execution on failure.

func NameFromURL added in v0.1.4 

func NameFromURL(uri string) string

NameFromURL extracts a filename-safe string from a URL, excluding the domain.

Parameters:

  • uri: The input URL string to be processed.

Returns:

  • A string derived from the URL path, suitable for use as a filename.

This function attempts to parse the input URL and extract a meaningful name from its path. It removes the scheme and domain, trims leading and trailing slashes, and handles empty paths. If the resulting name is empty, it returns "homepage". The final result is passed through the Filenamify function to ensure it's safe for use as a filename.

Example: 

https://en.wikipedia.org/wiki/Main_Page ==> wiki_Main_Page

If URL parsing fails, the function returns the original input string.

func NewBrowser 

func NewBrowser(opts ...BrowserOptionFunc) (*launcher.Launcher, *rod.Browser)

func NewChromeExtension 

func NewChromeExtension(line, savePath string) (string, error)

NewChromeExtension creates a Chrome extension for proxy configuration.

Parameters:

  • line: Proxy info string in format "host:port:username:password:<OTHER>"
  • savePath: Base directory to save the extension files

Returns:

  • string: Path to the created extension directory
  • error: nil if successful, error otherwise

It generates background.js and manifest.json in a subdirectory named after the proxy ID. The proxy ID is derived from the host or username for 'superproxy' types.

Example: 

extDir, err := NewChromeExtension("192.168.1.1:8080:user:pass", "/path/to/extensions")

func NewLauncher 

func NewLauncher(opts ...BrowserOptionFunc) *launcher.Launcher

func NewStringSlice 

func NewStringSlice(raw string, fixStep int, randomStep ...bool) []string

NewStringSlice splits a raw string into a slice of strings.

Parameters:

  • raw: The input string to be split into a slice.
  • fixStep: The base number of characters for each substring.
  • randomStep: Optional boolean to enable random step mode. If true, the step size will vary randomly.

Returns:

  • A slice of strings, where each string has approximately fixStep characters.

The function splits the input string into substrings. If randomStep is enabled, the actual number of characters in each substring may vary slightly from fixStep. This variation is determined by random chance, potentially adding or subtracting 1 or 2 characters from fixStep.

func NewUserMode 

func NewUserMode(opts ...BrowserOptionFunc) (*launcher.Launcher, *rod.Browser)

func NoEcho added in v0.1.4 

func NoEcho(...any)

NoEcho is a no-op function that accepts any number of arguments and does nothing.

This function can be used as a placeholder or to explicitly ignore values. It's particularly useful in situations where you need to satisfy an interface that requires a function, but you don't actually need to do anything with the arguments.

Parameters:

  • ...any: Variadic parameter that can accept any type and any number of arguments.

Returns:

  • Nothing.

func NormalizeSliceIndex added in v0.1.4 

func NormalizeSliceIndex(length, index int) int

NormalizeSliceIndex adjusts an index to be within the bounds of a slice or array.

Parameters:

  • length: The length of the slice or array.
  • index: The input index to be normalized.

Returns:

  • An integer representing the normalized index.

If length is 0, it returns -1. For negative indices, it wraps around from the end (e.g., -1 becomes length-1). If the index is out of bounds, it normalizes to the valid range [0, length-1].

func ParseCURLCookies added in v0.1.4 

func ParseCURLCookies(raw string) ([]*proto.NetworkCookieParam, error)

ParseCURLCookies converts a cURL-formatted cookie string into NetworkCookieParam objects.

It extracts cookies from the "Cookie" header in the cURL string, parses each cookie, and creates NetworkCookieParam objects with appropriate attributes based on the URL.

Parameters:

  • raw: string - The raw cURL-formatted cookie string.

Returns:

  • []*proto.NetworkCookieParam: Slice of parsed cookie objects.
  • error: Error if parsing fails or no cookies are found.

Usage: 

cookies, err := ParseCURLCookies(curlString)

func QuitIfY 

func QuitIfY()

QuitIfY prompts the user to quit or continue the program. It displays an interactive confirmation dialog with the current date, asking if the user wants to quit (Y) or continue (N). If the user chooses to quit, the program exits immediately. The default option is set to continue (N).

func QuitOnTimeout 

func QuitOnTimeout(args ...int)

QuitOnTimeout waits for a specified number of seconds before quitting the program.

  • If no argument is provided, it uses a default timeout of 3 seconds.
  • If the timeout is 0, it returns immediately without any action.
  • If the timeout is negative, it prompts the user to quit or continue.
  • Otherwise, it displays a spinner for the specified duration before quitting.

Parameters:

  • args: Variadic parameter for timeout in seconds. If not provided, default is used.

func RandFloatX1k 

func RandFloatX1k(minSec, maxSec float64) int

RandFloatX1k converts a range of seconds to random milliseconds. Returns an integer between minSec*1000 and maxSec*1000. If minSec equals maxSec, it returns minSec * 1000. Uses math.Round for accurate conversion.

func RandSleep 

func RandSleep(minSec, maxSec float64, msg ...string) int

RandSleep sleeps for a random duration between minSec and maxSec seconds. Returns the actual sleep duration in milliseconds.

func RandSleepNap added in v0.1.3 

func RandSleepNap() (sleepMills int)

RandSleepNap sleeps for a random duration between 1 and 2 seconds. It returns the actual sleep duration in milliseconds.

func RandSleepPT1Ms added in v0.1.3 

func RandSleepPT1Ms() (sleepMills int)

RandSleepPT1Ms sleeps for a random duration between 0.01 and 0.1 milliseconds. It returns the actual sleep duration in milliseconds.

func RetryIn3 added in v0.1.4 

func RetryIn3(retryableFunc retry.RetryableFunc) error

RetryIn3 attempts to execute a retryable function up to 3 times with a 1-second delay between attempts.

Parameters:

  • retryableFunc: A function of type retry.RetryableFunc to be executed with retry logic.

Returns:

  • An error if all retry attempts fail, or nil if the function succeeds within 3 attempts.

This function uses the retry.Do method with the following options:

  • LastErrorOnly(true): Only the last error is returned if all attempts fail.
  • Attempts(3): The function will be attempted a maximum of 3 times.
  • Delay(time.Second*1): There's a 1-second delay between each attempt.

func SleepN added in v0.1.4 

func SleepN(n float64) int

SleepN sleeps for a random duration between n and n*1.1 seconds. It returns the actual sleep duration in milliseconds.

func SleepPT100Ms added in v0.1.4 

func SleepPT100Ms()

SleepPT100Ms sleeps 0.1~0.2s

func SleepPT250Ms added in v0.1.4 

func SleepPT250Ms()

SleepPT250Ms sleeps 0.25~0.3s

func SleepPT500Ms added in v0.1.4 

func SleepPT500Ms()

SleepPT500Ms sleeps 0.5~0.6s

func SleepWithSpin 

func SleepWithSpin(timeInSeconds int, args ...string)

SleepWithSpin displays a spinner for a specified number of seconds with an optional message. It shows a visual indicator of waiting and then exits after the specified duration.

Parameters:

  • timeInSeconds: The duration to display the spinner, in seconds.
  • args: Optional. A custom message to display. If not provided, a default message is used.

func StrAorB 

func StrAorB(a, b string) string

StrAorB returns the first non-empty string between two input strings.

Parameters:

  • a: The first string to check.
  • b: The second string to check.

Returns:

  • If 'a' is not empty, it returns 'a'.
  • If 'a' is empty, it returns 'b', regardless of whether 'b' is empty or not.

This function is useful for providing a fallback value when the primary string might be empty.

func StrToNumChars 

func StrToNumChars(raw string, keptChars string) string

StrToNumChars extracts numeric characters and specified kept characters from a string.

Parameters:

  • raw: The input string to be processed.
  • keptChars: A string containing additional characters to be kept along with digits.

Returns:

  • A string containing only digits and the specified kept characters.

This function creates a regular expression pattern to match digits and the specified kept characters, finds all matching substrings in the input, and joins them together.

func Stringify 

func Stringify(data interface{}) (string, error)

Stringify converts an interface{} to its JSON string representation.

Parameters:

  • data: The interface{} to be converted to a JSON string.

Returns:

  • A string containing the JSON representation of the input data.
  • An error if the conversion to JSON fails.

This function uses json.Marshal to convert the input data to a JSON byte slice, then converts the byte slice to a string. If the marshaling process fails, it returns an empty string and the error from json.Marshal.

func TermScanf added in v0.1.4 

func TermScanf(msg ...string) string

TermScanf prompts the user for text input and returns the input as a string. It uses pterm's DefaultInteractiveTextInput to display the prompt and capture user input.

Parameters:

  • msg: Optional. A custom message for the input prompt. If not provided, a default prompt is used.

Returns:

  • string: The user's input as a string.

func TermScanfInt added in v0.1.4 

func TermScanfInt(msg ...string) int

TermScanfInt prompts the user for numeric input and returns the input as an integer. It uses pterm's DefaultInteractiveTextInput to display the prompt and capture user input, then converts the input to an integer using cast.ToInt.

Parameters:

  • msg: Optional. A custom message for the input prompt. If not provided, a default prompt is used.

Returns:

  • int: The user's input converted to an integer.

Types

type Bot 

type Bot struct {
	// UniqueID is the identifier of a bot.
	UniqueID string
	// contains filtered or unexported fields
}

func NewBot 

func NewBot(options ...BotOption) *Bot

func NewBotDefault 

func NewBotDefault(options ...BotOption) *Bot

NewBotDefault creates a bot with a default launcher/browser. The launcher/browser passed in will be ignored.

func NewBotForDebug 

func NewBotForDebug(options ...BotOption) *Bot

NewBotForDebug creates a bot which launches browser with option: `--show-paint-rects`,

refer: https://peter.sh/experiments/chromium-command-line-switches/#show-paint-rects

func NewBotHeadless added in v0.1.4 

func NewBotHeadless(options ...BotOption) *Bot

NewBotHeadless sets headless with NewBotDefault

func NewBotUserMode 

func NewBotUserMode(options ...BotOption) *Bot

NewBotUserMode creates a bot that connects to the system Chrome browser.r

func NewBotWithOptionsOnly 

func NewBotWithOptionsOnly(options ...BotOption) *Bot

NewBotWithOptionsOnly creates a new Bot instance with options only, without creating a page.

func (*Bot) AcceptCookies added in v0.1.4 

func (b *Bot) AcceptCookies(cookies ...string) error

AcceptCookies attempts to accept cookies by clicking elements matching the provided selectors.

Parameters:

  • cookies: Variadic string parameter of CSS selectors for cookie acceptance elements.

Returns:

  • error: nil if at least one selector was clicked and page loaded successfully, otherwise an error.

The function tries all selectors even if some fail, logging warnings for individual failures. It returns an error if no selectors were clicked or if the final page load fails.

func (*Bot) ActivateLastOpenedPage added in v0.1.4 

func (b *Bot) ActivateLastOpenedPage(pagesBefore rod.Pages, retry int) error

func (*Bot) ActivatePage added in v0.1.4 

func (b *Bot) ActivatePage(page *rod.Page) error

ActivatePage activates a page instead of current.

func (*Bot) ActivatePageByURLRegex 

func (b *Bot) ActivatePageByURLRegex(jsRegex string, retry int) error

ActivatePageByURLRegex

usage: 

if v := regexStr; v != "" {
	return c.activatePageByURLRegex(v)
}

func (*Bot) AllAttrs 

func (b *Bot) AllAttrs(selector string, opts ...ElemOptionFunc) ([]string, error)

func (*Bot) AllElementsAttrMap 

func (b *Bot) AllElementsAttrMap(elems []*rod.Element, opts ...ElemOptionFunc) []map[string]string

AllElementsAttrMap retrieves specified attributes from multiple elements.

This function iterates over a slice of Rod elements and extracts specified attributes from each element, returning the results as a slice of maps.

Parameters:

  • elems: A slice of *rod.Element pointers representing the elements to process.
  • opts: Optional variadic ElemOptionFunc arguments to customize attribute retrieval. Typically, WithAttrMap is used to specify which attributes to retrieve.

Returns:

  • []map[string]string: A slice of maps, where each map represents an element's attributes. The keys in the map are the attribute names specified in WithAttrMap, and the values are the corresponding attribute values from the element.

Behavior:

  1. Iterates through each element in the input slice.
  2. For each element, calls ElementAttrMap to retrieve the specified attributes.
  3. Compiles the results into a slice of maps, one map per element.

Options:

  • WithAttrMap(map[string]string): Specifies which attributes to retrieve and how to name them in the result map. The key is the name in the result, the value is the actual attribute name to retrieve from the element.

Usage example: 

elements, _ := b.Elems("a.external-link")
attrMap := map[string]string{
    "text": "innerText",
    "link": "href",
    "title": "title",
}
results := b.AllElementsAttrMap(elements, WithAttrMap(attrMap))
for _, elem := range results {
    fmt.Printf("Link: %s, Text: %s, Title: %s\n", elem["link"], elem["text"], elem["title"])
}

Note:

  • This function is useful for bulk attribute retrieval from multiple elements, especially when you need to collect the same set of attributes from each element.
  • If an attribute specified in WithAttrMap is not present on an element, its value in the result map will be an empty string.
  • Any errors encountered while retrieving attributes from individual elements are logged as warnings but do not stop the process for other elements.

See also:

  • ElementAttrMap: For retrieving attributes from a single element.
  • Elems: For selecting multiple elements to use with this function.

func (*Bot) AnyElem 

func (b *Bot) AnyElem(selectors []string, opts ...ElemOptionFunc) (string, error)

AnyElem attempts to find any element from a list of selectors within a specified timeout.

This function is useful when you're looking for one of several possible elements on a page, and you don't know which one will appear first or at all.

Parameters:

  • selectors: A slice of strings, each representing a CSS selector or a text-based selector (using SEP format) to search for.
  • opts: Optional variadic ElemOptionFunc arguments to customize the search behavior.

Returns:

  • string: The selector of the first element found.
  • error: An error if no element could be found within the timeout period or if other issues occurred during the search.

Behavior:

  1. Sets up default options (timeout: MediumToSec, retries: 1) which can be overridden.
  2. Uses Rod's Race method to concurrently search for all provided selectors.
  3. Returns as soon as any of the selectors matches an element.
  4. If no element is found within the timeout, it retries based on the retry option.

Options:

  • WithTimeout(duration): Sets the maximum time to wait for any element to appear.
  • WithRetries(count): Sets the number of times to retry the entire search if no element is found.
  • Other standard ElemOptions can also be used and will be applied to each selector search.

Usage example: 

selectors := []string{"#loading", ".content", "button@@@Submit"}
foundSelector, err := b.AnyElem(selectors, WithTimeout(10), WithRetries(3))
if err != nil {
    // Handle error: no element found
} else {
    fmt.Printf("Found element with selector: %s\n", foundSelector)
}

Note:

  • This function is particularly useful for handling dynamic content where different elements might appear depending on the state of the page.
  • It supports both standard CSS selectors and text-based selectors (using SEP format).
  • The function logs a debug message if finding the element takes longer than _logIfTimeout.

Error handling:

  • Returns an error if no element is found after all retries are exhausted.
  • Any errors encountered during the search process are propagated back to the caller.

See also:

  • MustAnyElem: A version of this function that panics on error.
  • Elem: For finding a single specific element.
  • AnyElemAttribute: If you need to retrieve an attribute from the found element.

func (*Bot) BlockInCleanUp added in v0.1.3 

func (b *Bot) BlockInCleanUp()

BlockInCleanUp performs cleanup and blocks execution.

func (*Bot) Browser 

func (b *Bot) Browser() *rod.Browser

func (*Bot) Cleanup 

func (b *Bot) Cleanup()

Cleanup closes the opened page and quits the browser in non-userMode. In userMode, by default it will skip cleanup.

func (*Bot) ClearStorageCookies added in v0.1.4 

func (b *Bot) ClearStorageCookies()

ClearStorageCookies calls StorageClearCookies, clears all history cookies.

func (*Bot) Click 

func (b *Bot) Click(selector string, opts ...ElemOptionFunc) error

Click finds an element using the given selector and attempts to click it.

This function performs the following steps:

  1. Locates an element matching the provided CSS selector.
  2. If found, attempts to click the element using the Bot's ClickElem method.

The function uses the Bot's configured timeout settings and other properties for element selection and interaction behavior. It supports customization through optional ElemOptionFunc arguments, which can modify both the element selection process and the click behavior.

If no element matches the selector, the function returns an ErrCannotFindSelector error. If an element is found but cannot be clicked (e.g., not interactable or covered by another element), the error from the ClickElem method is returned.

Usage: 

err := bot.Click("#submit-button")
if err != nil {
    // Handle error (element not found, not clickable, etc.)
}

This method is useful for simulating user interactions in web automation tasks, particularly when dealing with dynamic content or complex DOM structures. It encapsulates the logic of finding and interacting with elements, providing a simple interface for common click operations.

func (*Bot) ClickElem 

func (b *Bot) ClickElem(elem *rod.Element, opts ...ElemOptionFunc) error

ClickElem attempts to click the given element. It performs the following steps:

  1. Optionally highlights the element.
  2. Ensures the element is interactable (scrolls into view if necessary).
  3. Attempts to click the element using the left mouse button.
  4. If the click fails due to a timeout or invisibility, it falls back to clicking via JavaScript.

Parameters:

  • elem: The rod.Element to be clicked.
  • opts: Optional ElemOptionFunc to customize the click behavior.

Options:

  • handleCoverByEsc: If true, attempts to handle covered elements by pressing Escape.
  • highlight: If true, highlights the element before clicking.
  • clickByScript: If true, uses JavaScript to perform the click instead of simulating a mouse click.

Returns:

  • An error if the click operation fails, nil otherwise.

func (*Bot) ClickElemWithScript 

func (b *Bot) ClickElemWithScript(elem *rod.Element, opts ...ElemOptionFunc) error

ClickElemWithScript attempts to click the given element using JavaScript. This method is particularly useful when standard clicking methods fail due to element positioning, overlays, or other DOM-related issues.

The function performs the following steps:

  1. Optionally highlights the element for visual feedback.
  2. Executes a JavaScript click event on the element.
  3. Checks if the element is still interactable after the click.

Options:

  • timeout: Duration to wait for the click operation to complete (default: MediumToSec).
  • highlight: If true, highlights the element before clicking (default: true).

Note:

  • This method cancels the timeout after the initial JavaScript execution to allow for potential page reloads or redirects triggered by the click.
  • If the element becomes non-interactable (e.g., removed from DOM) after the click, it's considered a successful operation, assuming the click caused a page change.

func (*Bot) ClickWithScript 

func (b *Bot) ClickWithScript(selector string, opts ...ElemOptionFunc) error

func (*Bot) ClosePopover 

func (b *Bot) ClosePopover(sel string) (int, error)

ClosePopover attempts to close popovers matching the given selector. It finds all elements matching the selector, checks if they're interactable, highlights them, and tries to click each one.

Parameters:

  • sel: CSS selector string for the popover element(s).

Returns:

  • int: Number of popovers successfully closed.
  • error: nil if successful, otherwise an error describing the failure.

The function stops and returns an error if it encounters any issues during the process. If some popovers are closed before an error occurs, it returns the partial count along with the error.

Usage example: 

closed, err := bot.ClosePopover(".modal-close-button")
if err != nil {
    log.Printf("Error closing popovers: %v", err)
} else {
    log.Printf("Successfully closed %d popovers", closed)
}

func (*Bot) ClosePopovers added in v0.1.4 

func (b *Bot) ClosePopovers(popovers ...string) int

func (*Bot) CookieFile added in v0.1.2 

func (b *Bot) CookieFile() string

func (*Bot) CurrentURL added in v0.1.4 

func (b *Bot) CurrentURL() string

func (*Bot) CustomizePage 

func (b *Bot) CustomizePage()

func (*Bot) DOMStable added in v0.1.4 

func (b *Bot) DOMStable(d time.Duration, diff float64) error

func (*Bot) DisableImages 

func (b *Bot) DisableImages()

DisableImages blocks all image requests for the bot's browser session.

This method sets up a request interceptor that fails all requests for image resources, effectively preventing images from loading during web navigation.

Usage: 

bot.DisableImages()

Note:

  • This method is useful for reducing bandwidth usage and speeding up page loads.
  • It affects all subsequent page loads until the bot is reset or the browser is closed.

func (*Bot) DumpCookies 

func (b *Bot) DumpCookies() (string, error)

DumpCookies saves the current cookies of the bot's page to a file on disk.

This function performs the following operations:

1. Ensures the cookie file exists:

  • If b.cookieFile is empty, it generates a filename based on the current URL.
  • Creates any necessary parent directories for the cookie file.

2. Retrieves and marshals cookies:

  • Calls b.page.MustCookies() to get all cookies from the current page.
  • Marshals these cookies into JSON format.

3. Writes cookies to file:

  • Opens or creates the cookie file with 0644 permissions (rw-r--r--).
  • Writes the JSON-encoded cookie data to the file.

Returns:

  • string: The full path to the cookie file where the cookies were saved.
  • error: An error if any of the following occur:
  • Failure to ensure the cookie file (e.g., permission issues)
  • Failure to marshal cookies to JSON
  • Failure to write to the cookie file

Usage: 

filepath, err := bot.DumpCookies()
if err != nil {
    log.Fatalf("Failed to dump cookies: %v", err)
}
fmt.Printf("Cookies saved to: %s\n", filepath)

Notes:

  • This function is useful for persisting session data between bot runs.
  • The saved cookies can be later loaded using the LoadCookies method.
  • If b.cookieFile is not set, the function will automatically generate a filename based on the current URL, stored in the default or specified cookie folder.
  • Ensure the bot has necessary permissions to write to the cookie directory.

func (*Bot) DumpXHR 

func (b *Bot) DumpXHR(ptn []string, handler func(h *rod.Hijack))

DumpXHR intercepts and processes XMLHttpRequest (XHR) requests matching specified patterns.

Parameters:

  • ptn []string: URL patterns to match XHR requests. Use "*" as prefix and suffix for partial matches.
  • handler func(h *rod.Hijack): Function to process each intercepted XHR.

The method loads the full response for each matched XHR and calls the handler, allowing inspection of both request and response details without blocking the XHR.

Usage: 

bot.DumpXHR([]string{"*api/data*"}, func(h *rod.Hijack) {
    fmt.Printf("XHR: %s\n", h.Request.URL())
    fmt.Printf("Response Body: %s\n", h.Response.Body())
})

Pattern examples:

  • "*api*": Matches any URL containing "api"
  • "*example.com/api*": Matches URLs containing "example.com/api"
  • "*": Matches all XHR requests

Note:

  • Patterns are case-sensitive.
  • The handler is called after the response is loaded, allowing access to both request and response data.
  • This method is useful for logging, debugging, or analyzing specific XHR traffic during web automation.

Be cautious:

  • Overly broad patterns may intercept more requests than intended.
  • Intercepting many XHRs may impact performance. Use specific patterns when possible.
  • Modifying XHR responses can affect the functionality of the web application being automated.

func (*Bot) Elem 

func (b *Bot) Elem(selector string, opts ...ElemOptionFunc) (*rod.Element, error)

Elem gets an element by selector, with performance logging and error handling.

This function is a wrapper around the internal getElem method, providing additional functionality such as performance timing and debug logging.

Elem supports various selector strategies including:

  • Standard CSS selectors
  • Iframe-based selection (using IFrameSep)
  • Text content-based selection (using SEP)
  • Index-based selection for multiple matching elements

The function measures the time taken to find the element and logs a debug message if it exceeds _logIfTimeout seconds.

Parameters:

  • selector: string to identify the element. Can be a CSS selector, iframe selector, or text content.
  • opts: variadic ElemOptionFunc arguments to customize selection behavior.

Returns:

  • *rod.Element: pointer to the found element.
  • error: nil if successful, otherwise an error describing why the element couldn't be found.

The function respects the following options (passed via opts):

  • WithRoot: to specify a root element for the search
  • WithTimeout: to set a custom timeout for waiting for the element
  • WithIndex: to select a specific element when multiple matches are found
  • WithIframe: to specify an iframe context for the search

Example usage: 

elem, err := b.Elem("div.class", WithTimeout(5), WithIndex(2))
if err != nil {
    // Handle error
}
// Use elem

Note: This function is part of the Bot struct and is designed to be used as a public API. It includes error handling and performance logging, making it suitable for external use.

func (*Bot) ElemAttr 

func (b *Bot) ElemAttr(selector string, opts ...ElemOptionFunc) (string, error)

ElemAttr retrieves an attribute value from an element identified by a selector.

By default, retrieves 'innerText'. Use WithAttr(name) to specify a different attribute.

Example: 

href, err := b.ElemAttr("a.link", WithAttr("href"))

Note:

  • This function is useful for quickly retrieving an attribute value when you only need to perform this single operation on an element.
  • For multiple operations on the same element, it's more efficient to first select the element using Elem, then perform operations on the returned element.

See also:

  • Elem: For selecting an element without immediately retrieving an attribute.
  • MustElemAttr: For a version of this function that panics on error.
  • ElementAttr: For retrieving an attribute from an already selected element.

func (*Bot) ElemByIndex 

func (b *Bot) ElemByIndex(selector string, index int) (*rod.Element, error)

ElemByIndex retrieves an element at a specific index from a list of elements matching the selector.

Parameters:

  • selector: CSS selector string
  • index: desired element index (supports negative indexing)

Returns:

  • *rod.Element: element at specified index, or nil if index out of range
  • error: if elements can't be found or other issues occur

Note: This function doesn't wait for elements to appear.

func (*Bot) ElemByText 

func (b *Bot) ElemByText(selector string, opts ...ElemOptionFunc) (*rod.Element, error)

ElemByText finds an element by its text content using a combination of CSS selector and text matching.

This function provides a powerful way to select elements based on both their structural properties (via CSS selectors) and their text content. It supports two types of text matching: partial (contains) and exact match, with an option for case-insensitive matching.

Selector format:

  • Partial match: "cssSelector@@@textContent" Finds elements matching the CSS selector that contain the specified text.
  • Exact match: "cssSelector@@@---@@@textContent" Finds elements matching the CSS selector with text that exactly matches the specified content.

Parameters:

  • selector: A string combining CSS selector and text content, separated by SEP ("@@@"). The CSS selector comes first, followed by one or two SEP, and then the text to match.
  • opts: Optional variadic ElemOptionFunc arguments to customize the search behavior.

Returns:

  • *rod.Element: A pointer to the found element.
  • error: An error if the element couldn't be found or if any other issues occurred during the search.

Behavior:

  1. Parses the selector string to extract CSS selector and text content.
  2. For exact match (three parts after splitting), constructs a regex pattern.
  3. Uses Rod's ElementR method to perform the element search with text matching.
  4. Applies specified timeout to the search operation.

Options:

  • WithRoot(root): Specifies a root element to search within, limiting the scope of the search.
  • WithTimeout(timeout): Sets a duration to wait for the element to appear.
  • WithCaseInsensitive(): Enables case-insensitive matching for exact match selectors.

Example usage: 

// Partial match (case-sensitive)
elem, err := b.ElemByText("button@@@Submit")

// Exact match (case-sensitive)
elem, err := b.ElemByText("div@@@---@@@Exact Text")

// Exact match (case-insensitive)
elem, err := b.ElemByText("span@@@---@@@Case Insensitive", WithCaseInsensitive())

// With custom timeout
elem, err := b.ElemByText("p@@@Content", WithTimeout(10))

// Search within a specific root element
rootElem, _ := b.Elem("#root-container")
elem, err := b.ElemByText("li@@@Item", WithRoot(rootElem))

Internal workings:

  • For partial match, it uses Rod's built-in contains text matching.
  • For exact match, it constructs a regex pattern: /^textContent$/ With case-insensitive option, it adds the 'i' flag: /^textContent$/i

Error handling:

  • Returns an error if the element is not found within the specified timeout.
  • Propagates any errors encountered during the Rod operations.

Note:

  • This function is particularly useful for selecting elements in dynamic content where IDs or classes might change, but text content remains consistent.
  • The case-insensitive option only applies to exact match selectors.
  • When using WithRoot, ensure the root element is already present in the DOM.

See also:

  • Elem: For standard CSS selector-based element selection.
  • ElemsByText: For selecting multiple elements by text content.

func (*Bot) ElementAttr 

func (b *Bot) ElementAttr(elem *rod.Element, opts ...ElemOptionFunc) (string, error)

func (*Bot) ElementAttrMap 

func (b *Bot) ElementAttrMap(elem *rod.Element, opts ...ElemOptionFunc) map[string]string

ElementAttrMap retrieves specified attributes from a single element.

This function extracts multiple attributes from a given Rod element and returns them as a map of key-value pairs.

Behavior:

  1. Initializes an empty result map.
  2. Applies the provided options to determine which attributes to retrieve.
  3. For each specified attribute, calls getElementAttr to retrieve its value.
  4. Compiles the results into a map.

Options:

  • WithAttrMap(map[string]string): Specifies which attributes to retrieve and how to name them in the result map. The key is the name in the result, the value is the actual attribute name to retrieve from the element.

Usage example: 

element, _ := b.Elem("a.external-link")
attrMap := map[string]string{
    "text": "innerText",
    "link": "href",
    "title": "title",
}
result := b.ElementAttrMap(element, WithAttrMap(attrMap))
fmt.Printf("Link: %s, Text: %s, Title: %s\n", result["link"], result["text"], result["title"])

Error handling:

  • If an attribute cannot be retrieved, a warning is logged, and the attribute is omitted from the result map.
  • The function does not return an error; it attempts to retrieve as many attributes as possible and logs warnings for any that fail.

Note:

  • This function is useful when you need to retrieve multiple attributes from a single element in one operation.
  • If an attribute specified in WithAttrMap is not present on the element, it will be omitted from the result map and a warning will be logged.
  • The 'innerText' attribute is handled specially and retrieved using the Text() method.

See also:

  • getElementAttr: The underlying method used to retrieve individual attributes.
  • AllElementsAttrMap: For retrieving attributes from multiple elements at once.

func (*Bot) Elems 

func (b *Bot) Elems(selector string, opts ...ElemOptionFunc) ([]*rod.Element, error)

Elems retrieves all elements matching the given selector immediately.

This function is designed to find multiple elements on the page that match the provided selector. It has specific behavior for different types of selectors and includes built-in error checking.

Behavior:

  1. If the selector is empty, it returns an ErrSelectorEmpty error.
  2. If the selector contains the SEP ("@@@"), it calls ElemsByText for text-based selection.
  3. Otherwise, it calls the internal elems method for standard CSS selector-based selection.

The function does not wait for elements to appear and returns the current state of the page immediately.

Parameters:

  • selector: A string representing the selector to locate the elements.
  • For standard selection: CSS selector (e.g., "div.class", "#id")
  • For text-based selection: Selector with SEP (e.g., "div@@@text")
  • opts: Optional ElemOptionFunc arguments to customize the element search behavior. These are passed through to the underlying selection methods.

Returns:

  • []*rod.Element: A slice of pointers to the found elements. May be empty if no elements are found.
  • error: An error if elements couldn't be found or if any other issues occurred. Specifically returns ErrSelectorEmpty if the selector is an empty string.

Error handling:

  • Returns ErrSelectorEmpty if the selector is an empty string.
  • Other errors may be returned from the underlying ElemsByText or elems methods.

Usage examples:

  1. Standard CSS selector: elems, err := b.Elems("div.class")

  2. Text-based selector: elems, err := b.Elems("button@@@Submit")

Note:

  • This function is part of the Bot struct and is designed to be used as a public API.
  • It's suitable for cases where you need to find all matching elements without waiting.
  • For more specific element selection (e.g., by index), consider using other methods like ElemByIndex.

See also:

  • ElemsByText: For text-based element selection.
  • elems: The internal method used for standard CSS selector-based selection.

func (*Bot) ElemsByText 

func (b *Bot) ElemsByText(selector string, opts ...ElemOptionFunc) ([]*rod.Element, error)

ElemsByText finds all elements by their text content using a combination of CSS selector and text matching.

This function retrieves multiple elements that match both a CSS selector and contain specific text content. It's useful for scenarios where you need to find all instances of elements with certain text.

Selector format:

  • "cssSelector@@@textContent" The CSS selector and text content are separated by SEP ("@@@").

Parameters:

  • selector: A string combining CSS selector and text content, separated by SEP. Format: "cssSelector@@@textContent"
  • opts: Optional variadic ElemOptionFunc arguments to customize the search behavior.

Returns:

  • []*rod.Element: A slice of pointers to all found elements. May be empty if no elements are found.
  • error: An error if elements couldn't be found or if any other issues occurred during the search.

Behavior:

  1. Splits the selector by SEP to separate CSS selector and text content.
  2. Uses the internal elems method to find all elements matching the CSS selector.
  3. Filters these elements to find all that contain the specified text content.

Error handling:

  • Returns an error if the initial elems call fails.
  • Returns an empty slice and nil error if no elements match both criteria.

Options: 

Options are passed through to both the elems call and text matching. Notable options include:
- WithTimeout: Sets a duration to wait for elements to appear.
- WithRoot: Specifies a root element to search within, limiting the scope of the search.
- WithCaseInsensitive: Enables case-insensitive text matching.

Example usage: 

// Find all buttons containing "Submit"
elems, err := b.ElemsByText("button@@@Submit")

// Find all list items with specific text, case-insensitive
elems, err := b.ElemsByText("li@@@Item Text", WithCaseInsensitive())

Note:

  • This function is useful for finding multiple elements in dynamic content where both structure (CSS) and content (text) need to match.

See also:

  • ElemByText: For selecting a single element by text content.
  • Elems: For selecting multiple elements by CSS selector only.

func (*Bot) EnsureNonNilElem added in v0.1.5 

func (b *Bot) EnsureNonNilElem(sel string, opts []ElemOptionFunc) (*rod.Element, error)

EnsureNonNilElem retrieves a non-nil element by selector, ensuring its existence and validity.

This function combines element retrieval with a nil check, providing a robust way to fetch an element and verify its presence in the DOM. It's particularly useful for operations that require a guaranteed non-nil element, such as form interactions or complex UI manipulations.

Returns:

  • *rod.Element: A pointer to the found Rod element, guaranteed to be non-nil if no error is returned.
  • error: An error is returned in the following cases: 1. If the underlying Elem method fails to find the element (e.g., timeout, invalid selector). 2. If the element is found but is nil (which could happen in certain edge cases with dynamic content).

Behavior:

  1. Calls the underlying Elem method with the provided selector and options.
  2. If Elem returns an error, that error is immediately propagated back to the caller.
  3. If Elem succeeds but returns a nil element, an appropriate error (likely ErrCannotFindSelector) is returned.
  4. If a non-nil element is found, it is returned along with a nil error.

Usage example: 

elem, err := b.EnsureNonNilElem("#login-button", []ElemOptionFunc{WithTimeout(10)})
if err != nil {
    if errors.Is(err, ErrCannotFindSelector) {
        log.Println("Login button not found or is nil")
    } else {
        log.Println("Error finding login button:", err)
    }
    return
}
// Safely interact with the non-nil element
err = elem.Click()

Error handling:

  • Propagates any errors from the underlying Elem method, which could include timeout errors, selector syntax errors, or other Rod-specific errors.
  • Returns a specific error (likely ErrCannotFindSelector) if the element is found but nil, allowing for differentiated handling of "not found" vs "found but nil" scenarios.

Best practices:

  • Use this function when you need to ensure an element's presence before performing critical operations.
  • Combine with appropriate timeout options to balance between waiting for dynamic content and failing fast.
  • Handle both the general error case and the specific ErrCannotFindSelector case in your error handling logic.

Note:

  • This function is essential for robust web automation scripts, as it helps prevent nil pointer dereference errors that could occur if an element is unexpectedly absent.
  • It's particularly valuable in scenarios with dynamic content where elements might not be immediately available or could be temporarily removed from the DOM.

See also:

  • Elem: The underlying method used for basic element retrieval.
  • ElemOptions: For understanding the available options for element selection.
  • ErrCannotFindSelector: The likely error type returned for nil elements.

func (*Bot) Eval added in v0.1.3 

func (b *Bot) Eval(script string) (*proto.RuntimeRemoteObject, error)

func (*Bot) FocusAndHighlight 

func (b *Bot) FocusAndHighlight(elem *rod.Element)

FocusAndHighlight focuses on an element and optionally highlights it.

This method provides visual emphasis on a specified web element, either by focusing on it or by scrolling to it and applying a highlight effect. The behavior is determined by the Bot's configuration.

Parameters:

  • elem: A pointer to the rod.Element to focus on or highlight.

Behavior:

  1. If Bot.highlightTimes == 0: - The method only focuses on the element using elem.Focus(). - No scrolling or highlighting is performed.
  2. If Bot.highlightTimes > 0: - The method scrolls to the element using Bot.ScrollToElement(elem). - Then it calls Bot.HighlightElem(elem) to apply a visual highlight effect.

The HighlightElem method:

  • Applies a pulsating highlight effect to the element.
  • The number of pulses is determined by Bot.highlightTimes.
  • Uses a predefined style (_style1) for the highlight effect.
  • Runs asynchronously in a separate goroutine.

Use cases:

  • Automated testing: To visually verify which elements are being interacted with.
  • Debugging: To easily spot the elements being processed during script execution.
  • Demonstrations: To create visual cues when showcasing web automation.

Note:

  • This method does not return any value or error.
  • The highlight effect, if applied, does not block the execution of subsequent code.

Example usage: 

bot := &Bot{highlightTimes: 3}
element := page.MustElement("#some-id")
bot.FocusAndHighlight(element)

In this example, the element will be scrolled into view and highlighted 3 times.

func (*Bot) GetElemBox 

func (b *Bot) GetElemBox(elem *rod.Element) (*proto.DOMRect, error)

func (*Bot) GetScrollHeight 

func (b *Bot) GetScrollHeight() (float64, error)

func (*Bot) GetWindowInnerHeight 

func (b *Bot) GetWindowInnerHeight() float64

func (*Bot) Highlight 

func (b *Bot) Highlight(elem *rod.Element, show, hide float64, style string, count int) float64

Highlight applies a pulsating visual effect to a specified web element.

This method creates a series of highlight pulses on the given element by alternating between a custom style and the element's original style.

Parameters:

  • elem: A pointer to the rod.Element to be highlighted.
  • show: Duration (in seconds) for which the highlight is visible.
  • hide: Duration (in seconds) for which the highlight is hidden.
  • style: CSS style string to be applied for the highlight effect.
  • count: Number of times to repeat the highlight effect. If 0, uses Bot.highlightTimes.

Returns:

  • float64: The total duration (in seconds) taken to complete the highlight effect.

Behavior:

  1. If Bot.highlightTimes == 0 or elem is nil, the method returns immediately.
  2. Retrieves the original style of the element.
  3. Applies the highlight effect 'count' number of times: - Sets the element's style to the provided highlight style. - Waits for 'show' duration (with slight randomization). - Reverts the element's style to its original state. - Waits for 'hide' duration (with slight randomization).
  4. Measures and returns the total time taken for the entire process.

Notes:

  • The method uses JavaScript evaluation to modify the element's style.
  • There's a small randomization (±0.05s) added to show/hide durations for a more natural effect.
  • This method is synchronous and will block until all highlight pulses are completed.

Example usage: 

bot := &Bot{highlightTimes: 3}
element := page.MustElement("#some-id")
duration := bot.Highlight(element, 0.5, 0.3, "border: 2px solid red;", 0)
fmt.Printf("Highlighting took %.2f seconds\n", duration)

func (*Bot) Hijack 

func (b *Bot) Hijack(patterns []string, networkResourceType proto.NetworkResourceType, handler HijackHandler, continueRequest bool)

Hijack intercepts network requests that match specified patterns and resource types. 

This method sets up a network request interceptor for the bot's browser, allowing
inspection and manipulation of requests based on URL patterns and resource types.

Parameters:

  1. patterns []string - A slice of URL patterns to match against request URLs. - Supports glob patterns (e.g., "*.example.com"). - If empty, the method returns without setting up interception.

  2. networkResourceType proto.NetworkResourceType - Specifies the type of network resource to intercept (e.g., XHR, Image, Script). - Only requests of this type will be intercepted.

  3. handler HijackHandler - A function of type func(*rod.Hijack) that processes intercepted requests. - Called for each request that matches both the pattern and resource type criteria.

  4. continueRequest bool - Determines whether to continue the request after handler execution. - If true: The request continues its normal flow after handler execution. - If false: The request is terminated after handler execution, unless explicitly continued in the handler.

Implementation Details:

  • Uses rod's pattern matching for URLs, which supports glob patterns.
  • Only requests matching both pattern and resource type are intercepted.
  • Non-matching requests (by type) are automatically continued.

Use Cases:

  • Intercepting specific types of requests (e.g., only XHR or Image requests).
  • Modifying responses for certain domains or URL patterns.
  • Implementing fine-grained control over network traffic during web automation.

Example: 

bot.Hijack(
    []string{"api.example.com/*"},
    proto.NetworkResourceTypeXHR,
    func(h *rod.Hijack) {
        fmt.Printf("Intercepted XHR: %s\n", h.Request.URL().String())
        // Modify XHR response
        h.Response.SetBody([]byte(`{"status":"mocked"}`))
    },
    true
)

Notes:

  • This method starts a goroutine, so it returns immediately.
  • Be cautious with resource-intensive operations in the handler to avoid performance issues.
  • Modifying responses can affect page functionality; use with care.

See Also:

  • Bot.HijackAny: For intercepting requests based on URL content without type restrictions.
  • proto.NetworkResourceType documentation for available resource types.
  • rod.Hijack documentation for available methods on the Hijack object.

func (*Bot) HijackAny 

func (b *Bot) HijackAny(resources []string, handler HijackHandler, continueRequest bool)

HijackAny intercepts network requests that contain any of the specified resource strings in their URLs. 

This method sets up a network request interceptor for the bot's browser, allowing
inspection and manipulation of requests based on URL content.

Parameters:

  1. resources []string - A slice of strings to match against request URLs. - If a request URL contains any of these strings, it will be intercepted. - Case-sensitive matching is used. - If empty, the method returns without setting up interception.

  2. handler HijackHandler - A function of type func(*rod.Hijack) that processes intercepted requests. - Called for each request that matches the resources criteria. - Can inspect request details and modify the response.

  3. continueRequest bool - Determines the flow after handler execution. - If true: The request continues its normal flow after handler execution. - If false: The request is terminated after handler execution, unless explicitly continued in the handler.

Use Cases:

  • Monitoring specific API calls or resource loads.
  • Modifying responses for testing or mocking purposes.
  • Blocking unwanted resources (e.g., ads, trackers).
  • Logging network activity for debugging.

Example: 

bot.HijackAny(
    []string{"api/users", "images"},
    func(h *rod.Hijack) {
        fmt.Printf("Intercepted: %s\n", h.Request.URL().String())
        if strings.Contains(h.Request.URL().String(), "api/users") {
            // Modify API response
            h.Response.SetBody([]byte(`{"users":[]}`))
        }
    },
    true
)

Notes:

  • This method starts a goroutine, so it returns immediately.
  • Be cautious with resource-intensive operations in the handler to avoid performance issues.
  • Modifying responses can affect page functionality; use with care.

See Also:

  • Bot.Hijack: For more specific request targeting by pattern and resource type.
  • rod.Hijack documentation for available methods on the Hijack object.

func (*Bot) IframeElem 

func (b *Bot) IframeElem(iframe, selector string, opts ...ElemOptionFunc) (*rod.Element, error)

IframeElem finds and returns an element within an iframe.

This function performs a two-step element selection: 1. It locates the iframe element in the main document. 2. It then searches for the specified element within the iframe's content.

Parameters:

  • iframe: A string selector to locate the iframe element in the main document. This should be a valid CSS selector targeting the iframe.
  • selector: A string CSS selector to find the desired element within the iframe's content.
  • opts: Optional variadic ElemOptionFunc arguments to customize the element search behavior. These options are applied to both the iframe selection and the inner element selection.

Returns:

  • *rod.Element: A pointer to the found element within the iframe.
  • error: An error if either the iframe or the inner element couldn't be found, or if any other issues occurred during the process.

The function uses the Bot's Elem method to find the iframe, ensuring all standard element selection options and timeouts are respected. It then creates a new Rod Frame from the iframe element and searches for the inner element within this frame.

Usage example: 

elem, err := b.IframeElem("#my-iframe", ".content-class")
if err != nil {
    // Handle error
}
// Use elem

Note: This function is useful for interacting with elements that are within iframes, which are not directly accessible from the main document context.

func (*Bot) Input 

func (b *Bot) Input(sel, text string, opts ...ElemOptionFunc) (string, error)

Input first clear all content, and then input text content.

func (*Bot) InputSelect 

func (b *Bot) InputSelect(sel, text string, opts ...ElemOptionFunc) error

func (*Bot) Interactable 

func (b *Bot) Interactable(elem *rod.Element) error

func (*Bot) LoadCookies 

func (b *Bot) LoadCookies(filepath string) ([]*proto.NetworkCookieParam, error)

LoadCookies loads cookies from a file or from previously stored cURL data and converts them into a format usable by the bot.

Parameters:

  • filepath: string - The path to the cookie file. If empty, the function will attempt to generate a filename based on the bot's current URL.

Returns:

  • []*proto.NetworkCookieParam: A slice of NetworkCookieParam objects representing the loaded cookies.
  • error: An error if the loading process fails at any step.

Function behavior: 1. Retrieves raw cookie data:

  • If copyAsCURLCookies is not empty, it uses this data.
  • Otherwise, it reads from the specified file or an auto-generated file based on the current URL.

2. Determines the format of the cookie data:

  • If the data is in JSON format, it parses it as proto.NetworkCookie objects.
  • If not JSON, it assumes cURL format and parses accordingly.

3. For JSON format:

  • Unmarshals the data into proto.NetworkCookie objects.
  • Converts these into proto.NetworkCookieParam objects, filling in additional fields.

4. For cURL format:

  • Calls ParseCURLCookies to handle the conversion.

Usage: 

cookies, err := bot.LoadCookies("path/to/cookies.json")
if err != nil {
    log.Fatalf("Failed to load cookies: %v", err)
}
// Use the cookies with the bot...

Notes:

  • This function is flexible and can handle both JSON-formatted cookie files (typically saved by DumpCookies) and cURL-formatted cookie strings.
  • When loading from a file, ensure the bot has read permissions for the cookie file.
  • If filepath is empty and the bot can't determine the current URL, this will result in an error.
  • The returned cookies are in a format ready to be used with the bot's page or browser instance.
  • Any parsing or file reading errors will be returned, allowing the caller to handle them appropriately.

See also: - DumpCookies: For saving cookies to a file. - ParseCURLCookies: For the specific handling of cURL-formatted cookie strings.

func (*Bot) LogTimeSpent added in v0.1.4 

func (b *Bot) LogTimeSpent(start time.Time, skips ...int)

func (*Bot) MakeElemInteractable added in v0.1.4 

func (b *Bot) MakeElemInteractable(elem *rod.Element, byEsc bool) error

func (*Bot) MarkElem 

func (b *Bot) MarkElem(timeout time.Duration, elem *rod.Element, style string)

MarkElem applies a temporary visual marker to a specified web element.

This method adds a visual style to an element for a specified duration, then reverts the element to its original style. It's useful for temporarily highlighting or marking elements during automated browsing or testing.

Parameters:

  • timeout: Duration for which the marker style should be applied.
  • elem: A pointer to the rod.Element to be marked.
  • style: CSS style string to be applied as the marker. If empty, a default style is used.

Behavior:

  1. Captures the original style of the element (currently commented out).
  2. Applies the specified style (or default if not provided) to the element.
  3. Waits for the specified timeout duration.
  4. Reverts the element's style to its original state (deferred operation).

Default Style: 

If no style is provided, it uses the _style constant:
`box-shadow: rgb(255, 156, 85) 0px 0px 0px 8px, rgb(255, 85, 85) 0px 0px 0px 10px;
 transition: all 0.5s ease-in-out; animation-delay: 0.1s;`

Notes:

  • This method is synchronous and will block for the duration of the timeout.
  • The original style restoration is deferred, ensuring it occurs even if there's a panic.
  • Currently, the original style capture is commented out, which means the element will revert to having no inline style after the timeout.

Example usage: 

bot := &Bot{}
element := page.MustElement("#some-id")
bot.MarkElem(5*time.Second, element, "border: 3px solid blue;")

This will mark the element with a blue border for 5 seconds.

func (*Bot) MarkElems 

func (b *Bot) MarkElems(timeout time.Duration, elems ...*rod.Element)

MarkElems applies a temporary visual marker to multiple web elements concurrently.

Parameters:

  • timeout: Duration for which the marker style should be applied.
  • elems: Variadic parameter of rod.Element pointers to be marked.

Behavior:

  • Applies a purple box-shadow style to each element for the specified duration.
  • Marking is done concurrently for each element using goroutines.
  • Uses MarkElem internally to apply the style to each element.

Note:

  • This method returns immediately, with marking occurring asynchronously.
  • The default style is a purple box-shadow.

func (*Bot) MustAcceptCookies 

func (b *Bot) MustAcceptCookies(cookies ...string)

func (*Bot) MustActivatePage added in v0.1.4 

func (b *Bot) MustActivatePage(page *rod.Page)

func (*Bot) MustAllElemAttrs added in v0.1.5 

func (b *Bot) MustAllElemAttrs(selector string, opts ...ElemOptionFunc) []string

func (*Bot) MustAnyElem 

func (b *Bot) MustAnyElem(selectors []string, opts ...ElemOptionFunc) string

func (*Bot) MustClick 

func (b *Bot) MustClick(selector string, opts ...ElemOptionFunc)

func (*Bot) MustClickAll 

func (b *Bot) MustClickAll(selectors []string, opts ...ElemOptionFunc)

func (*Bot) MustClickAndWait 

func (b *Bot) MustClickAndWait(selector string, opts ...ElemOptionFunc)

func (*Bot) MustClickElem 

func (b *Bot) MustClickElem(elem *rod.Element, opts ...ElemOptionFunc)

func (*Bot) MustClickElemAndWait 

func (b *Bot) MustClickElemAndWait(elem *rod.Element, opts ...ElemOptionFunc)

func (*Bot) MustClickSequentially added in v0.1.4 

func (b *Bot) MustClickSequentially(selectors ...string)

MustClickSequentially clicks on a series of elements specified by selectors in order. It waits 500ms between clicks and ensures DOM stability after all clicks.

Parameters:

  • selectors: Variadic string parameter of CSS selectors to be clicked in sequence.

This function will panic if any click operation fails.

func (*Bot) MustDOMStable added in v0.1.4 

func (b *Bot) MustDOMStable()

func (*Bot) MustElem 

func (b *Bot) MustElem(selector string, opts ...ElemOptionFunc) *rod.Element

func (*Bot) MustElemAttr 

func (b *Bot) MustElemAttr(selector string, opts ...ElemOptionFunc) string

func (*Bot) MustElems 

func (b *Bot) MustElems(selector string, opts ...ElemOptionFunc) []*rod.Element

func (*Bot) MustElemsForSelectors added in v0.1.5 

func (b *Bot) MustElemsForSelectors(selectors []string, opts ...ElemOptionFunc) []*rod.Element

func (*Bot) MustEval 

func (b *Bot) MustEval(script string) string

MustEval a wrapper with MediumTo to rod.Page.MustEval

refer: https://github.com/go-rod/rod/blob/main/examples_test.go#L53 

@param script `() => console.log("hello world")` or "`(a, b) => a + b`, 1, 2"
@return string

func (*Bot) MustIframeElem 

func (b *Bot) MustIframeElem(iframe, selector string, opts ...ElemOptionFunc) *rod.Element

func (*Bot) MustInput 

func (b *Bot) MustInput(sel, text string, opts ...ElemOptionFunc) string

func (*Bot) MustOpen 

func (b *Bot) MustOpen(uri string)

func (*Bot) MustPressEscape 

func (b *Bot) MustPressEscape(elem *rod.Element)

func (*Bot) MustScrollToBottom 

func (b *Bot) MustScrollToBottom(opts ...ElemOptionFunc)

func (*Bot) MustScrollToTop 

func (b *Bot) MustScrollToTop()

func (*Bot) MustScrollToXY 

func (b *Bot) MustScrollToXY(x, y float64)

func (*Bot) MustWaitLoad 

func (b *Bot) MustWaitLoad()

func (*Bot) Open 

func (b *Bot) Open(url string, timeouts ...time.Duration) error

func (*Bot) OpenAndSetCookies added in v0.1.4 

func (b *Bot) OpenAndSetCookies(uri string, timeouts ...time.Duration) error

OpenAndSetCookies open uri with cookies.

typically with following steps:

func (*Bot) OpenElementInNewTab added in v0.1.5 

func (b *Bot) OpenElementInNewTab(elem *rod.Element) error

OpenElementInNewTab opens an element (usually a link or button) in a new browser tab.

This function simulates a Ctrl+Click (or Cmd+Click on macOS) on the given element, which typically opens the link in a new tab in most web browsers. After opening the new tab, it switches the bot's focus to the newly opened tab.

Parameters:

  • elem: A *rod.Element pointer representing the element to be opened in a new tab. This is typically a link (<a> tag) or a button that triggers navigation.

Returns:

  • error: An error if the operation fails, or nil if successful.

Behavior:

  1. Retrieves the current list of open pages in the browser.
  2. Determines the appropriate key for Ctrl/Cmd based on the operating system.
  3. Focuses on and highlights the target element.
  4. Simulates a Ctrl/Cmd + Enter key press on the element.
  5. Waits for and activates the newly opened page.

Usage example: 

linkElem, _ := b.Elem("a.external-link")
err := b.OpenElementInNewTab(linkElem)
if err != nil {
    log.Println("Failed to open link in new tab:", err)
    return
}
// Bot is now focused on the newly opened tab

Error handling:

  • Returns an error if the Ctrl/Cmd + Enter key action fails.
  • Returns an error if unable to switch to the newly opened page.

Note:

  • This function assumes that Ctrl/Cmd + Enter will open the link in a new tab, which is standard behavior for most websites but may not work in all cases.
  • The function waits for up to 10 seconds for the new tab to open before timing out.
  • Ensure that pop-ups are not blocked in the browser settings, as this might prevent new tabs from opening.
  • This function automatically switches the bot's context to the new tab.

OS Compatibility:

  • Automatically uses Cmd key on macOS and Ctrl key on other operating systems.

See also:

  • FocusAndHighlight: Used internally to focus on the element before interaction.
  • ActivateLastOpenedPage: Used to switch to the newly opened page.

func (*Bot) OpenOrClickElement added in v0.1.5 

func (b *Bot) OpenOrClickElement(elem *rod.Element, opts ...ElemOptionFunc) error

OpenOrClickElement attempts to open or interact with a given element, with options for opening in a new tab.

This function provides a flexible way to interact with elements, particularly links or buttons that lead to new content. It can either click the element directly or open it in a new tab, depending on the options provided.

Parameters:

  • elem: A *rod.Element pointer representing the element to open or interact with.
  • opts: Variadic ElemOptionFunc arguments to customize the opening behavior. Notable options include:
  • WithOpenInTab(bool): If true, opens the element in a new tab instead of clicking it directly.

Returns:

  • error: An error if the operation fails, or nil if successful.

Behavior:

  1. Applies the provided options to determine the interaction method.
  2. Attempts to focus on the element using TryFocusElem.
  3. If focusing fails, returns ErrNotInteractable.
  4. If WithOpenInTab is true, calls OpenInNewTab.
  5. Otherwise, calls ClickElem for a standard click interaction.

Usage example: 

elem, _ := b.Elem("a.external-link")
err := b.OpenOrClickElement(elem, WithOpenInTab(true))
if err != nil {
    if errors.Is(err, ErrNotInteractable) {
        log.Println("Element is not interactable")
    } else {
        log.Println("Failed to open element:", err)
    }
    return
}
// Element has been opened in a new tab

Error handling:

  • Returns ErrNotInteractable if the element cannot be focused or made interactable.
  • Propagates errors from OpenInNewTab or ClickElem operations.

Options:

  • WithOpenInTab(bool): Determines whether to open the element in a new tab.
  • Other standard ElemOptions can be used to customize the element interaction.

Note:

  • This function is particularly useful for handling navigation elements or interactive elements that may open new content.
  • The behavior of opening in a new tab may depend on the browser's settings and the website's implementation.
  • Ensure that the browser instance supports multiple tabs when using WithOpenInTab.

See also:

  • TryFocusElem: Used internally to ensure the element is interactable.
  • OpenInNewTab: Called when opening the element in a new tab.
  • ClickElem: Used for standard click interactions.

func (*Bot) OpenURLInNewTab added in v0.1.4 

func (b *Bot) OpenURLInNewTab(uri string) error

OpenURLInNewTab opens url in a new tab and activates it

func (*Bot) Page 

func (b *Bot) Page() *rod.Page

func (*Bot) ParseCURLCookiesFromFile added in v0.1.4 

func (b *Bot) ParseCURLCookiesFromFile(filePath string) ([]*proto.NetworkCookieParam, error)

ParseCURLCookiesFromFile reads a file containing cURL-formatted cookies, parses the content, and returns NetworkCookieParam objects. It returns an error if the file cannot be read or parsed.

func (*Bot) Press 

func (b *Bot) Press(elem *rod.Element, keys ...input.Key) error

func (*Bot) PressEscape 

func (b *Bot) PressEscape(elem *rod.Element) error

func (*Bot) PressPageDown added in v0.1.4 

func (b *Bot) PressPageDown(times int) error

func (*Bot) ResetToOriginalPage 

func (b *Bot) ResetToOriginalPage() error

func (*Bot) Scroll 

func (b *Bot) Scroll(x, y float64, step int) error

Scroll Scroll with mouse.

func (*Bot) ScrollLikeHuman 

func (b *Bot) ScrollLikeHuman(offsetX, offsetY float64, opts ...ElemOptionFunc) error

func (*Bot) ScrollToBottom added in v0.1.4 

func (b *Bot) ScrollToBottom(opts ...ElemOptionFunc) error

func (*Bot) ScrollToElemDirectly 

func (b *Bot) ScrollToElemDirectly(elem *rod.Element) error

func (*Bot) ScrollToElement 

func (b *Bot) ScrollToElement(elem *rod.Element, opts ...ElemOptionFunc)

ScrollToElement just a do the best operation

func (*Bot) ScrollToElementE added in v0.1.4 

func (b *Bot) ScrollToElementE(elem *rod.Element, opts ...ElemOptionFunc) error

func (*Bot) SetPanicWith 

func (b *Bot) SetPanicWith(panicWith PanicByType)

func (*Bot) SetTimeout 

func (b *Bot) SetTimeout()

func (*Bot) TimeTrack added in v0.1.4 

func (b *Bot) TimeTrack(start time.Time, skip int)

func (*Bot) TryFocusElement added in v0.1.5 

func (b *Bot) TryFocusElement(elem *rod.Element, scrollToRight bool) error

TryFocusElement attempts to focus on a given element, with optional scrolling to the right.

This function tries to make an element interactable by focusing on it and, if necessary, scrolling the page. It's particularly useful for elements that might be out of view or require scrolling to become accessible.

Behavior:

  1. Attempts to check if the element is interactable.
  2. If not interactable, it tries to scroll to the element.
  3. If scrollToRight is true, it also scrolls horizontally.
  4. This process is retried up to 3 times (using RetryIn3).

The function performs the following steps in each attempt:

  1. Checks if the element is interactable using elem.Interactable().
  2. If not interactable: - Scrolls to the element using b.ScrollToElemDirectly(elem). - If scrollToRight is true, scrolls 1024 pixels to the right.
  3. If the element becomes interactable, the function returns nil.
  4. If it remains non-interactable after 3 attempts, it returns an error.

Usage example: 

elem, _ := b.Elem("#my-button")
err := b.TryFocusElement(elem, false)
if err != nil {
    log.Println("Failed to focus on element:", err)
    return
}
// Element is now focused and interactable
elem.Click()

Error handling:

  • Returns ErrNotInteractable if the element cannot be made interactable after 3 attempts.
  • Any errors from scrolling operations are ignored within the retry loop.

Note:

  • This function is useful for handling elements that may be initially out of view or require page manipulation to become accessible.
  • The scrolling behavior uses direct page manipulation, which might not trigger certain JavaScript events associated with normal scrolling.
  • The horizontal scroll amount (1024 pixels) is hardcoded and may need adjustment for specific use cases.

See also:

  • ScrollToElemDirectly: The method used for scrolling to the element.
  • RetryIn3: The retry mechanism used in this function.

func (*Bot) TryScrollToBottom added in v0.1.4 

func (b *Bot) TryScrollToBottom(opts ...ElemOptionFunc)

TryScrollToBottom just try scroll to bottom, error will be ignored.

func (*Bot) TypeCharsOneByOne 

func (b *Bot) TypeCharsOneByOne(elem *rod.Element, value string)

func (*Bot) WaitURLContains added in v0.1.4 

func (b *Bot) WaitURLContains(str string, timeouts ...float64) error

WaitURLContains uses `decodeURIComponent(window.location.href).includes` to check if url has str or not, Scenario:

  • if a page's loading status can be determined by url

type BotOption 

type BotOption func(*Bot)

func AcceptLanguage added in v0.1.4 

func AcceptLanguage(s string) BotOption

func Browser added in v0.1.4 

func Browser(brw *rod.Browser) BotOption

func ClearCookies added in v0.1.4 

func ClearCookies(b bool) BotOption

func CopyAsCURLCookies added in v0.1.4 

func CopyAsCURLCookies(b []byte) BotOption

CopyAsCURLCookies reads the raw content `Copy as cURL` from clipboard

func ForceCleanup added in v0.1.4 

func ForceCleanup(b bool) BotOption

func Headless 

func Headless(b bool) BotOption

func Humanized added in v0.1.4 

func Humanized(b bool) BotOption

func Launcher added in v0.1.4 

func Launcher(l *launcher.Launcher) BotOption

func Logger added in v0.1.4 

func Logger(l *zap.Logger) BotOption

func StealthMode added in v0.1.4 

func StealthMode(b bool) BotOption

func TrackTime added in v0.1.4 

func TrackTime(b bool) BotOption

TrackTime tracktime shows logs with level `debug`.

func UserAgent added in v0.1.4 

func UserAgent(s string) BotOption

func UserDataDir added in v0.1.4 

func UserDataDir(s string) BotOption

func UserMode added in v0.1.4 

func UserMode(b bool) BotOption

func WithCookieFile 

func WithCookieFile(s string) BotOption

WithCookieFile uses the cookie file specified, cookie file should be json array.

func WithCookieFolder added in v0.1.2 

func WithCookieFolder(s string) BotOption

WithCookieFolder will load cookies from folder passed in.

func WithCookies 

func WithCookies(b bool) BotOption

WithCookies will automatically load cookies from current dir for ".cookies/xxx".

func WithHighlightTimes 

func WithHighlightTimes(i int) BotOption

func WithLeftPosition added in v0.1.4 

func WithLeftPosition(i int) BotOption

WithLeftPosition sets the left position of the browser window.

func WithPage 

func WithPage(b bool) BotOption

func WithPanicBy 

func WithPanicBy(i PanicByType) BotOption

func WithPopovers 

func WithPopovers(popovers ...string) BotOption

WithPopovers is useful when popovers appear randomly.

type BrowserOptionFunc 

type BrowserOptionFunc func(o *BrowserOptions)

func BrowserExtensions added in v0.1.4 

func BrowserExtensions(extArr ...string) BrowserOptionFunc

BrowserExtensions dirs of extensions, only unpacked extension is supported.

func BrowserFlags added in v0.1.4 

func BrowserFlags(arr ...string) BrowserOptionFunc

BrowserFlags flags those are not pre-defined

func BrowserHeadless added in v0.1.4 

func BrowserHeadless(b bool) BrowserOptionFunc

func BrowserIgnoreCertErrors added in v0.1.4 

func BrowserIgnoreCertErrors(b bool) BrowserOptionFunc

func BrowserIncognito added in v0.1.4 

func BrowserIncognito(b bool) BrowserOptionFunc

func BrowserNoDefaultDevice added in v0.1.4 

func BrowserNoDefaultDevice(b bool) BrowserOptionFunc

func BrowserPaintRects added in v0.1.4 

func BrowserPaintRects(b bool) BrowserOptionFunc

func BrowserProxy added in v0.1.4 

func BrowserProxy(s string) BrowserOptionFunc

func BrowserSlowMotionDelay added in v0.1.4 

func BrowserSlowMotionDelay(i int) BrowserOptionFunc

func BrowserUserDataDir added in v0.1.4 

func BrowserUserDataDir(s string) BrowserOptionFunc

func LaunchLeakless added in v0.1.4 

func LaunchLeakless(b bool) BrowserOptionFunc

type BrowserOptions 

type BrowserOptions struct {
	// contains filtered or unexported fields
}

type ElemOptionFunc 

type ElemOptionFunc func(o *ElemOptions)

func ClearBeforeInput added in v0.1.3 

func ClearBeforeInput(b bool) ElemOptionFunc

func DisableHandleCoverByEsc 

func DisableHandleCoverByEsc() ElemOptionFunc

func EndWithEscape added in v0.1.3 

func EndWithEscape(b bool) ElemOptionFunc

func OpenInTab added in v0.1.4 

func OpenInTab(b bool) ElemOptionFunc

func Trigger added in v0.1.4 

func Trigger(b bool) ElemOptionFunc

func WithAttr 

func WithAttr(s string) ElemOptionFunc

func WithAttrMap 

func WithAttrMap(m map[string]string) ElemOptionFunc

func WithCaseInsensitive 

func WithCaseInsensitive(b bool) ElemOptionFunc

func WithClickByScript 

func WithClickByScript(b bool) ElemOptionFunc

func WithHighlight 

func WithHighlight(b bool) ElemOptionFunc

func WithHumanized 

func WithHumanized(b bool) ElemOptionFunc

func WithIframe 

func WithIframe(page *rod.Page) ElemOptionFunc

func WithIndex 

func WithIndex(i int) ElemOptionFunc

func WithRetries added in v0.1.4 

func WithRetries(i uint) ElemOptionFunc

func WithRoot 

func WithRoot(root *rod.Element) ElemOptionFunc

func WithScrollAsHuman 

func WithScrollAsHuman(ah *ScrollAsHuman) ElemOptionFunc

func WithSelectorType 

func WithSelectorType(t rod.SelectorType) ElemOptionFunc

func WithSteps 

func WithSteps(i int) ElemOptionFunc

func WithSubmit 

func WithSubmit(b bool) ElemOptionFunc

func WithTimeout 

func WithTimeout(t float64) ElemOptionFunc

func WithWaitStable 

func WithWaitStable(b bool) ElemOptionFunc

type ElemOptions 

type ElemOptions struct {
	// contains filtered or unexported fields
}

type HijackHandler 

type HijackHandler func(*rod.Hijack)

HijackHandler is a function type used for handling hijacked network resources in web automation tasks.

Type Definition: 

type HijackHandler func(*rod.Hijack)

Purpose: 

This type defines a callback function that is invoked when a network request
matching specified criteria is intercepted during web automation or scraping tasks.

Parameters:

  • *rod.Hijack: A pointer to the rod.Hijack object, which encapsulates the intercepted request and provides methods to inspect and manipulate both the request and its response.

Functionality:

  1. Request Inspection: Examine details of the intercepted request such as URL, headers, method, and body.
  2. Response Manipulation: Modify the response before it reaches the browser, including changing status codes, headers, or body content.
  3. Request Blocking: Prevent certain requests from being sent.
  4. Custom Logic: Implement any custom behavior based on the intercepted request.

Usage Context:

  • Used as a parameter in methods like HijackAny and Hijack of the Bot struct.
  • Allows for fine-grained control over network traffic during web automation.

Common Use Cases:

  1. Modifying API responses for testing purposes.
  2. Blocking unwanted resources (e.g., advertisements).
  3. Logging or analyzing network traffic.
  4. Injecting custom JavaScript or CSS into responses.

Example Implementation: 

handler := func(h *rod.Hijack) {
    // Log the URL of each request
    fmt.Println("Intercepted request to:", h.Request.URL().String())

    // Modify a specific API response
    if strings.Contains(h.Request.URL().String(), "api/data") {
        h.Response.SetBody([]byte(`{"modified":"data"}`))
    }

    // Block image requests
    if h.Request.Type() == proto.NetworkResourceTypeImage {
        h.Response.Fail(proto.NetworkErrorReasonBlockedByClient)
    }
}

Note:

  • The exact behavior after the handler executes depends on the 'continueRequest' parameter in the hijacking method and any actions taken within the handler.
  • Care should be taken when modifying responses, as it may affect the functionality of the web page being automated.

Related Methods:

  • Bot.HijackAny: Hijacks requests matching any of the specified resource strings.
  • Bot.Hijack: Hijacks requests matching specified patterns and resource types.

See Also:

  • rod.Hijack documentation for more details on available methods and properties.

type PanicByType 

type PanicByType int
const (
	PanicByDft PanicByType = iota
	PanicByDump
	PanicByLogError
)

type ScrollAsHuman 

type ScrollAsHuman struct {
	// contains filtered or unexported fields
}

func NewScrollAsHuman added in v0.1.4 

func NewScrollAsHuman(long, short, up float64) *ScrollAsHuman

Source Files

View all Source files

Directories

Path Synopsis
examples
basic/baidu
basic/google
basic/sannysoft
blocket
books
express
hijack
incolumitas
jandan
modal
nfcloud
qiyueba
usermode
wikipedia

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.