chromedpundetected

package module
v1.3.8-lambda.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 14, 2024 License: MIT Imports: 28 Imported by: 0

README

Undetected chromedp

Go.Dev reference Go Report Card Unit Tests GitHub

A small library that provides a chromedp context with a browser configured to mimick a regular browser to prevent triggering anti-bot measures. This is not a fool proof method, and how you use it will still dictate whether you will run into anti-bot detection, but at least it won't trigger on all the basic detection tests.

The headless option only works on linux, and requires Xvfb to be installed. Could theoretically work on Mac OS with xquartz but I don't have a Mac to test with, so feel free to PR.

A Docker container example is provided in Dockerfile. The most important things to note is to not use the headless chrome image as base, but to normally install chrome or chromium, and to install xvfb. Note that this image is neither secure nor optimized, and merely serves as an example.

package main

import (
	"time"

	"github.com/chromedp/chromedp"

	cu "github.com/Davincible/chromedp-undetected"
)

func main() {
	// New creates a new context for use with chromedp. With this context
	// you can use chromedp as you normally would.
	ctx, cancel, err := cu.New(cu.NewConfig(
		// Remove this if you want to see a browser window.
		cu.WithHeadless(),

		// If the webelement is not found within 10 seconds, timeout.
		cu.WithTimeout(10 * time.Second),
	))
	if err != nil {
		panic(err)
	}
	defer cancel()

	if err := chromedp.Run(ctx,
		// Check if we pass anti-bot measures.
		chromedp.Navigate("https://nowsecure.nl"),
		chromedp.WaitVisible(`//div[@class="hystericalbg"]`),
	); err != nil {
		panic(err)
	}

	fmt.Println("Undetected!")
}

Based on undetected-chromedriver

Utilities

Some utility functions are included I was missing in chromedp itself.

// BlockURLs blocks a set of URLs in Chrome.
func BlockURLs(url ...string) chromedp.ActionFunc 

// LoadCookies will load a set of cookies into the browser.
func LoadCookies(cookies []Cookie) chromedp.ActionFunc

// LoadCookiesFromFile takes a file path to a json file containing cookies,
// and loads in the cookies into the browser.
func LoadCookiesFromFile(path string) chromedp.ActionFunc

// SaveCookies extracts the cookies from the current URL and appends them to
// provided array.
func SaveCookies(cookies *[]Cookie) chromedp.ActionFunc

// SaveCookiesTo extracts the cookies from the current page and saves them
// as JSON to the provided path.
func SaveCookiesTo(path string) chromedp.ActionFunc

// RunCommand runs any Chrome Dev Tools command, with any params.
// 
// In contrast to the native method of chromedp, with this method you can
// directly pass in a map with the data passed to the command.
func RunCommand(method string, params any) chromedp.ActionFunc

// RunCommandWithRes runs any Chrome Dev Tools command, with any params and
// sets the result to the res parameter. Make sure it is a pointer.
// 
// In contrast to the native method of chromedp, with this method you can
// directly pass in a map with the data passed to the command.
func RunCommandWithRes(method string, params, res any) chromedp.ActionFunc

// UserAgentOverride overwrites the Chrome user agent.
// 
// It's better to use this method than emulation.UserAgentOverride.
func UserAgentOverride(userAgent string) chromedp.ActionFunc

// SendKeys does the same as chromedp.SendKeys excepts it randomly waits 100-500ms
// between sending key presses.
func SendKeys(sel any, v string, opts ...chromedp.QueryOption) chromedp.ActionFunc

Documentation

Overview

Package chromedpundetected provides a chromedp context with an undetected Chrome browser.

Package chromedpundetected provides a chromedp context with an undetected Chrome browser.

Index

Constants

View Source 
const (
	// DefaultNoSandbox enables the 'no-sandbox' flag by default.
	DefaultNoSandbox = true
)

Variables

View Source 
var (
	DefaultUserDirPrefix = "chromedp-undetected-"
)

Defaults.

View Source 
var (
	ErrXvfbNotFound = errors.New("xvfb not found. Please install (Linux only)")
)

Errors.

Functions

func BlockURLs 

func BlockURLs(url ...string) chromedp.ActionFunc

BlockURLs blocks a set of URLs in Chrome.

func LoadCookies 

func LoadCookies(cookies []Cookie) chromedp.ActionFunc

LoadCookies will load a set of cookies into the browser.

func LoadCookiesFromFile 

func LoadCookiesFromFile(path string) chromedp.ActionFunc

LoadCookiesFromFile takes a file path to a json file containing cookies, and loads in the cookies into the browser.

func MoveMouseToPosition 

func MoveMouseToPosition(x, y float64, setters ...MoveOptionSetter) chromedp.ActionFunc

MoveMouseToPosition moves the mouse to the given position, mimic random human mouse movements.

If desired you can tweak the mouse movement behavior, defaults are set to mimic human mouse movements.

func NetworkIdleListener 

func NetworkIdleListener(ctx context.Context, networkIdleTimeout, totalTimeout time.Duration) chan IdleEvent

NetworkIdleListener sets up a listener to monitor for network idle events.

This can be used as a proxy to know for e.g. when a page has fully loaded, assuming that the page doesn't send any network requests within the networkIdleTimeout period after is has finished loading.

This function creates a listener that monitors the target specified by the given context for network activity. It triggers a "NETWORK IDLE" event if no network request is sent within the provided `networkIdleTimeout` duration after a network idle lifecycle event.

After the network is considered idle, the listener will terminate and the channel will close.

Parameters:

  • ctx: The context for which the listener is set up. Usually, this context is tied to a specific browser tab or page.
  • networkIdleTimeout: The duration to wait for no network activity after a network idle lifecycle event before considering the network to be truly idle.
  • totalTimeout: The duration to wait for the network to be idle before terminating the listener.

Returns: 1. A channel of type IdleEvent. When the network is considered idle, an IdleEvent with IsIdle set to true is sent through this channel.

func NetworkIdlePermanentListener 

func NetworkIdlePermanentListener(ctx context.Context, networkIdleTimeout time.Duration) (chan IdleEvent, func())

This can be used as a proxy to know for e.g. when a page has fully loaded, assuming that the page doesn't send any network requests within the networkIdleTimeout period after is has finished loading.

This function creates a listener that monitors the target specified by the given context for network activity. It triggers a "NETWORK IDLE" event if no network request is sent within the provided `networkIdleTimeout` duration after a network idle lifecycle event.

It's designed to run indefinitely, i.e., it doesn't automatically stop listening after an idle event or after a certain period. To manually stop listening and to free up associated resources, one should call the returned cancel function.

Parameters:

  • ctx: The context for which the listener is set up. Usually, this context is tied to a specific browser tab or page.
  • networkIdleTimeout: The duration to wait for no network activity after a network idle lifecycle event before considering the network to be truly idle.

Returns:

  1. A channel of type IdleEvent. When the network is considered idle, an IdleEvent with IsIdle set to true is sent through this channel.
  2. A cancel function. This function can be called to terminate the listener and close the channel.

func New 

func New(config Config) (context.Context, context.CancelFunc, error)

New creates a context with an undetected Chrome executor.

func RunCommand 

func RunCommand(method string, params any) chromedp.ActionFunc

RunCommand runs any Chrome Dev Tools command, with any params.

In contrast to the native method of chromedp, with this method you can directly pass in a map with the data passed to the command.

func RunCommandWithRes 

func RunCommandWithRes(method string, params, res any) chromedp.ActionFunc

RunCommandWithRes runs any Chrome Dev Tools command, with any params and sets the result to the res parameter. Make sure it is a pointer.

In contrast to the native method of chromedp, with this method you can directly pass in a map with the data passed to the command.

func SaveCookies 

func SaveCookies(cookies *[]Cookie) chromedp.ActionFunc

SaveCookies extracts the cookies from the current URL and appends them to provided array.

func SaveCookiesTo 

func SaveCookiesTo(path string) chromedp.ActionFunc

SaveCookiesTo extracts the cookies from the current page and saves them as JSON to the provided path.

func SendKeys 

func SendKeys(sel any, v string, opts ...chromedp.QueryOption) chromedp.ActionFunc

SendKeys does the same as chromedp.SendKeys excepts it randomly waits 100-500ms between sending key presses.

func UserAgentOverride 

func UserAgentOverride(userAgent string) chromedp.ActionFunc

UserAgentOverride overwrites the Chrome user agent.

It's better to use this method than emulation.UserAgentOverride.

Types

type Config 

type Config struct {
	// Ctx is the base context to use. By default a background context will be used.
	Ctx context.Context `json:"-" yaml:"-"`

	// ContextOptions are chromedp context option.
	ContextOptions []chromedp.ContextOption `json:"-" yaml:"-"`

	// ChromeFlags are additional Chrome flags to pass to the browser.
	//
	// NOTE: adding additional flags can make the detection unstable, so test,
	// and be careful of what flags you add. Mostly intended to configure things
	// like a proxy. Also check if the flags you want to set are not already set
	// by this library.
	ChromeFlags []chromedp.ExecAllocatorOption

	// UserDataDir is the path to the directory where Chrome user data is stored.
	//
	// By default a temporary directory will be used.
	UserDataDir string `json:"userDataDir" yaml:"userDataDir"`

	// LogLevel is the Chrome log level, 0 by default.
	LogLevel int `json:"logLevel" yaml:"logLevel"`

	// NoSandbox dictates whether the no-sanbox flag is added. Defaults to true.
	NoSandbox bool `json:"noSandbox" yaml:"noSandbox"`

	// ChromePath is a specific binary path for Chrome.
	//
	// By default the chrome or chromium on your PATH will be used.
	ChromePath string `json:"chromePath" yaml:"chromePath"`

	// Port is the Chrome debugger port. By default a random port will be used.
	Port int `json:"port" yaml:"port"`

	// Timeout is the context timeout.
	Timeout time.Duration `json:"timeout" yaml:"timeout"`

	// Headless dicates whether Chrome will start headless (without a visible window)
	//
	// It will NOT use the '--headless' option, rather it will use a virtual display.
	// Requires Xvfb to be installed, only available on Linux.
	Headless bool `json:"headless" yaml:"headless"`

	// Extensions are the paths to the extensions to load.
	Extensions []string `json:"extensions" yaml:"extensions"`

	// language to be used otherwise system/OS defaults are used
	// https://developer.chrome.com/docs/webstore/i18n/#localeTable
	Language string
}

Config is a undetected Chrome config.

func NewConfig 

func NewConfig(opts ...Option) Config

NewConfig creates a new config object with defaults. 

type Cookie struct {
	Name     string  `json:"name" yaml:"name"`
	Value    string  `json:"value" yaml:"value"`
	Domain   string  `json:"domain" yaml:"domain"`
	Path     string  `json:"path" yaml:"path"`
	Expires  float64 `json:"expires" yaml:"expires"`
	HTTPOnly bool    `json:"httpOnly" yaml:"httpOnly"`
	Secure   bool    `json:"secure" yaml:"secure"`
}

Cookie is used to set browser cookies.

type IdleEvent 

type IdleEvent struct {
	IsIdle bool
}

IdleEvent is sent through the channel returned by NetworkIdleListener when the network is considered idle. This event can be used to determine when a page has finished loading.

type MouseMoveOptions 

type MouseMoveOptions struct {
	// contains filtered or unexported fields
}

MouseMoveOptions contains options for mouse movement.

type MoveOptionSetter 

type MoveOptionSetter func(*MouseMoveOptions)

MoveOptionSetter defines a function type to set mouse move options.

func WithDelayRange 

func WithDelayRange(min, max time.Duration) MoveOptionSetter

WithDelayRange returns a MoveOptionSetter that sets the delay range between steps.

func WithRandomJitter 

func WithRandomJitter(jitter float64) MoveOptionSetter

WithRandomJitter returns a MoveOptionSetter that sets the random jitter to introduce in mouse movement.

func WithSteps 

func WithSteps(s int) MoveOptionSetter

WithSteps returns a MoveOptionSetter that sets the number of steps for the mouse movement.

func WithVisualizeMouse 

func WithVisualizeMouse() MoveOptionSetter

WithVisualizeMouse returns a MoveOptionSetter that enables mouse movement visualization.

type Option 

type Option func(*Config)

Option is a functional option.

func WithChromeBinary 

func WithChromeBinary(path string) Option

WithChromeBinary sets the chrome binary path.

func WithChromeFlags 

func WithChromeFlags(opts ...chromedp.ExecAllocatorOption) Option

WithChromeFlags add chrome flags.

func WithContext 

func WithContext(ctx context.Context) Option

WithContext adds a base context.

func WithExtensions 

func WithExtensions(extensions ...string) Option

WithExtensions adds chrome extensions.

Provide the paths to the extensions to load.

func WithHeadless 

func WithHeadless() Option

WithHeadless creates a headless chrome instance.

func WithLogLevel 

func WithLogLevel(level int) Option

WithLogLevel sets the chrome log level.

func WithNoSandbox 

func WithNoSandbox(b bool) Option

WithNoSandbox enable/disable sandbox. Disabled by default.

func WithPort 

func WithPort(port int) Option

WithPort sets the chrome debugger port.

func WithTimeout 

func WithTimeout(timeout time.Duration) Option

WithTimeout sets the context timeout.

func WithUserDataDir 

func WithUserDataDir(dir string) Option

WithUserDataDir sets the user data directory to a custom path.

Source Files

View all Source files

Directories

Path Synopsis
util
easyjson
Package easyjson provides utilities for use with easyjson.
 Package easyjson provides utilities for use with easyjson.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.