bartender

README

Overview

It's design to make SEO for single page application easier, so that you don't have to use Server-side rendering tricks.

It acts like a transparent http proxy by default, it only actives when the client looks like a web crawler, such as Googlebot, Baiduspider, etc.

Installation

You can simplify use bartender as the gateway in front of your web server:

docker run -p 3000:3000 ghcr.io/go-rod/bartender ./serve -p :3000 -t http://your-web-server:8080

A common data flow looks like this:

graph TD;
  C[Client]-->B;

  subgraph B[Bartender]
    J{Is web crawler?};
    J-->|Yes|R[Render with headless browser];
    J-->|No|D[Transparent proxy];
  end

  R-->H[Your web server];
  D-->H;

If you want the best performance, you can install bartender behind a gateway like nginx, configure the gateway to proxy the request to bartender when the client looks like a web crawler.

A common way to detect web crawler: link.

A common data flow looks like this:

graph TD;
  C[Client]-->T[Gateway];
  T-->J{Is web crawler?};
  J-->|Yes|B[Bartender];
  J-->|No|H[Your web server];
  B-->H;

Documentation

Overview

Package bartender is a service to make web crawlers consume webpages easier

Index

• Variables
• type Bartender
  • func New(addr, target string, poolSize int) *Bartender
  • func (b *Bartender) AutoFree(interval time.Duration)
  • func (b *Bartender) BlockRequests(patterns ...string)
  • func (b *Bartender) BypassUserAgentNames(list ...string)
  • func (b *Bartender) MaxWait(d time.Duration)
  • func (b *Bartender) RenderPage(w http.ResponseWriter, r *http.Request) bool
  • func (b *Bartender) ServeHTTP(w http.ResponseWriter, r *http.Request)
  • func (b *Bartender) WarmUp()

Variables

var DefaultBypassUserAgentNames = []string{
	useragent.Opera,
	useragent.OperaMini,
	useragent.OperaTouch,
	useragent.Chrome,
	useragent.HeadlessChrome,
	useragent.Firefox,
	useragent.InternetExplorer,
	useragent.Safari,
	useragent.Edge,
	useragent.Vivaldi,
}

Types

type Bartender

type Bartender struct {
	// contains filtered or unexported fields
}

func New

func New(addr, target string, poolSize int) *Bartender

func (*Bartender) AutoFree

func (b *Bartender) AutoFree(interval time.Duration)

AutoFree automatically closes the each headless browser after a period of time. It prevent the memory leak of the headless browser.

func (*Bartender) BlockRequests

func (b *Bartender) BlockRequests(patterns ...string)

func (*Bartender) BypassUserAgentNames

func (b *Bartender) BypassUserAgentNames(list ...string)

func (*Bartender) MaxWait

func (b *Bartender) MaxWait(d time.Duration)

MaxWait sets the max wait time for the headless browser to render the page. If the max wait time is reached, bartender will stop waiting for page rendering and immediately return the current html.

func (*Bartender) RenderPage

func (b *Bartender) RenderPage(w http.ResponseWriter, r *http.Request) bool

RenderPage returns true if the page is rendered by the headless browser.

func (*Bartender) ServeHTTP

func (b *Bartender) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*Bartender) WarmUp

func (b *Bartender) WarmUp()

WarmUp pre-creates the headless browsers.