pagescanner

Documentation

Index

• type HTTP
  • func NewHTTP(config spiderconfiginterface.Configurator, ...) *HTTP
  • func (s *HTTP) Scan(ctx context.Context, url *url.URL, userAgent string, ...)

Types

type HTTP

type HTTP struct {
	// contains filtered or unexported fields
}

func NewHTTP

func NewHTTP(
	config spiderconfiginterface.Configurator,
	parser pageparserinterface.PageParser,
	logger logger.Logger,
) *HTTP

NewHTTP is a constructor of HTTP scanner.

func (*HTTP) Scan

func (s *HTTP) Scan(
	ctx context.Context,
	url *url.URL,
	userAgent string,
	resultCh chan<- scannerdtointerface.Result,
	retries int,
)

Scan is method which request target page and parse it into *entity.Page struct (recursive: depends on retries arg.).