goxuiter

package module
v0.0.0-...-d3ed488 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 13, 2021 License: Apache-2.0 Imports: 11 Imported by: 0

README

go-xuiter

A simple photo download crawler for blog xuite.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FolderExit

func FolderExit(prefix, folder string) bool

func SaveFile

func SaveFile(client *http.Client, prefix, folder, filename string, url string) error

Types

type CollyController

type CollyController struct {
	*colly.Collector
	// contains filtered or unexported fields
}

func NewCollyController

func NewCollyController(qc SizeController, concurrency int) *CollyController

func (*CollyController) AddRequest

func (c *CollyController) AddRequest(r *colly.Request) error

func (*CollyController) AddURL

func (c *CollyController) AddURL(url string) error

func (*CollyController) OnQueueSizeMonitor

func (c *CollyController) OnQueueSizeMonitor(f QueueSizeCallback)

func (*CollyController) Run

func (c *CollyController) Run()

type NoOpController

type NoOpController struct{}

func (*NoOpController) CheckAndWait

func (n *NoOpController) CheckAndWait()

func (*NoOpController) Pause

func (n *NoOpController) Pause()

func (*NoOpController) Resume

func (n *NoOpController) Resume()

type QueueSizeCallback

type QueueSizeCallback func(size int)

type QueueSizeController

type QueueSizeController struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func NewQueueSizeController

func NewQueueSizeController() *QueueSizeController

func (*QueueSizeController) CheckAndWait

func (q *QueueSizeController) CheckAndWait()

func (*QueueSizeController) Pause

func (q *QueueSizeController) Pause()

func (*QueueSizeController) Resume

func (q *QueueSizeController) Resume()

type SizeController

type SizeController interface {
	Pause()
	Resume()
	CheckAndWait()
}

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL