Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security Policy
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

article_spider

package module

Go to main page

Versions in this module

v3

v3.0.6

Apr 12, 2022

Changes in this version

type Field

+ NotSelector []string

v3.0.5

Apr 7, 2022

Changes in this version

type Field

+ LazyImageAttrName string

v3.0.4

Jan 20, 2022

v3.0.3

Jan 20, 2022

v3.0.2

Jan 19, 2022

v3.0.1

Jan 19, 2022

v3.0.0

Jan 17, 2022

Changes in this version

+ const Debug

+ const Error

+ const Finish

+ const Info

+ const Log

+ const Process

+ func If(condition bool, trueVal, falseVal interface{}) interface

+ func NewApi(s *Spider) *api

+ func NewAuto(s *Spider) *auto

+ func NewExcel(s *Spider) *excel

+ func NewNormal(s *Spider) *normal

+ func NewResult(s *Spider) *result

+ type Field struct

+ AttrKey string

+ ConversionFunc func(data string, resList map[string]string) string

+ ExcelHeader string

+ ImageDir string

+ ImagePrefix func(form *Form, imageName string) string

+ RegularIndex int

+ Selector string

+ Types FieldTypes

+ type FieldTypes int

+ const Attr

+ const Fixed

+ const HtmlWithImage

+ const Image

+ const MultipleImages

+ const OnlyHtml

+ const Regular

+ const Text

+ type Form struct

+ AllowImageExtension []string

+ ApiConversion func(html string, form *Form) []string

+ AutoDetailForceNewTab bool

+ AutoDetailWaitSelector string

+ AutoListWaitSelector string

+ AutoNextPageMode NextPageMode

+ AutoNextSelector string

+ AutoPrefixEvent func(chromedpCtx context.Context)

+ Channel string

+ ChannelFunc func(form *Form) []string

+ CustomExcelHeader bool

+ DefaultImg func(form *Form, item Field) string

+ DetailCoroutineNumber int

+ DetailFields map[string]Field

+ DisableAutoCoding bool

+ DisableImageExtensionCheck bool

+ Host string

+ HrefSelector string

+ HttpHeader map[string]string

+ HttpProxy string

+ HttpTimeout time.Duration

+ LazyImageAttrName string

+ Length int

+ ListFields map[string]Field

+ ListSelector string

+ MiddleSelector []string

+ PageStart int

+ ResultCallback func(item map[string]string, form *Form)

+ func (f *Form) DealCoding(html string, header http2.Header) (string, error)

+ func (f *Form) DownImg(url string, item Field, res *sync.Map) string

+ func (f *Form) GetCharsetByContentType(contentType string) string

+ func (f *Form) GetDir(path string, res *sync.Map) string

+ func (f *Form) GetHref(href string) string

+ func (f *Form) GetHtml(url string) (string, error)

+ func (f *Form) ResolveSelector(html string, selector map[string]Field, originUrl string) (map[string]string, error)

+ type Mode int

+ const Api

+ const Auto

+ const Normal

+ type NextPageMode int

+ const LoadMore

+ const Pagination

+ type Notice struct

+ func NewNotice(s *Spider) *Notice

+ func (n *Notice) Debug(content ...interface{})

+ func (n *Notice) Error(content ...interface{})

+ func (n *Notice) Finish(content ...interface{})

+ func (n *Notice) Info(content ...interface{})

+ func (n *Notice) Log(content ...interface{})

+ func (n *Notice) Process(content ...interface{})

+ func (n *Notice) Service()

+ type Spider struct

+ func NewSpider(f Form, mode Mode) *Spider

+ func (s *Spider) Debug() *Spider

+ func (s *Spider) Start() error

Other modules containing this package

github.com/PeterYangs/article-spider

github.com/PeterYangs/article-spider/v2

github.com/PeterYangs/article-spider/v4