snapshot

Documentation

Index

• type ProviderFailure
• type Snapshot
  • func NewSnapshot(fileName string) *Snapshot
  • func (snapshot *Snapshot) CollectFrom(pro []providers.Provider, collector func(job *providers.Job), ...)
  • func (snapshot *Snapshot) Collector() func(job *providers.Job)
  • func (snapshot *Snapshot) Erase()
  • func (snapshot *Snapshot) RetrieveJobs(fn func(job *providers.Job)) error
  • func (snapshot *Snapshot) Save() error

Types

type ProviderFailure

type ProviderFailure struct {
	Provider *providers.Provider
	Error    error
}

ProviderFailure refers to an error occurred in a provider

type Snapshot

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot helps you to collect and store your scrapped jobs

func NewSnapshot

func NewSnapshot(fileName string) *Snapshot

NewSnapshot returns a new provider

func (*Snapshot) CollectFrom

func (snapshot *Snapshot) CollectFrom(pro []providers.Provider, collector func(job *providers.Job), fn func(int, *ProviderFailure))

CollectFrom a slice of providers

func (*Snapshot) Collector

func (snapshot *Snapshot) Collector() func(job *providers.Job)

Collector returns a thread-safe function to use with a provider

Example :

snap := snapshot.NewSnapshot("./test")
p.RetrieveJobs(snap.Collector())

func (*Snapshot) Erase

func (snapshot *Snapshot) Erase()

Erase removes every entry of a snapshot

func (*Snapshot) RetrieveJobs

func (snapshot *Snapshot) RetrieveJobs(fn func(job *providers.Job)) error

RetrieveJobs replays your snapshot jobs

func (*Snapshot) Save

func (snapshot *Snapshot) Save() error

Save compresses your collected data and saves it