downloader

Documentation

Index

• type Downloader
  • func New(qReader *kafka.Reader, s3Config *config.S3Config, ...) *Downloader

Types

type Downloader

type Downloader struct {
	*worker.Worker
	// contains filtered or unexported fields
}

Downloader reads download jobs from kafka, downloads pictures from posts, stores them in S3

and writes the S3 path to posts

func New

func New(qReader *kafka.Reader, s3Config *config.S3Config, postgresConfig *config.PostgresConfig) *Downloader

New returns an initilized scraper