downloader

Documentation

Index

• Constants
• type DownloadItem
• type Downloader
  • func NewDownloader(remoteStorage data.Storage) *Downloader
  • func (d *Downloader) AddToQueue(URI string, callback func(string, []byte), pin bool)
  • func (d *Downloader) ImportFailedQueueSize() int
  • func (d *Downloader) PrintLogInfo(period time.Duration)
  • func (d *Downloader) QueueSize() int32
  • func (d *Downloader) Start()
  • func (d *Downloader) Stop()
  • func (d *Downloader) TotalItemsAdded() int32

Constants

const (
	// ImportQueueRoutines is the number of parallel routines processing the
	// remote file download queue.
	ImportQueueRoutines = 32
	// ImportRetrieveTimeout the maximum duration the import queue will wait
	// for retrieving a remote file.
	ImportRetrieveTimeout = 3 * time.Minute
	// ImportPinTimeout is the maximum duration the import queue will wait
	// for pinning a remote file.
	ImportPinTimeout = 2 * time.Minute
	// MaxFileSize is the maximum size of a file that can be imported.
	MaxFileSize = 100 * 1024 * 1024 // 100MB

)

Types

type DownloadItem

type DownloadItem struct {
	URI      string
	Callback func(URI string, data []byte)
	Pin      bool
}

DownloadItem is a remote file to be downloaded.

type Downloader

type Downloader struct {
	RemoteStorage data.Storage
	// contains filtered or unexported fields
}

Downloader is a remote file downloader that uses queues.

func NewDownloader

func NewDownloader(remoteStorage data.Storage) *Downloader

NewDownloader returns a new Downloader. After creating a new instance, the process should be started by calling "Start()"

func (*Downloader) AddToQueue

func (d *Downloader) AddToQueue(URI string, callback func(string, []byte), pin bool)

AddToQueue adds a new URI to the queue for being imported remotely. Once the file is downloaded, the callback is called with the URI as argument.

func (*Downloader) ImportFailedQueueSize

func (d *Downloader) ImportFailedQueueSize() int

ImportFailedQueueSize is the size of the list of remote census imported that failed.

func (*Downloader) PrintLogInfo

func (d *Downloader) PrintLogInfo(period time.Duration)

PrintLogInfo prints the current status of the downloader. This method is blocking.

func (*Downloader) QueueSize

func (d *Downloader) QueueSize() int32

QueueSize returns the size of the import census queue.

func (*Downloader) Start

func (d *Downloader) Start()

Start starts the import queue daemons. This is a non-blocking method.

func (*Downloader) Stop

func (d *Downloader) Stop()

Stop stops the import queue daemons.

func (*Downloader) TotalItemsAdded

func (d *Downloader) TotalItemsAdded() int32

TotalItemsAdded is the number of items that has been added to the queue on this instance.