downloader

Documentation

Index

• Constants
• type Job
  • func NewJob(object *gcs.MinObject, bucket gcs.Bucket, fileInfoCache *lru.Cache, ...) (job *Job)
  • func (job *Job) Download(ctx context.Context, offset int64, waitForDownload bool) (jobStatus JobStatus, err error)
  • func (job *Job) GetStatus() JobStatus
  • func (job *Job) Invalidate()
• type JobManager
  • func NewJobManager(fileInfoCache *lru.Cache, filePerm os.FileMode, dirPerm os.FileMode, ...) (jm *JobManager)
  • func (jm *JobManager) CreateJobIfNotExists(object *gcs.MinObject, bucket gcs.Bucket) (job *Job)
  • func (jm *JobManager) Destroy()
  • func (jm *JobManager) GetJob(objectName string, bucketName string) *Job
  • func (jm *JobManager) InvalidateAndRemoveJob(objectName string, bucketName string)
• type JobStatus

Constants

const (
	NotStarted  jobStatusName = "NotStarted"
	Downloading jobStatusName = "Downloading"
	Completed   jobStatusName = "Completed"
	Failed      jobStatusName = "Failed"
	Invalid     jobStatusName = "Invalid"
)

const ReadChunkSize = 8 * cacheutil.MiB

Types

type Job

type Job struct {
	// contains filtered or unexported fields
}

Job downloads the requested object from GCS into the specified local file path with given permissions and ownership.

func NewJob

func NewJob(object *gcs.MinObject, bucket gcs.Bucket, fileInfoCache *lru.Cache,
	sequentialReadSizeMb int32, fileSpec data.FileSpec, removeJobCallback func()) (job *Job)

func (*Job) Download

func (job *Job) Download(ctx context.Context, offset int64, waitForDownload bool) (jobStatus JobStatus, err error)

Download downloads object till the given offset and returns the status of job. If the object is already downloaded or there was failure in download, then it returns the job status. The caller shouldn't read data from file in cache if jobStatus is Failed or Invalid.

Acquires and releases LOCK(job.mu)

func (*Job) GetStatus

func (job *Job) GetStatus() JobStatus

GetStatus returns the status of download job.

Acquires and releases LOCK(job.mu)

func (*Job) Invalidate

func (job *Job) Invalidate()

Invalidate invalidates the download job i.e. changes the state to Invalid. If the async download is in progress, this function cancels that. The caller should not read from the file in cache if job is in Invalid state. Note: job.removeJobCallback function is also executed as part of invalidation.

Acquires and releases LOCK(job.mu)

type JobManager

type JobManager struct {
	// contains filtered or unexported fields
}

JobManager is responsible for maintaining, getting and removing file download jobs. It is created only once at the time of mounting.

func NewJobManager

func NewJobManager(fileInfoCache *lru.Cache, filePerm os.FileMode, dirPerm os.FileMode, cacheDir string, sequentialReadSizeMb int32) (jm *JobManager)

func (*JobManager) CreateJobIfNotExists

func (jm *JobManager) CreateJobIfNotExists(object *gcs.MinObject, bucket gcs.Bucket) (job *Job)

CreateJobIfNotExists creates and returns downloader.Job for given object and bucket. If there is already an existing job then this method returns that.

Acquires and releases Lock(jm.mu)

func (*JobManager) Destroy

func (jm *JobManager) Destroy()

Destroy invalidates and deletes all the jobs that job manager is managing.

Acquires and releases Lock(jm.mu)

func (*JobManager) GetJob

func (jm *JobManager) GetJob(objectName string, bucketName string) *Job

GetJob returns downloader.Job for given object and bucket if present. If the job is not present, it returns nil.

Acquires and releases Lock(jm.mu)

func (*JobManager) InvalidateAndRemoveJob

func (jm *JobManager) InvalidateAndRemoveJob(objectName string, bucketName string)

InvalidateAndRemoveJob invalidates downloader.Job for given object and bucket. If there is no existing job present then this method does nothing. Note: Invalidating a job also removes job from jm.jobs map.

Acquires and releases Lock(jm.mu)

type JobStatus

type JobStatus struct {
	Name   jobStatusName
	Err    error
	Offset int64
}

JobStatus represents the status of job.