s3pipe

Documentation

Index

• Constants
• type Downloader
  • func (dl Downloader) Download(ctx context.Context, input *s3.GetObjectInput) io.ReadCloser

Constants

const DefaultReadBufferSize = 32 * 1024

These values are very conservative defaults.

Types

type Downloader

type Downloader s3manager.Downloader

func (Downloader) Download

func (dl Downloader) Download(ctx context.Context, input *s3.GetObjectInput) io.ReadCloser

Download creates a new reader that reads the contents of an S3 object. It uses a downloader to fetch the file in chunks to enable recovery from read errors (e.g., connection resets). It prefetches the next chunk while the previous one is being processed. The chunk size is controlled by dl.PartSize and cost is 2 times dl.PartSize due to double buffer. Making the chunk size large reduces S3 API calls at the expense of memory.