net

Documentation

Index

• Constants
• func DefaultHttpRequestFunc(ctx context.Context, params *HttpRequestParams) (*http.Response, error)
• func GetRangedHttpReader(readCloser io.ReadCloser, offset, length int64) (io.ReadCloser, error)
• func HttpClient() *http.Client
• func ProcessHeader(origin, override http.Header) http.Header
• func RequestHttp(ctx context.Context, httpMethod string, headerOverride http.Header, URL string) (*http.Response, error)
• func ServeHTTP(w http.ResponseWriter, r *http.Request, name string, modTime time.Time, ...)
• type Buf
  • func NewBuf(ctx context.Context, maxSize int, id int) *Buf
  • func (br *Buf) Close()
  • func (br *Buf) Read(p []byte) (n int, err error)
  • func (br *Buf) Reset(size int)
  • func (br *Buf) Write(p []byte) (n int, err error)
• type Downloader
  • func NewDownloader(options ...func(*Downloader)) *Downloader
  • func (d Downloader) Download(ctx context.Context, p *HttpRequestParams) (readCloser io.ReadCloser, err error)
• type HttpRequestFunc
• type HttpRequestParams
• type LimitedReadCloser
  • func (l *LimitedReadCloser) Close() error
  • func (l *LimitedReadCloser) Read(buf []byte) (int, error)
• type MultiReadCloser
  • func NewMultiReadCloser(buf *Buf, c closerFunc, fb finishBufFUnc) *MultiReadCloser
  • func (mr MultiReadCloser) Close() error
  • func (mr MultiReadCloser) Read(p []byte) (n int, err error)

Constants

const DefaultDownloadConcurrency = 2

DefaultDownloadConcurrency is the default number of goroutines to spin up when using Download().

const DefaultDownloadPartSize = 1024 * 1024 * 10

DefaultDownloadPartSize is the default range of bytes to get at a time when using Download().

const DefaultPartBodyMaxRetries = 3

DefaultPartBodyMaxRetries is the default number of retries to make when a part fails to download.

Functions

func DefaultHttpRequestFunc

func DefaultHttpRequestFunc(ctx context.Context, params *HttpRequestParams) (*http.Response, error)

func GetRangedHttpReader

func GetRangedHttpReader(readCloser io.ReadCloser, offset, length int64) (io.ReadCloser, error)

GetRangedHttpReader some http server doesn't support "Range" header, so this function read readCloser with whole data, skip offset, then return ReaderCloser.

func HttpClient

func HttpClient() *http.Client

func ProcessHeader

func ProcessHeader(origin, override http.Header) http.Header

func RequestHttp

func RequestHttp(ctx context.Context, httpMethod string, headerOverride http.Header, URL string) (*http.Response, error)

RequestHttp deal with Header properly then send the request

func ServeHTTP

func ServeHTTP(w http.ResponseWriter, r *http.Request, name string, modTime time.Time, size int64, RangeReaderFunc model.RangeReaderFunc)

ServeHTTP replies to the request using the content in the provided RangeReadCloser. The main benefit of ServeHTTP over io.Copy is that it handles Range requests properly, sets the MIME type, and handles If-Match, If-Unmodified-Since, If-None-Match, If-Modified-Since, and If-Range requests.

If the response's Content-Type header is not set, ServeHTTP first tries to deduce the type from name's file extension and, if that fails, falls back to reading the first block of the content and passing it to DetectContentType. The name is otherwise unused; in particular it can be empty and is never sent in the response.

If modtime is not the zero time or Unix epoch, ServeHTTP includes it in a Last-Modified header in the response. If the request includes an If-Modified-Since header, ServeHTTP uses modtime to decide whether the content needs to be sent at all.

The content's RangeReadCloser method must work: ServeHTTP gives a range, caller will give the reader for that Range.

If the caller has set w's ETag header formatted per RFC 7232, section 2.3, ServeHTTP uses it to handle requests using If-Match, If-None-Match, or If-Range.

Types

type Buf

type Buf struct {
	// contains filtered or unexported fields
}

func NewBuf

func NewBuf(ctx context.Context, maxSize int, id int) *Buf

NewBuf is a buffer that can have 1 read & 1 write at the same time. when read is faster write, immediately feed data to read after written

func (*Buf) Close

func (br *Buf) Close()

func (*Buf) Read

func (br *Buf) Read(p []byte) (n int, err error)

func (*Buf) Reset

func (br *Buf) Reset(size int)

func (*Buf) Write

func (br *Buf) Write(p []byte) (n int, err error)

type Downloader

type Downloader struct {
	PartSize int

	// PartBodyMaxRetries is the number of retry attempts to make for failed part downloads.
	PartBodyMaxRetries int

	// The number of goroutines to spin up in parallel when sending parts.
	// If this is set to zero, the DefaultDownloadConcurrency value will be used.
	//
	// Concurrency of 1 will download the parts sequentially.
	Concurrency int

	//RequestParam        HttpRequestParams
	HttpClient HttpRequestFunc
}

func NewDownloader

func NewDownloader(options ...func(*Downloader)) *Downloader

func (Downloader) Download

func (d Downloader) Download(ctx context.Context, p *HttpRequestParams) (readCloser io.ReadCloser, err error)

Download The Downloader makes multi-thread http requests to remote URL, each chunk(except last one) has PartSize, cache some data, then return Reader with assembled data Supports range, do not support unknown FileSize, and will fail if FileSize is incorrect memory usage is at about Concurrency*PartSize, use this wisely

type HttpRequestFunc

type HttpRequestFunc func(ctx context.Context, params *HttpRequestParams) (*http.Response, error)

type HttpRequestParams

type HttpRequestParams struct {
	URL string
	//only want data within this range
	Range     http_range.Range
	HeaderRef http.Header
	//total file size
	Size int64
}

type LimitedReadCloser

type LimitedReadCloser struct {
	// contains filtered or unexported fields
}

LimitedReadCloser wraps a io.ReadCloser and limits the number of bytes that can be read from it.

func (*LimitedReadCloser) Close

func (l *LimitedReadCloser) Close() error

func (*LimitedReadCloser) Read

func (l *LimitedReadCloser) Read(buf []byte) (int, error)

type MultiReadCloser

type MultiReadCloser struct {
	// contains filtered or unexported fields
}

func NewMultiReadCloser

func NewMultiReadCloser(buf *Buf, c closerFunc, fb finishBufFUnc) *MultiReadCloser

NewMultiReadCloser to save memory, we re-use limited Buf, and feed data to Read()

func (MultiReadCloser) Close

func (mr MultiReadCloser) Close() error

func (MultiReadCloser) Read

func (mr MultiReadCloser) Read(p []byte) (n int, err error)