source

package
v2.0.9-beta.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 23, 2023 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	LastModifiedLayout = http.TimeFormat
	ExpireLayout       = http.TimeFormat
)
View Source 
const (
	// Range is different with HTTP Range, it's without "bytes=" prefix
	Range = "X-Dragonfly-Range" // startIndex-endIndex
)
View Source 
const (
	TimeFormat = "Mon, 02 Jan 2006 15:04:05 GMT"
)
View Source 
const (
	UnknownSourceFileLen = -2
)

Variables

View Source 
var (
	ListMetadataScheme       = "d7ylist"
	ListMetadataOriginScheme = "X-Dragonfly-List-Origin-Scheme"
	ListMetadataExpire       = "X-Dragonfly-List-Expire"
)
View Source 
var (
	// ErrResourceNotReachable represents the url resource is a not reachable.
	ErrResourceNotReachable = errors.New("resource is not reachable")

	// ErrNoClientFound represents no source client to resolve url
	ErrNoClientFound = errors.New("no source client found")

	// ErrClientNotSupportList represents the source client not support list action
	ErrClientNotSupportList = errors.New("source client not support list")

	// ErrClientNotSupportContentLength represents the source client not support get content length action
	ErrClientNotSupportContentLength = errors.New("source client not support get content length")

	// ErrClientNotSupportGetMetadata represents the source client not support get metadata
	ErrClientNotSupportGetMetadata = errors.New("source client not support get metadata")
)
View Source 
var ProxyEnv = "D7Y_SOURCE_PROXY"

Functions

func CanonicalHeaderKey added in v2.0.2 

func CanonicalHeaderKey(s string) string

func CheckResponseCode added in v2.0.2 

func CheckResponseCode(respCode int, allowed []int) error

CheckResponseCode returns UnexpectedStatusError if the given response code is not one of the allowed status codes; otherwise nil.

func DefaultTransport added in v2.0.9 

func DefaultTransport() *http.Transport

func GetContentLength 

func GetContentLength(request *Request) (int64, error)

func GetLastModified added in v2.0.2 

func GetLastModified(request *Request) (int64, error)

func InitSourceClients added in v2.0.9 

func InitSourceClients(opts map[string]interface{}) error

InitSourceClients will initialize all resource clients which registered by RegisterBuilder.

func IsExpired 

func IsExpired(request *Request, info *ExpireInfo) (bool, error)

func IsNoClientFoundError added in v2.0.2 

func IsNoClientFoundError(err error) bool

func IsResourceNotReachableError added in v2.0.2 

func IsResourceNotReachableError(err error) bool

func IsSupportRange 

func IsSupportRange(request *Request) (bool, error)

func ListClients added in v2.0.3 

func ListClients() []string

func ParseToHTTPClient added in v2.0.9 

func ParseToHTTPClient(optionYaml []byte) (*http.Client, error)

func Register 

func Register(scheme string, resourceClient ResourceClient, adaptor RequestAdapter, hooks ...Hook) error

func RegisterBuilder added in v2.0.9 

func RegisterBuilder(scheme string, builder ResourceClientBuilder)

RegisterBuilder register ResourceClientBuilder into global resourceClientBuilder, the InitSourceClients will use it.

func RequestEq added in v2.0.2 

func RequestEq(url string) gomock.Matcher

RequestEq for gomock

func UnRegister 

func UnRegister(scheme string)

func UnRegisterBuilder added in v2.0.9 

func UnRegisterBuilder(scheme string)

func UpdatePluginDir added in v2.0.2 

func UpdatePluginDir(pluginDir string)

func UpdateTransportOption added in v2.0.9 

func UpdateTransportOption(transport *http.Transport, optionYaml []byte) error

func WithContentLength added in v2.0.2 

func WithContentLength(length int64) func(*Response)

func WithExpireInfo added in v2.0.2 

func WithExpireInfo(info ExpireInfo) func(*Response)

WithExpireInfo is used for no-http resource client to set expire info

func WithHeader added in v2.0.2 

func WithHeader(header map[string]string) func(*Response)

func WithStatus added in v2.0.2 

func WithStatus(code int, status string) func(*Response)

func WithTemporary added in v2.0.5 

func WithTemporary(temporary bool) func(*Response)

func WithValidate added in v2.0.3 

func WithValidate(validate func() error) func(*Response)

Types

type ClientManager 

type ClientManager interface {
	// Register registers a source client with scheme
	Register(scheme string, resourceClient ResourceClient, adapter RequestAdapter, hook ...Hook) error

	// UnRegister revoke a source client from manager
	UnRegister(scheme string)

	// GetClient gets a source client by scheme
	GetClient(scheme string, options ...Option) (ResourceClient, bool)

	// ListClients lists all supported client scheme
	ListClients() []string
}

func NewManager 

func NewManager() ClientManager

type ExpireInfo added in v2.0.2 

type ExpireInfo struct {
	Expire       string // Mon, 02 Jan 2006 15:04:05 GMT
	LastModified string // Mon, 02 Jan 2006 15:04:05 GMT
	ETag         string
}
type Header map[string][]string

A Header represents the key-value pairs in a Dragonfly source header.

The keys should be in canonical form, as returned by CanonicalHeaderKey.

func (Header) Add added in v2.0.2 

func (h Header) Add(key, value string)

Add adds the key, value pair to the header. It appends to any existing values associated with key. The key is case insensitive; it is canonicalized by CanonicalHeaderKey.

func (Header) Clone added in v2.0.2 

func (h Header) Clone() Header

Clone returns a copy of h or nil if h is nil.

func (Header) Del added in v2.0.2 

func (h Header) Del(key string)

Del deletes the values associated with key. The key is case insensitive; it is canonicalized by CanonicalHeaderKey.

func (Header) Get added in v2.0.2 

func (h Header) Get(key string) string

Get gets the first value associated with the given key. If there are no values associated with the key, Get returns "". It is case insensitive; textproto.CanonicalMIMEHeaderKey is used to canonicalize the provided key. To use non-canonical keys, access the map directly.

func (Header) Set added in v2.0.2 

func (h Header) Set(key, value string)

Set sets the header entries associated with key to the single element value. It replaces any existing values associated with key. The key is case insensitive; it is canonicalized by textproto.CanonicalMIMEHeaderKey. To use non-canonical keys, assign to the map directly.

func (Header) Values added in v2.0.2 

func (h Header) Values(key string) []string

Values returns all values associated with the given key. It is case insensitive; textproto.CanonicalMIMEHeaderKey is used to canonicalize the provided key. To use non-canonical keys, access the map directly. The returned slice is not a copy.

type Hook added in v2.0.2 

type Hook interface {
	BeforeRequest(request *Request) error
	AfterResponse(response *Response) error
}

Hook TODO hook

type ListMetadata added in v2.0.8 

type ListMetadata struct {
	URLEntries []URLEntry
}

type ListMetadataClient added in v2.0.8 

type ListMetadataClient struct {
}

func (*ListMetadataClient) Download added in v2.0.8 

func (lm *ListMetadataClient) Download(request *Request) (*Response, error)

func (*ListMetadataClient) GetContentLength added in v2.0.8 

func (lm *ListMetadataClient) GetContentLength(request *Request) (int64, error)

func (*ListMetadataClient) GetLastModified added in v2.0.8 

func (lm *ListMetadataClient) GetLastModified(request *Request) (int64, error)

func (*ListMetadataClient) IsExpired added in v2.0.8 

func (lm *ListMetadataClient) IsExpired(request *Request, info *ExpireInfo) (bool, error)

func (*ListMetadataClient) IsSupportRange added in v2.0.8 

func (lm *ListMetadataClient) IsSupportRange(request *Request) (bool, error)

type Metadata added in v2.0.5 

type Metadata struct {
	Status     string
	StatusCode int
	Header     Header
	// SupportRange indicates source supports partial download, like Range in http request
	SupportRange bool
	// ContentLength indicates the current content length for the target request
	// ContentLength int64
	// TotalContentLength indicates the total content length for the target request
	// eg, for http response header:
	//      Content-Range: bytes 0-9/2443
	// 2443 is the TotalContentLength, 10 is the ContentLength
	TotalContentLength int64

	Validate  func() error
	Temporary bool
}

func GetMetadata added in v2.0.5 

func GetMetadata(request *Request) (*Metadata, error)

type Option added in v2.0.2 

type Option func(c *clientManager)

type Request added in v2.0.2 

type Request struct {
	URL    *url.URL
	Header Header
	// contains filtered or unexported fields
}

func NewListMetadataRequest added in v2.0.8 

func NewListMetadataRequest(request *Request) *Request

func NewRequest added in v2.0.2 

func NewRequest(rawURL string) (*Request, error)

func NewRequestWithContext added in v2.0.2 

func NewRequestWithContext(ctx context.Context, rawURL string, header map[string]string) (*Request, error)

func NewRequestWithHeader added in v2.0.2 

func NewRequestWithHeader(rawURL string, header map[string]string) (*Request, error)

func (*Request) Clone added in v2.0.2 

func (r *Request) Clone(ctx context.Context) *Request

Clone returns a deep copy of r with its context changed to ctx. The provided ctx must be non-nil.

For an outgoing client request, the context controls the entire lifetime of a request and its response: obtaining a connection, sending the request, and reading the response headers and body.

func (*Request) Context added in v2.0.2 

func (r *Request) Context() context.Context

Context returns the request's context. To change the context, use WithContext.

The returned context is always non-nil; it defaults to the background context.

For outgoing client requests, the context controls cancellation.

For incoming server requests, the context is canceled when the client's connection closes, the request is canceled (with HTTP/2), or when the ServeHTTP method returns.

func (*Request) WithContext added in v2.0.2 

func (r *Request) WithContext(ctx context.Context) *Request

WithContext returns a shallow copy of r with its context changed to ctx. The provided ctx must be non-nil.

For outgoing client request, the context controls the entire lifetime of a request and its response: obtaining a connection, sending the request, and reading the response headers and body.

To create a new request with a context, use NewRequestWithContext. To change the context of a request, such as an incoming request you want to modify before sending back out, use Request.Clone. Between those two uses, it's rare to need WithContext.

type RequestAdapter added in v2.0.4 

type RequestAdapter func(request *Request) *Request

type ResourceClient 

type ResourceClient interface {
	// GetContentLength get length of resource content
	// return source.UnknownSourceFileLen if response status is not StatusOK and StatusPartialContent
	GetContentLength(request *Request) (int64, error)

	// IsSupportRange checks if resource supports breakpoint continuation
	// return false if response status is not StatusPartialContent
	IsSupportRange(request *Request) (bool, error)

	// IsExpired checks if a resource received or stored is the same.
	// return false and non-nil err to prevent the source from exploding if
	// fails to get the result, it is considered that the source has not expired
	IsExpired(request *Request, info *ExpireInfo) (bool, error)

	// Download downloads from source
	Download(request *Request) (*Response, error)

	// GetLastModified gets last modified timestamp milliseconds of resource
	GetLastModified(request *Request) (int64, error)
}

ResourceClient defines the API interface to interact with source.

func LoadPlugin 

func LoadPlugin(dir, scheme string) (ResourceClient, error)

type ResourceClientBuilder added in v2.0.9 

type ResourceClientBuilder interface {
	// Build return the target resource with custom option
	Build(optionYaml []byte) (resourceClient ResourceClient, adaptor RequestAdapter, hooks []Hook, err error)
}

ResourceClientBuilder is used to build resource client with custom option

func NewPlainResourceClientBuilder added in v2.0.9 

func NewPlainResourceClientBuilder(
	build func(optionYaml []byte) (resourceClient ResourceClient, adaptor RequestAdapter, hooks []Hook, err error)) ResourceClientBuilder

type ResourceLister added in v2.0.2 

type ResourceLister interface {
	// List only list current level resources in request url
	// the request may represent a single file or a directory
	// if the request represent a directory, the result should return all file entries and subdirectory entries below the request directory
	// if the request represent a single file, the result should return a single file entry or empty slice
	List(request *Request) (urls []URLEntry, err error)
}

ResourceLister defines the interface to list all downloadable resources in request url

type ResourceMetadataGetter added in v2.0.5 

type ResourceMetadataGetter interface {
	GetMetadata(request *Request) (*Metadata, error)
}

ResourceMetadataGetter defines the API interface to get metadata for special resource The metadata will be used for concurrent multiple pieces downloading

type Response added in v2.0.2 

type Response struct {
	Status        string
	StatusCode    int
	Header        Header
	Body          io.ReadCloser
	ContentLength int64
	// Validate this response is okay to transfer in p2p network, like status 200 or 206 in http is valid to do this,
	// otherwise return status code to original client
	Validate func() error
	// Temporary indecates the error whether the error is temporary, if is true, we can retry it later
	Temporary bool
}

func Download 

func Download(request *Request) (*Response, error)

func NewResponse added in v2.0.2 

func NewResponse(rc io.ReadCloser, opts ...func(*Response)) *Response

func (*Response) ExpireInfo added in v2.0.2 

func (resp *Response) ExpireInfo() ExpireInfo

type URLEntry added in v2.0.5 

type URLEntry struct {
	//URL download url
	URL *url.URL

	// Name returns the name of the file (or subdirectory) described by the entry.
	// Name will be used in recursive downloading as file name or subdirectory name
	// This name is only the final element of the path (the commonv1 name), not the entire path.
	// For example, Name would return "hello.go" not "home/gopher/hello.go".
	Name string
	// IsDir reports whether the entry describes a directory.
	IsDir bool
	// Entry attribute, like oss md5 and expire time
	Attribute map[string]string
}

URLEntry is an entry which read from url with specific protocol It is used in recursive downloading

func List added in v2.0.2 

func List(request *Request) ([]URLEntry, error)

type UnexpectedStatusCodeError added in v2.0.2 

type UnexpectedStatusCodeError struct {
	// contains filtered or unexported fields
}

UnexpectedStatusCodeError is returned when a source responds with neither an error nor with a status code indicating success.

func (UnexpectedStatusCodeError) Error added in v2.0.2 

func (e UnexpectedStatusCodeError) Error() string

Error implements interface error

func (UnexpectedStatusCodeError) Got added in v2.0.2 

func (e UnexpectedStatusCodeError) Got() int

Got is the actual status code returned by source.

Source Files

View all Source files

Directories

Path Synopsis
clients
example
hdfsprotocol
httpprotocol
orasprotocol
ossprotocol
s3protocol
Package mocks is a generated GoMock package.
 Package mocks is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.