backend

package
v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 6, 2020 License: Apache-2.0 Imports: 12 Imported by: 29

Documentation

Overview

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

View Source 
const (
	DefaultPartSize    int64             = int64(64 * memsize.MB)
	DefaultBufferGuard datasize.ByteSize = 10 * datasize.MB
	DefaultConcurrency int               = 10
	DefaultListMaxKeys int               = 250
)
View Source 
const NoopNamespace = "__noop__"

NoopNamespace is a special namespace which always returns a NoopClient.

Variables

View Source 
var (
	ErrNamespaceNotFound = errors.New("no matches for namespace")
)

Manager errors.

Functions

func Register 

func Register(name string, factory ClientFactory)

Register registers new Factory with corresponding backend client name.

Types

type Auth 

type Auth map[string]AuthConfig

Auth defines auth credentials for corresponding namespaces. It has to be different due to langley secrets overlay structure.

type AuthConfig 

type AuthConfig map[string]interface{}

AuthConfig defines the union of authentication credentials for all type of remote backends.

type BandwidthWatcher 

type BandwidthWatcher struct {
	// contains filtered or unexported fields
}

BandwidthWatcher is a hashring.Watcher which adjusts bandwidth on throttled backends when hashring membership changes.

func NewBandwidthWatcher 

func NewBandwidthWatcher(manager *Manager) *BandwidthWatcher

NewBandwidthWatcher creates a new BandwidthWatcher for manager.

func (*BandwidthWatcher) Notify 

func (w *BandwidthWatcher) Notify(latest stringset.Set)

Notify splits bandwidth across the size of latest.

type Client 

type Client interface {
	// Stat returns blob info for name. All implementations should return
	// backenderrors.ErrBlobNotFound when the blob was not found.
	//
	// Stat is useful when we need to quickly know if a blob exists (and maybe
	// some basic information about it), without downloading the entire blob,
	// which may be very large.
	Stat(namespace, name string) (*core.BlobInfo, error)

	// Upload uploads src into name.
	Upload(namespace, name string, src io.Reader) error

	// Download downloads name into dst. All implementations should return
	// backenderrors.ErrBlobNotFound when the blob was not found.
	Download(namespace, name string, dst io.Writer) error

	// List lists entries whose names start with prefix.
	List(prefix string, opts ...ListOption) (*ListResult, error)
}

Client defines an interface for accessing blobs on a remote storage backend.

Implementations of Client must be thread-safe, since they are cached and used concurrently by Manager.

type ClientFactory 

type ClientFactory interface {
	Create(config interface{}, authConfig interface{}) (Client, error)
}

ClientFactory creates backend client given name.

type Config 

type Config struct {
	Namespace string                 `yaml:"namespace"`
	Backend   map[string]interface{} `yaml:"backend"`

	// If enabled, throttles upload / download bandwidth.
	Bandwidth bandwidth.Config `yaml:"bandwidth"`
}

Config defines the union of configuration for all backends. Only one is allowed for each config file.

type ListOption added in v0.1.3 

type ListOption func(*ListOptions)

ListOption is used to configure list calls via variadic functional options.

func ListWithContinuationToken added in v0.1.3 

func ListWithContinuationToken(token string) ListOption

ListWithContinuationToken configures the list command return results starting at the continuation token if pagination is enabled.

func ListWithMaxKeys added in v0.1.3 

func ListWithMaxKeys(max int) ListOption

ListWithMaxKeys configures the list command to return a max number of keys if pagination is enabled.

func ListWithPagination added in v0.1.3 

func ListWithPagination() ListOption

ListWithPagination configures the list command to use pagination.

type ListOptions added in v0.1.3 

type ListOptions struct {
	Paginated         bool
	MaxKeys           int
	ContinuationToken string
}

ListOptions defines the options which can be specified when listing names. It is used to enable pagination in list requests.

func DefaultListOptions added in v0.1.3 

func DefaultListOptions() *ListOptions

DefaultListOptions defines the defaults for list operations.

type ListResult added in v0.1.3 

type ListResult struct {
	Names             []string
	ContinuationToken string
}

ListResult defines the response from a client list operation. The names of the entries found will always be populated, and the continuation token will only be populated if pagination was enabled and there are remaining entries to list.

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages backend clients for namespace regular expressions.

func ManagerFixture 

func ManagerFixture() *Manager

ManagerFixture returns a Manager with no clients for testing purposes.

func NewManager 

func NewManager(configs []Config, auth AuthConfig) (*Manager, error)

NewManager creates a new backend Manager.

func (*Manager) AdjustBandwidth 

func (m *Manager) AdjustBandwidth(denominator int) error

AdjustBandwidth adjusts bandwidth limits across all throttled clients to the originally configured bandwidth divided by denominator.

func (*Manager) GetClient 

func (m *Manager) GetClient(namespace string) (Client, error)

GetClient matches namespace to the configured Client. Returns ErrNamespaceNotFound if no clients match namespace.

func (*Manager) Register 

func (m *Manager) Register(namespace string, c Client) error

Register dynamically registers a namespace with a provided client. Register should be primarily used for testing purposes -- normally, namespaces should be statically configured and provided upon construction of the Manager.

type NoopClient 

type NoopClient struct{}

NoopClient is a special Client whose uploads always succeeds and whose blob lookups always 404. It is useful for users who want to operate on blobs that will be temporarily stored in the origin cluster and not backed up in remote storage.

func (NoopClient) Download 

func (c NoopClient) Download(namespace, name string, dst io.Writer) error

Download always returns ErrBlobNotFound.

func (NoopClient) List 

func (c NoopClient) List(prefix string, opts ...ListOption) (*ListResult, error)

List always returns nil.

func (NoopClient) Stat 

func (c NoopClient) Stat(namespace, name string) (*core.BlobInfo, error)

Stat always returns ErrBlobNotFound.

func (NoopClient) Upload 

func (c NoopClient) Upload(namespace, name string, src io.Reader) error

Upload always returns nil.

type ThrottledClient 

type ThrottledClient struct {
	Client
	// contains filtered or unexported fields
}

ThrottledClient is a backend client with speed limit.

func (*ThrottledClient) Download 

func (c *ThrottledClient) Download(namespace, name string, dst io.Writer) error

Download downloads name into dst.

func (*ThrottledClient) EgressLimit 

func (c *ThrottledClient) EgressLimit() int64

EgressLimit returns egress limit.

func (*ThrottledClient) IngressLimit 

func (c *ThrottledClient) IngressLimit() int64

IngressLimit returns ingress limit.

func (*ThrottledClient) Upload 

func (c *ThrottledClient) Upload(namespace, name string, src io.Reader) error

Upload uploads src into name.

Source Files

View all Source files

Directories

Path Synopsis
webhdfs
security

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.