chunky

README

Chunky

Chunky is a high performance, deduplicated, object store. It is designed to be used as a backend for efficiently sharing machine learning datasets and weights between machines while minimizing network bandwidth.

Chunky is in a very early stage of development. It is not ready for general production use.

Features

• FastCDC content defined chunking.
• LZ4 transit compression.
• HighwayHash based content hashing.
• Offline network bypass mode.

Links

chunky-server

License

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtai one at https://mozilla.org/MPL/2.0/.

For more information see the LICENSE file.

Documentation

Index

• Constants
• type ChunkClient
  • func NewChunkClient(httpClient *http.Client, url string, cache storage.Storage) *ChunkClient
  • func (c *ChunkClient) Fetch(ctx context.Context, id contenthash.Hash) ([]byte, error)
  • func (c *ChunkClient) Push(ctx context.Context, id contenthash.Hash, data []byte) error
• type Client
  • func NewClient(grpcObjectClient pb.ObjectClient, chunkClient *ChunkClient, ...) *Client
  • func (c *Client) Fetch(ctx context.Context, id contenthash.Hash, opts *FetchOpts) (io.ReadCloser, error)
  • func (c *Client) Push(ctx context.Context, src io.Reader, opts *PushOpts) (*pb.ObjectManifest, error)
• type FetchOpts
• type PushOpts

Constants

const (
	// DefaultChunkSize is the default chunk size (1 MB).
	DefaultChunkSize = 1000000
	// DefaultBufferSize is the number of chunks to buffer in memory.
	DefaultBufferSize = 4
	// DefaultNetworkThreads is the number of connections to use for network ops.
	DefaultNetworkThreads = 4
	// DefaultRetryAttempts is the number of times to retry a failed operation.
	DefaultRetryAttempts = 3
)

Types

type ChunkClient

type ChunkClient struct {
	// contains filtered or unexported fields
}

ChunkClient is a client for fetching and pushing chunks.

func NewChunkClient

func NewChunkClient(httpClient *http.Client, url string, cache storage.Storage) *ChunkClient

NewChunkClient creates a new chunk client.

func (*ChunkClient) Fetch

func (c *ChunkClient) Fetch(ctx context.Context, id contenthash.Hash) ([]byte, error)

Fetch retrieves a chunk from the server.

func (*ChunkClient) Push

func (c *ChunkClient) Push(ctx context.Context, id contenthash.Hash, data []byte) error

Push pushes a chunk to the server.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is an object storage client.

func NewClient

func NewClient(grpcObjectClient pb.ObjectClient, chunkClient *ChunkClient, cache storage.Storage) *Client

NewClient creates a new object storage client.

func (*Client) Fetch

func (c *Client) Fetch(ctx context.Context, id contenthash.Hash, opts *FetchOpts) (io.ReadCloser, error)

Fetch an object from a remote server.

func (*Client) Push

func (c *Client) Push(ctx context.Context, src io.Reader, opts *PushOpts) (*pb.ObjectManifest, error)

Push an object to a remote server.

type FetchOpts

type FetchOpts struct {
	// NetworkThreads is the number of parallel threads to use for fetching chunks.
	NetworkThreads *int
	// BufferSize is the maximum number of chunks to buffer in memory.
	// Larger values will result in greater throughput, but will use more memory.
	BufferSize *int
	// RetryAttempts is the number of times to retry fetching a chunk before giving up.
	RetryAttempts *uint
}

FetchOpts is used to configure the behaviour of fetching an object.

type PushOpts

type PushOpts struct {
	// ChunkSize is the nominal size of each chunk.
	ChunkSize int64
	// NetworkThreads is the number of parallel threads to use for pushing chunks.
	NetworkThreads *int
	// RetryAttempts is the number of times to retry pushing a chunk before giving up.
	RetryAttempts *uint
}

PushOpts is used to configure the behaviour of pushing an object.