cas

README

cas

Content addressible storage in Go.

Package cas provides a framework for working with Content Addressible Storage (CAS).

Installation

As with most projects for Go, installation is as easy as go get.

Documentation

Please refer to the documentation on godoc.org for API documentation.

Contributing

Development of this project is ongoing. If you find a bug or have any suggestions, please open an issue.

If you'd like to contribute, please fork the repository and make changes. Pull requests are welcome.

Related projects and documentation

In progress

Licensing

This project is licensed under the 3-Clause BSD License. See the LICENSE in the repository.

Documentation

Overview

Package cas provides a framework for working with content addressible storage (CAS). This package only provides the interface, as several types of drivers are possible.

Index

• func NewHash() hash.Hash
• func ReadAll(s Storage, id ID) ([]byte, error)
• func VerifyID(s Storage, id ID) (bool, error)
• func VerifyStorage(ctx context.Context, storage Storage, onerror func(id ID) bool) error
• type Error
  • func (e Error) Error() string
• type ID
  • func CreateFromBytes(s Storage, b []byte) (ID, error)
  • func CreateFromString(s Storage, b string) (ID, error)
  • func NewID(data []byte) ID
  • func ParseID(id string) (ID, error)
  • func (id ID) IsZero() bool
  • func (id ID) String() string
• type IndexStorage
• type RemoveStorage
• type Storage

Examples

• Storage

Functions

func NewHash

func NewHash() hash.Hash

NewHash returns a new hash.Hash computing the checksum.

func ReadAll

func ReadAll(s Storage, id ID) ([]byte, error)

ReadAll reads all of the contents associated with the ID into a byte slice.

func VerifyID

func VerifyID(s Storage, id ID) (bool, error)

VerifyID loads an object from the storage, and checks that the contents have not been corrupted.

func VerifyStorage

func VerifyStorage(ctx context.Context, storage Storage, onerror func(id ID) bool) error

VerifyStorage scans the objects in the storage, and checks that the contents have not been corrupted. If there is concurrent access to the storage, some objects may not be checked.

A callback will be called for every mismatch detected. The ID will indicate the item for which the contents don't match. The callback should return true to indicate that the search should continue.

The storage must also support the IndexStorage interface.

Types

type Error

type Error string

Error records errors that are common across all concrete implementations of Storage.

const (
	ErrNotExist   Error = "object with ID does not exist"
	ErrNotRemover Error = "storage does not support removing of contents"
	ErrNotIndexer Error = "storage does not support indexing of contents"
)

Common errors that should be used by all concrete implementations of Storage.

func (Error) Error

func (e Error) Error() string

Error returns the message associated with the error.

type ID

type ID [sha1.Size]byte

An ID identifies a piece of content, which can be any binary blob. The ID is a checksum of the content, ensuring that identical data is always mapped to the same ID.

func CreateFromBytes

func CreateFromBytes(s Storage, b []byte) (ID, error)

CreateFromBytes is a utility function that inserts binary data into a storage.

func CreateFromString

func CreateFromString(s Storage, b string) (ID, error)

CreateFromString is a utility function that inserts binary data into a storage.

func NewID

func NewID(data []byte) ID

NewID calculates the ID for a binary blob.

func ParseID

func ParseID(id string) (ID, error)

ParseID returns the ID that corresponds to the string.

func (ID) IsZero

func (id ID) IsZero() bool

IsZero returns true if the ID is the zero object.

func (ID) String

func (id ID) String() string

String returns a human readable representation of the ID.

type IndexStorage

type IndexStorage interface {
	// Index will concurrently return all of the IDs currently in the storage.
	// The completeness or accuracy of the list is undefined when mixed with
	// concurrent access to the storage.
	Index(ctx context.Context) (<-chan ID, error)
}

IndexStorage is an extension interface for a content-addressable storage (CAS) that can list its contents.

type RemoveStorage

type RemoveStorage interface {
	Storage

	// Remove deletes the content associated with the id.
	Remove(id ID) error
}

RemoveStorage is an extension interface for a content-addressable storage (CAS) that can remove items.

type Storage

type Storage interface {
	// Close releases any resources linked to the storage.  Further calls to
	// any methods are an error.
	Close() error

	// Create copies data from the reader into the storage, and returns an ID
	// that can be used to identify the data.
	Create(src io.Reader) (ID, error)

	// Open locates the content associated with the ID, and returns a reader
	// that can be used to access the data.  If the object does not exist in
	// storage, then the error will be ErrNotExist.
	Open(id ID) (io.ReadCloser, error)
}

Storage represents an open content-addressable storage (CAS). Adding items to the storage should be idempotent. For example, inserting the same content twice should return the same id.

The methods Create and Open should support concurrent access.

Example

package main

import (
	"fmt"
	"io/ioutil"

	"gitlab.com/stone.code/cas"
	"gitlab.com/stone.code/cas/memory"
)

func main() {
	// Initialize an in-memory store.
	storage := memory.New()
	defer storage.Close()

	// Insert the data into the storage.
	id, err := cas.CreateFromString(storage, "Hello, world!")
	if err != nil {
		fmt.Println("error: ", err)
		return
	}

	// Find the storage based on the ID.
	r, err := storage.Open(id)
	if err != nil {
		fmt.Println("error: ", err)
		return
	}
	defer r.Close()

	data, err := ioutil.ReadAll(r)
	if err != nil {
		fmt.Println("error: ", err)
		return
	}
	fmt.Println(string(data))

}

Output:

Hello, world!