storage

README

storage

A unified storage layer for Golang.

Goal

• Production ready
• High performance
• Vendor lock free

Current Status

This lib is in heavy development, break changes could be introduced at any time. All public interface or functions expected to be stable at v1.0.0.

Installation

Install will go get

go get -u github.com/Xuanwo/storage

Import

import "github.com/Xuanwo/storage"

Quickstart

// Init a service.
srv, store, err := coreutils.Open("qingstor://hmac:test_access_key:test_secret_key@https:qingstor.com:443/test_bucket_name")
if err != nil {
    log.Fatalf("service init failed: %v", err)
}

// Use Storager API to maintain data.
ch := make(chan *types.Object, 1)
defer close(ch)

err := store.ListDir("prefix", pairs.WithFileFunc(func(*types.Object){
    ch <- o
}))
if err != nil {
    log.Printf("storager listdir failed: %v", err)
}

Services

Service	Description	Status
azblob	Azure Blob storage	alpha (-segments, -unittests)
cos	Tencent Cloud Object Storage	planned
fs	Local file system	stable (-segments)
gcs	Google Cloud Storage	alpha (-segments, -unittests)
kodo	qiniu kodo	planned
oss	Aliyun Object Storage	alpha (-segments, -unittests)
qingstor	QingStor Object Storage	stable
s3	Amazon S3	alpha (-segments, -unittests)
uss	UPYUN Storage Service	planned

azblob

azblob://hmac:<access_key>:<secret_key>/<bucket_name>/<prefix>

fs

fs:///path/to/dir

gcs

gcs://apikey:<api_key>/<bucket_name>/<prefix>?project=<project_id>

oss

oss://hmac:<access_key>:<secret_key>@<protocol>:<host>:<port>/<bucket_name>/<prefix>

qingstor

qingstor://hmac:<access_key>:<secret_key>@<protocol>:<host>:<port>/<bucket_name>/<prefix>

s3

s3://hmac:<access_key>:<secret_key>/<bucket_name>/<prefix>

Documentation

Overview

Package storage intend to provide a unified storage layer for Golang.

Goals

- Production ready: high test coverage, enterprise storage software adaptation, semantic versioning, well documented.

- High performance: more code generation, less runtime reflect.

- Vendor lock free: more generic abstraction, less internal details.

Details

There two public interfaces: Servicer and Storager. Storager is a fully functional storage client, and Servicer is a manager of Storager instances, which will be useful for services like object storage. For any service, Storager is required to implement and Servicer is optional.

Examples

The most common case to use a Storager service could be following:

1. Init a service.

    srv, store, err := coreutils.Open("qingstor://hmac:test_access_key:test_secret_key@https:qingstor.com:443/test_bucket_name")
	if err != nil {
		log.Fatalf("service init failed: %v", err)
	}

2. Use Storager API to maintain data.

ch := make(chan *types.Object, 1)
defer close(ch)

err := store.ListDir("prefix", pairs.WithFileFunc(func(*types.Object){
	ch <- o
}))
if err != nil {
	log.Printf("storager listdir failed: %v", err)
}

Notes

- Storage uses error wrapping added by go 1.13, go version before 1.13 could be behaved as unexpected.

Index

• type Copier
• type Mover
• type Reacher
• type Segmenter
• type Servicer
• type Statistician
• type Storager
• type StoragerFunc

Types

type Copier

type Copier interface {
	// Copy will copy an Object or multiple object in the service.
	//
	// Implementer:
	//   - MAY accept a recursive pairs to support copy recursively.
	Copy(src, dst string, pairs ...*types.Pair) (err error)
}

Copier is the interface for Copy.

type Mover

type Mover interface {
	// Move will move an object or multiple object in the service.
	//
	// Implementer:
	//   - MAY accept a recursive pairs to support move recursively.
	Move(src, dst string, pairs ...*types.Pair) (err error)
}

Mover is the interface for Move.

type Reacher

type Reacher interface {
	// Reach will provide a way, which can reach the object.
	//
	// Implementer:
	//   - SHOULD return a publicly reachable http url.
	Reach(path string, pairs ...*types.Pair) (url string, err error)
}

Reacher is the interface for Reach.

type Segmenter

type Segmenter interface {

	// ListSegments will list segments.
	//
	// Implementer:
	//   - If path == "/", services should return all segments.
	ListSegments(path string, pairs ...*types.Pair) (err error)
	// InitSegment will init a segment which could be a File after complete.
	//
	// Implementer:
	//   - MUST maintain whole segment operation runtime data, including part number and any
	//     other similar things.
	// Caller:
	//   - SHOULD call InitSegment before Write, Complete or Abort.
	InitSegment(path string, pairs ...*types.Pair) (id string, err error)
	// WriteSegment will read data into segment.
	//
	// Implementer:
	//   - SHOULD return error while caller call WriteSegment without init.
	// Caller:
	//   - SHOULD call InitSegment before WriteSegment.
	WriteSegment(id string, offset, size int64, r io.Reader, pairs ...*types.Pair) (err error)
	// CompleteSegment will complete a segment and merge them into a File.
	//
	// Implementer:
	//   - SHOULD return error while caller call CompleteSegment without init.
	// Caller:
	//   - SHOULD call InitSegment before CompleteSegment.
	CompleteSegment(id string, pairs ...*types.Pair) (err error)
	// AbortSegment will abort a segment.
	//
	// Implementer:
	//   - SHOULD return error while caller call AbortSegment without init.
	// Caller:
	//   - SHOULD call InitSegment before AbortSegment.
	AbortSegment(id string, pairs ...*types.Pair) (err error)
}

Segmenter is the interface for Segment.

type Servicer

type Servicer interface {
	// String will implement Stringer.
	String() string

	// List will list all storager instances under this service.
	List(pairs ...*types.Pair) (err error)
	// Get will get a valid storager instance for service.
	Get(name string, pairs ...*types.Pair) (Storager, error)
	// Create will create a new storager instance.
	Create(name string, pairs ...*types.Pair) (Storager, error)
	// Delete will delete a storager instance.
	Delete(name string, pairs ...*types.Pair) (err error)
}

Servicer can maintain multipart storage services.

Implementer can choose to implement this interface or not.

type Statistician

type Statistician interface {
	// Statistical will count service's statistics, such as Size, Count.
	//
	// Implementer:
	//   - Statistical SHOULD only return dynamic data like Size, Count.
	// Caller:
	//   - Statistical call COULD be expensive.
	Statistical() (metadata.Metadata, error)
}

Statistician is the interface for Statistical.

type Storager

type Storager interface {
	// String will implement Stringer.
	String() string

	// Init will init storager itself.
	//
	// Caller:
	//   - Init MUST be called after created.
	Init(pairs ...*types.Pair) (err error)

	// Metadata will return current storager's metadata.
	//
	// Implementer:
	//   - Metadata SHOULD only return static data without API call or with a cache.
	// Caller:
	//   - Metadata SHOULD be cheap.
	Metadata() (m metadata.Storage, err error)

	// ListDir will return an ObjectIterator which can list all object under the Dir.
	ListDir(path string, pairs ...*types.Pair) (err error)
	// Read will read the file's data.
	//
	// Caller:
	//   - MUST close reader while error happened or all data read.
	Read(path string, pairs ...*types.Pair) (r io.ReadCloser, err error)
	// WriteFile will write data into file.
	//
	// Caller:
	//   - MUST close reader while error happened or all data written.
	Write(path string, r io.Reader, pairs ...*types.Pair) (err error)
	// Stat will stat a path to get info of an object.
	//
	// Implementer:
	//   - MUST fill object's name and type.
	Stat(path string, pairs ...*types.Pair) (o *types.Object, err error)
	// Delete will delete an Object or multiple object from service.
	// path could be File, Stream or Dir
	//
	// Implementer:
	//   - MAY accept a recursive pair to support delete Dir recursively.
	Delete(path string, pairs ...*types.Pair) (err error)
}

Storager is the interface for storage service.

Currently, we support two different type storage services: prefix based and directory based. Prefix based storage service is usually a object storage service, such as AWS; And directory based service is often a POSIX file system. We used to treat them as different abstract level services, but in this project, we will unify both of them to make a production ready high performance vendor lock free storage layer.

Every storager will implement the same interface but with different capability and operation pairs set.

Everything in a storager is an Object with three types: File, Dir. File is the smallest unit in service, it will have content and metadata. Dir is a container for File and Dir. In prefix-based storage service, Dir is usually an empty key end with "/" or with special content type. For directory-based service, Dir will be corresponded to the real directory on file system.

Every API call in storager is relative to it's workdir which set in Init().

In the comments of every method, we will use following rules to standardize the Storager's behavior:

• The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.
• Implementer is the provider of the service, while you trying to implement Storager interface, you need to follow.
• Caller is the user of the service, while you trying to use the Storager interface, you need to follow.

type StoragerFunc

type StoragerFunc func(Storager)

StoragerFunc will handle a storager.