storage

README

storage

An application-oriented unified storage layer for Golang.

Goal

• Production ready
• High performance
• Vendor agnostic

Features

Widely services support

• azblob: Azure Blob storage
• cos: Tencent Cloud Object Storage
• dropbox: Dropbox
• fs: Local file system
• gcs: Google Cloud Storage
• kodo: qiniu kodo
• oss: Aliyun Object Storage
• qingstor: QingStor Object Storage
• s3: Amazon S3
• uss: UPYUN Storage Service

Servicer operation support

• List: list all Storager in service
• Get: get a Storager via name
• Create: create a Storager
• Delete: delete a Storager

Storager operation support

Basic operations

• Metadata: get storager's metadata
• Read: read file content
• Write: write content into file
• Stat: get file's metadata
• Delete: delete a file or directory

Extended operations

• Copy: copy a file inside storager
• Move: move a file inside storager
• Reach: generate a public accessible url
• Statistical: get storage service's statistics

Multiple list style support

• ListDir: list files and directories under a directory
• ListPrefix: list files under a prefix

Segment/Multipart support

• ListPrefixSegment: list segments under a prefix
• InitIndexSegment: initiate an index type segment
• WriteIndexSegment: write content into an index type segment
• CompleteSegment: complete a segment to create a file
• AbortSegment: abort a segment

File metadata support

Required metadata

• id: unique key in service
• name: relative path towards service's work dir
• size: size of this object
• updated_at: last update time of this object

Optional metadata

• content-md5: md5 digest as defined in rfc2616
• content-type: media type as defined in rfc2616
• etag: entity tag as defined in rfc2616
• storage-class: object's storage class as defined in storage proposal

Quick Start

import (
    "log"

    "github.com/Xuanwo/storage"
    "github.com/Xuanwo/storage/coreutils"
    "github.com/Xuanwo/storage/types/pairs"
)

// Init a service.
store, err := coreutils.OpenStorager("fs", pairs.WithWorkDir("/tmp"))
if err != nil {
    log.Fatalf("service init failed: %v", err)
}

// Use Storager API to maintain data.
r, err := store.Read("path/to/file")
if err != nil {
    log.Printf("storager read: %v", err)
}

Documentation

Overview

Package storage intend to provide a unified storage layer for Golang.

Goals

- Production ready: high test coverage, enterprise storage software adaptation, semantic versioning, well documented.

- High performance: more code generation, less runtime reflect.

- Vendor agnostic: more generic abstraction, less internal details.

Details

There two main public interfaces: Servicer and Storager. Storager is a fully functional storage client, and Servicer is a manager of Storager instances, which will be useful for services like object storage. For any service, Storager is required to implement and Servicer is optional.

Examples

The most common case to use a Storager service could be following:

1. Init a service.

    store, err := coreutils.OpenStorager("fs", pairs.WithWorkDir("/tmp"))
	if err != nil {
		log.Fatalf("service init failed: %v", err)
	}

2. Use Storager API to maintain data.

r, err := store.Read("path/to/file")
if err != nil {
	log.Printf("storager read: %v", err)
}

Notes

- Storage uses error wrapping added by go 1.13, go version before 1.13 could be behaved as unexpected.

Index

• type Copier
• type DirLister
• type DirSegmentsLister
• type IndexSegmenter
• type Mover
• type PrefixLister
• type PrefixSegmentsLister
• type Reacher
• type Servicer
• type Statistician
• type Storager
• type StoragerFunc

Types

type Copier

type Copier interface {

	// Copy will copy an Object or multiple object in the service.
	Copy(src string, dst string, pairs ...*types.Pair) (err error)
	// CopyWithContext will copy an Object or multiple object in the service.
	CopyWithContext(ctx context.Context, src string, dst string, pairs ...*types.Pair) (err error)
}

Copier is the interface for Copy.

type DirLister

type DirLister interface {

	// ListDir will return list a specific dir.
	ListDir(dir string, pairs ...*types.Pair) (err error)
	// ListDirWithContext will return list a specific dir.
	ListDirWithContext(ctx context.Context, dir string, pairs ...*types.Pair) (err error)
}

DirLister is used for directory based storage service to list objects under a dir.

type DirSegmentsLister

type DirSegmentsLister interface {

	// ListDirSegments will list segments via dir.
	ListDirSegments(dir string, pairs ...*types.Pair) (err error)
	// ListDirSegmentsWithContext will list segments via dir.
	ListDirSegmentsWithContext(ctx context.Context, dir string, pairs ...*types.Pair) (err error)
	// contains filtered or unexported methods
}

DirSegmentsLister is used for directory based storage service to list segments under a dir.

type IndexSegmenter

type IndexSegmenter interface {

	// InitIndexSegment will init an index based segment.
	InitIndexSegment(path string, pairs ...*types.Pair) (seg segment.Segment, err error)
	// InitIndexSegmentWithContext will init an index based segment.
	InitIndexSegmentWithContext(ctx context.Context, path string, pairs ...*types.Pair) (seg segment.Segment, err error)

	// WriteIndexSegment will write a part into an index based segment.
	WriteIndexSegment(seg segment.Segment, r io.Reader, index int, size int64, pairs ...*types.Pair) (err error)
	// WriteIndexSegmentWithContext will write a part into an index based segment.
	WriteIndexSegmentWithContext(ctx context.Context, seg segment.Segment, r io.Reader, index int, size int64, pairs ...*types.Pair) (err error)
	// contains filtered or unexported methods
}

IndexSegmenter is the interface for index based segment.

type Mover

type Mover interface {

	// Move will move an object in the service.
	Move(src string, dst string, pairs ...*types.Pair) (err error)
	// MoveWithContext will move an object in the service.
	MoveWithContext(ctx context.Context, src string, dst string, pairs ...*types.Pair) (err error)
}

Mover is the interface for Move.

type PrefixLister

type PrefixLister interface {

	// ListPrefix will return list a specific dir.
	ListPrefix(prefix string, pairs ...*types.Pair) (err error)
	// ListPrefixWithContext will return list a specific dir.
	ListPrefixWithContext(ctx context.Context, prefix string, pairs ...*types.Pair) (err error)
}

PrefixLister is used for prefix based storage service to list objects under a prefix.

type PrefixSegmentsLister

type PrefixSegmentsLister interface {

	// ListPrefixSegments will list segments.
	ListPrefixSegments(prefix string, pairs ...*types.Pair) (err error)
	// ListPrefixSegmentsWithContext will list segments.
	ListPrefixSegmentsWithContext(ctx context.Context, prefix string, pairs ...*types.Pair) (err error)
	// contains filtered or unexported methods
}

PrefixSegmentsLister is used for prefix based storage service to list segments under a prefix.

type Reacher

type Reacher interface {

	// Reach will provide a way, which can reach the object.
	Reach(path string, pairs ...*types.Pair) (url string, err error)
	// ReachWithContext will provide a way, which can reach the object.
	ReachWithContext(ctx context.Context, path string, pairs ...*types.Pair) (url string, err error)
}

Reacher is the interface for Reach.

type Servicer

type Servicer interface {
	String() string

	// Create will create a new storager instance.
	Create(name string, pairs ...*types.Pair) (store Storager, err error)
	// CreateWithContext will create a new storager instance.
	CreateWithContext(ctx context.Context, name string, pairs ...*types.Pair) (store Storager, err error)

	// Delete will delete a storager instance.
	Delete(name string, pairs ...*types.Pair) (err error)
	// DeleteWithContext will delete a storager instance.
	DeleteWithContext(ctx context.Context, name string, pairs ...*types.Pair) (err error)

	// Get will get a valid storager instance for service.
	Get(name string, pairs ...*types.Pair) (store Storager, err error)
	// GetWithContext will get a valid storager instance for service.
	GetWithContext(ctx context.Context, name string, pairs ...*types.Pair) (store Storager, err error)

	// List will list all storager instances under this service.
	List(pairs ...*types.Pair) (err error)
	// ListWithContext will list all storager instances under this service.
	ListWithContext(ctx context.Context, pairs ...*types.Pair) (err error)
}

Servicer can maintain multipart storage services.

type Statistician

type Statistician interface {

	// Statistical will count service's statistics, such as Size, Count.
	Statistical(pairs ...*types.Pair) (statistic info.StorageStatistic, err error)
	// StatisticalWithContext will count service's statistics, such as Size, Count.
	StatisticalWithContext(ctx context.Context, pairs ...*types.Pair) (statistic info.StorageStatistic, err error)
}

Statistician is the interface for Statistical.

type Storager

type Storager interface {
	String() string

	// Delete will delete an Object from service.
	Delete(path string, pairs ...*types.Pair) (err error)
	// DeleteWithContext will delete an Object from service.
	DeleteWithContext(ctx context.Context, path string, pairs ...*types.Pair) (err error)

	// Metadata will return current storager's metadata.
	Metadata(pairs ...*types.Pair) (meta info.StorageMeta, err error)
	// MetadataWithContext will return current storager's metadata.
	MetadataWithContext(ctx context.Context, pairs ...*types.Pair) (meta info.StorageMeta, err error)

	// Read will read the file's data.
	Read(path string, pairs ...*types.Pair) (rc io.ReadCloser, err error)
	// ReadWithContext will read the file's data.
	ReadWithContext(ctx context.Context, path string, pairs ...*types.Pair) (rc io.ReadCloser, err error)

	// Stat will stat a path to get info of an object.
	Stat(path string, pairs ...*types.Pair) (o *types.Object, err error)
	// StatWithContext will stat a path to get info of an object.
	StatWithContext(ctx context.Context, path string, pairs ...*types.Pair) (o *types.Object, err error)

	// Write will write data into a file.
	Write(path string, r io.Reader, pairs ...*types.Pair) (err error)
	// WriteWithContext will write data into a file.
	WriteWithContext(ctx context.Context, path string, r io.Reader, pairs ...*types.Pair) (err error)
}

Storager is the interface for storage service.

type StoragerFunc

type StoragerFunc func(Storager)

StoragerFunc will handle a storager.