teos3

README

TeoS3 package and s3cp utilite

The TeoS3 package contains Golang features that make it easy to use S3 storage as a key-value database.

This project contain also the s3cp utilite which copy files from disk to s3 storage and back

---

Using s3cp utilite

Install

To install s3cp application use next command:

go install github.com/teonet-go/teos3/cmd/s3cp

Description

The s3cp application copy file to/from S3 storage.

The S3 storage credentials may be set in application parameters or in environment variables:

TEOS3_ACCESSKEY -- S3 storage Access key
TEOS3_SECRETKEY -- S3 storage Secret key
TEOS3_ENDPOINT  -- S3 storage Endpoint
TEOS3_BUCKET    -- S3 storage Bucket

Parameters and arguments:

s3cp [OPTION] source target
use s3:/folder_and_object_name to define S3 in source or target

Usage of /tmp/go-build1982013444/b001/exe/s3cp:

-accesskey string
  S3 storage Access key
-bucket string
  S3 storage Bucket
-endpoint string
  S3 storage Endpoint
-secretkey string
  S3 storage Secret key
-secure
  set secure=false to enable insecure (HTTP) access (default true)

Logs

The s3cp application sends logs to syslog. To read current log messages in archlinux use journalctl -f command.

---

TeoS3 package description

The teos3 package contains Golang functions to rasy use S3 storage as KeyValue Database. There is functions to Set, Get, GetList using string key with any data value.

You can find complete packets documentation at: https://pkg.go.dev/github.com/teonet-go/teos3

The keyval example

There is basic keyval example which used most of packets functions.

This example executes following tasks:

• sets some numer of data records (objects) by key (objects name) to the key/value db baset on s3 bucket;

• gets list of saved data records by prefix;

• gets some numer of data records (objects) by key (objects name) to the key/value db baset on s3 bucket;

• deletes all saved data records (objects) by key (object name) from the key/value database in the s3 bucket.

All this tasks are performed in parallel mode.

Fill next environment variables to run this example:

export TEOS3_ACCESSKEY=YOUR_ACCESSKEY TEOS3_SECRETKEY=YOUR_SECRETKEY TEOS3_ENDPOINT=YOUR_GATEWAT TEOS3_BUCKET=YOUR_BUCKET

Use next command to run this example:

go run ./examples/keyval/

There is a part of code with connect, set and get key value:

// Connect to S3 storage
con, err := teos3.Connect(accessKey, secretKey, endpoint, secure)
if err != nil {
    log.Fatalln(err)
}

// Set key to teos3 Map
err = con.Map.Set(key, data)
if err != nil {
    log.Fatalln(err)
}

// Get key from TeoS3 Map
data, err := con.Map.Get(key)
if err != nil {
    log.Fatalln(err)
}

---

Licence

BSD

Documentation

Overview

The TeoS3 package contains Golang functions that make it easy to use S3 storage as a key-value database. This package uses [minio-go](https://github.com/minio/minio-go) S3 client.

Index

• Constants
• Variables
• func Copy(accessKey, secretKey, endpoint, bucket string, args []string, secures ...bool) (err error)
• type CopyOptions
• type DelObjectOptions
• type DelOptions
• type GetInfoOptions
• type GetObjectOptions
• type GetOptions
• type ListObjectsOptions
• type ListOptions
  • func (l *ListOptions) SetMaxKeys(maxKeys int) *ListOptions
  • func (l *ListOptions) SetStartAfter(startAfter string) *ListOptions
• type MapData
• type SetObjectOptions
• type SetOptions
• type StatObjectOptions
• type TeoS3
  • func Connect(accessKey, secretKey, endpoint string, secure bool, buckets ...string) (teos3 *TeoS3, err error)
  • func (m *TeoS3) Copy(source, destination string, options ...*CopyOptions) (err error)
  • func (m *TeoS3) Del(key string, options ...*DelOptions) (err error)
  • func (m *TeoS3) Get(key string, options ...*GetOptions) (data []byte, err error)
  • func (m *TeoS3) GetInfo(key string, options ...*GetInfoOptions) (minio.ObjectInfo, error)
  • func (m *TeoS3) GetObject(key string, options ...*GetOptions) (*minio.Object, error)
  • func (m *TeoS3) List(prefix string, options ...*ListOptions) (keys chan string)
  • func (m *TeoS3) ListAr(prefix string, options ...*ListOptions) (list []string)
  • func (m *TeoS3) ListBody(prefix string, options ...*ListOptions) (mapDataChan chan MapData)
  • func (m *TeoS3) ListBodyAr(prefix string, options ...*ListOptions) (listBody []MapData)
  • func (m *TeoS3) ListLen(prefix string, options ...*ListOptions) int
  • func (m *TeoS3) Move(source, destination string, options ...*CopyOptions) (err error)
  • func (m *TeoS3) NewDelOptions() *DelOptions
  • func (m *TeoS3) NewGetOptions() *GetOptions
  • func (m *TeoS3) NewGetStatOptions() *GetInfoOptions
  • func (m *TeoS3) NewListOptions() *ListOptions
  • func (m *TeoS3) NewSetOptions() *SetOptions
  • func (m *TeoS3) Set(key string, data []byte, options ...*SetOptions) error
  • func (m *TeoS3) SetContext(ctx context.Context) *TeoS3
  • func (m *TeoS3) SetObject(key string, reader io.Reader, objectSize int64, options ...*SetOptions) (err error)

Constants

const Version = "0.1.2"

Variables

var ErrDestinationObjectAlreadyExists = errors.New(
	"destination object already exists",
)

Functions

func Copy

func Copy(accessKey, secretKey, endpoint, bucket string, args []string,
	secures ...bool) (err error)

Copy copys s3 object from source to target. Source or Target may be s3 storage object. Use 's3:' prefix to define s3 object.

Types

type CopyOptions

type CopyOptions struct {
	context.Context
}

CopyOptions contains context.Context for Copy or Move requests.

type DelObjectOptions

type DelObjectOptions minio.RemoveObjectOptions

type DelOptions

type DelOptions struct {
	context.Context
	DelObjectOptions
}

DelOptions contains context.Context and options for Remove requests.

type GetInfoOptions

type GetInfoOptions struct {
	context.Context
	StatObjectOptions
}

GetInfoOptions contains context.Context and options are used to specify additional headers or options during GET Object Satat requests.

type GetObjectOptions

type GetObjectOptions minio.GetObjectOptions

type GetOptions

type GetOptions struct {
	context.Context
	GetObjectOptions
}

GetOptions contains context.Context and options are used to specify additional headers or options during GET requests.

type ListObjectsOptions

type ListObjectsOptions minio.ListObjectsOptions

type ListOptions

type ListOptions struct {
	context.Context
	ListObjectsOptions
}

ListOptions contains context.Context and options for List requests.

func (*ListOptions) SetMaxKeys

func (l *ListOptions) SetMaxKeys(maxKeys int) *ListOptions

SetMaxKeys sets MaxKeys list options value

func (*ListOptions) SetStartAfter

func (l *ListOptions) SetStartAfter(startAfter string) *ListOptions

SetStartAfter sets StartAfter list options value

type MapData

type MapData struct {
	Key   string `json:"key"`
	Value []byte `json:"value"`
}

MapData is data structure used in ListBody output

type SetObjectOptions

type SetObjectOptions minio.PutObjectOptions

type SetOptions

type SetOptions struct {
	context.Context
	SetObjectOptions
}

SetOptions contains context.Context and options specified by user for Set requests

type StatObjectOptions

type StatObjectOptions minio.StatObjectOptions

type TeoS3

type TeoS3 struct {
	// contains filtered or unexported fields
}

TeoS3 objects data and methods receiver

func Connect

func Connect(accessKey, secretKey, endpoint string, secure bool,
	buckets ...string) (teos3 *TeoS3, err error)

Connect creates new cinnwction to S3 storage using accessKey, secretKey, endpoint, secure flag and bucket (if omitted then default 'teos3' buckets name used). The enpoind argument must be specified without http/https prefix(just domain and path), and the secure argument defines HTTPS if true or HTTP if false.

func (*TeoS3) Copy

func (m *TeoS3) Copy(source, destination string, options ...*CopyOptions) (
	err error)

Copy copys source object to destination object

func (*TeoS3) Del

func (m *TeoS3) Del(key string, options ...*DelOptions) (err error)

Del remove key from map by key. The options parameter may be omitted and than default DelObjectOptions with context.Background and empty minio.RemoveObjectOptions used.

func (*TeoS3) Get

func (m *TeoS3) Get(key string, options ...*GetOptions) (
	data []byte, err error)

Get map data by key. The options parameter may be omitted and than default GetObjectOptions with context.Background and empty minio.SetObjectOptions used.

func (*TeoS3) GetInfo

func (m *TeoS3) GetInfo(key string, options ...*GetInfoOptions) (
	minio.ObjectInfo, error)

GetInfo fetchs metadata of an object by key.

func (*TeoS3) GetObject

func (m *TeoS3) GetObject(key string, options ...*GetOptions) (
	*minio.Object, error)

GetObject gets map object by key. The options parameter may be omitted and than default GetObjectOptions with context.Background and empty minio. SetObjectOptions used. Returned object must be cloused with obj.Close() after use.

func (*TeoS3) List

func (m *TeoS3) List(prefix string, options ...*ListOptions) (keys chan string)

List gets list of map keys by prefix. The options parameter may be omitted and than default ListObjectsOptions with context.Background and empty minio.ListObjectsOptions used. The Prefix parameter of the ListObjectsOptions will be always overwritten with the prefix functions argument (so it may be empty).

func (*TeoS3) ListAr

func (m *TeoS3) ListAr(prefix string, options ...*ListOptions) (
	list []string)

ListAr gets string array of map keys by prefix.

func (*TeoS3) ListBody

func (m *TeoS3) ListBody(prefix string, options ...*ListOptions) (
	mapDataChan chan MapData)

ListBody gets all keys and values in MapData struct by prefix asynchronously.

func (*TeoS3) ListBodyAr

func (m *TeoS3) ListBodyAr(prefix string, options ...*ListOptions) (
	listBody []MapData)

ListBodyAr gets MapData array with all keys and values by prefix.

func (*TeoS3) ListLen

func (m *TeoS3) ListLen(prefix string, options ...*ListOptions) int

ListLen returns the number of records in the list by prefix and options.

func (*TeoS3) Move

func (m *TeoS3) Move(source, destination string, options ...*CopyOptions) (
	err error)

Move movess source object to destination object

func (*TeoS3) NewDelOptions

func (m *TeoS3) NewDelOptions() *DelOptions

NewDelOptions creates a new DelOptions object

func (*TeoS3) NewGetOptions

func (m *TeoS3) NewGetOptions() *GetOptions

NewGetOptions creates a new GetOptions object

func (*TeoS3) NewGetStatOptions

func (m *TeoS3) NewGetStatOptions() *GetInfoOptions

NewGetStatOptions creates a new GetStatOptions object

func (*TeoS3) NewListOptions

func (m *TeoS3) NewListOptions() *ListOptions

NewListOptions creates a new ListOptions object

func (*TeoS3) NewSetOptions

func (m *TeoS3) NewSetOptions() *SetOptions

NewSetOptions creates a new GetOptions object

func (*TeoS3) Set

func (m *TeoS3) Set(key string, data []byte, options ...*SetOptions) error

Set sets data to map by key. The options parameter may be omitted and than default SetObjectOptions with context.Background and empty minio.PutObjectOptions used.

func (*TeoS3) SetContext

func (m *TeoS3) SetContext(ctx context.Context) *TeoS3

SetContext sets context which will be used in all TeS3 operations if another context does not send in function call options argument.

func (*TeoS3) SetObject

func (m *TeoS3) SetObject(key string, reader io.Reader, objectSize int64,
	options ...*SetOptions) (err error)

SetObject sets object to map by key. The options parameter may be omitted and than default SetObjectOptions with context.Background and empty minio.PutObjectOptions used.