s3sync

README

s3sync

Golang utility for syncing between s3 and local

Usage

Use New to create a manager, and Sync function syncs between s3 and local filesystem.

import (
  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/aws/session"
  "github.com/seqsense/s3sync"
)

func main() {
  // Creates an AWS session
  sess, _ := session.NewSession(&aws.Config{
    Region: aws.String("us-east-1"),
  })

  syncManager := s3sync.New(sess)

  // Sync from s3 to local
  syncManager.Sync("s3://yourbucket/path/to/dir", "local/path/to/dir")

  // Sync from local to s3
  syncManager.Sync("local/path/to/dir", "s3://yourbucket/path/to/dir")
}

• Note: Sync from s3 to s3 is not implemented yet.

Sets the custom logger

You can set your custom logger.

import "github.com/seqsense/s3sync"

...
s3sync.SetLogger(&CustomLogger{})
...

The logger needs to implement Log and Logf methods. See the godoc for details.

Sets up the parallelism

You can configure the number of parallel jobs for sync. Default is 16.

s3sync.new(sess, s3sync.WithParallel(16)) // This is the same as default.
s3sync.new(sess, s3sync.WithParallel(1)) // You can sync one by one.

License

Apache 2.0 License. See LICENSE.

Documentation

Overview

Copyright 2019 SEQSENSE, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright 2019 SEQSENSE, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

• Constants
• func SetLogger(l LoggerIF)
• type LoggerIF
• type Manager
  • func New(sess *session.Session, options ...Option) *Manager
  • func (m *Manager) GetStatistics() SyncStatistics
  • func (m *Manager) Sync(source, dest string) error
• type Option
  • func WithACL(acl string) Option
  • func WithContentType(mime string) Option
  • func WithDelete() Option
  • func WithDownloaderOptions(opts ...func(*s3manager.Downloader)) Option
  • func WithDryRun() Option
  • func WithParallel(n int) Option
  • func WithUploaderOptions(opts ...func(*s3manager.Uploader)) Option
  • func WithoutGuessMimeType() Option
• type SyncStatistics

Constants

const (
	// Default number of parallel file sync jobs.
	DefaultParallel = 16
)

Functions

func SetLogger

func SetLogger(l LoggerIF)

SetLogger sets the logger.

Types

type LoggerIF

type LoggerIF interface {
	// Log inserts a log entry. Arguments are handled in the manner
	// of fmt.Print.
	Log(v ...interface{})
	// Log inserts a log entry. Arguments are handled in the manner
	// of fmt.Printf.
	Logf(format string, v ...interface{})
}

LoggerIF is the logger interface which this library requires.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages the sync operation.

func New

func New(sess *session.Session, options ...Option) *Manager

New returns a new Manager.

func (*Manager) GetStatistics

func (m *Manager) GetStatistics() SyncStatistics

GetStatistics returns the structure that contains the sync statistics

func (*Manager) Sync

func (m *Manager) Sync(source, dest string) error

Sync syncs the files between s3 and local disks.

type Option

type Option func(*Manager)

Option is a functional option type of Manager.

func WithACL

func WithACL(acl string) Option

WithACL sets Access Control List string for uploading.

func WithContentType

func WithContentType(mime string) Option

WithContentType overwrites uploading MIME type.

func WithDelete

func WithDelete() Option

WithDelete enables to delete files unexisting on source directory.

func WithDownloaderOptions

func WithDownloaderOptions(opts ...func(*s3manager.Downloader)) Option

WithDownloaderOptions sets underlying s3manager's options.

func WithDryRun

func WithDryRun() Option

WithDryRun enables dry-run mode.

func WithParallel

func WithParallel(n int) Option

WithParallel sets maximum number of parallel file sync jobs.

func WithUploaderOptions

func WithUploaderOptions(opts ...func(*s3manager.Uploader)) Option

WithUploaderOptions sets underlying s3manager's options.

func WithoutGuessMimeType

func WithoutGuessMimeType() Option

WithoutGuessMimeType disables guessing MIME type from contents.

type SyncStatistics

type SyncStatistics struct {
	Bytes        int64
	Files        int64
	DeletedFiles int64
	// contains filtered or unexported fields
}

SyncStatistics captures the sync statistics.