qscamel

command module
v2.0.0-beta.3+incompat... Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 14, 2018 License: Apache-2.0 Imports: 4 Imported by: 0

README

qscamel

Build Status Go Report Card License

qscamel is a command line tool to migrate data between different endpoint efficiently.

Features

  • Easy task management

  • Resume from where task stopped

  • Automatic retry mechanism

  • Concurrent migrating with goroutine pool

  • Support both copy and fetch migrating methods

  • Mutiple endpoint support

Quick start

Create a task file like following and save as example-task.yaml:

name: example-task
type: copy

source:
  type: fs
  path: /path/to/source

destination:
  type: qingstor
  path: /path/to/destination
  options:
    bucket_name: example_bucket
    access_key_id: example_access_key_id
    secret_access_key: example_secret_access_key

Use qscamel:

qscamel run example-task.yaml

Have a cup of tea and you will see all file under /path/to/source will be migrated to qingstor bucket example_bucket with prefix /path/to/destination.

See the detailed usage with qscamel -h or qscamel --help, and read the following docs.

Installation

Get the latest qscamel for Linux, macOS and Windows from releases

Configuration

qscamel has following config options:

# concurrency controls haw many goroutine run at the same time.
# if not set or set to 0, qscamel will use the numer of logic CPU * 100.
concurrency: 0
# log_level controls the log_level.
# Available value (from more to less): debug, info, warn, error, fatal, panic.
# Default value: info
log_level: info
# pid_file controls where the pid file will create.
# Default value: ~/.qscamel/qscamel.pid
pid_file: ~/.qscamel/qscamel.pid
# log_file controls where the log file will create.
# Default value: ~/.qscamel/qscamel.log
log_file: ~/.qscamel/qscamel.log
# database_file controls where the database file will create.
# Default value: ~/.qscamel/qscamel.db
database_file: ~/.qscamel/qscamel.db

The default config will read from ~/.qscamel/qscamel.yaml, you can also specify the config path with flag -c or --config.

For example:

qscamel run example-task -c /path/to/config/file

Task

Task file will define a task, and the task has following options:

# name is the unique identifier for a task, qscamel will use it to distingush
# different tasks.
name: example-task
# type is the type for current task.
# Available value: copy, fetch, verify.
type: copy

# source is the source endpoint for current task.
source:
  # type is the type for endpoint.
  # Available value: aliyun, fs, gcs, qingstor, qiniu, s3, upyun.
  type: fs
  # path is the path for endpoint.
  path: "/path/to/source"

# destination is the destination endpoint for current task.
destination:
  # type is the type for endpoint.
  # Available value: fs, qingstor.
  type: qingstor
  # path is the path for endpoint.
  path: /aaa
  # options is the options for differenty endpoint.
  options:
    bucket_name: example_bucket
    access_key_id: example_access_key_id
    secret_access_key: example_secret_access_key
Endpoint aliyun

Can be used as source endpoint.

Aliyun is the object storage service provided by Alibaba.

aliyun endpoint has following options:

endpoint: example_endpoint
bucket_name: example_bucket
access_key_id: example_access_key_id
access_key_secret: example_access_key_secret
Endpoint fs

Can be used as source and destination endpoint.

There is no more config for fs endpoint.

Endpoint gcs

Can be used as source endpoint.

GCS(Google Cloud Storage) is the object storage service provided by Google.

gcs endpoint has following options:

api_key: example_api_key
bucket_name: exmaple_bukcet
Endpoint qingstor

Can be used as source and destination endpoint.

qingstor endpoint has following options:

# protocol controls protocol for qingstor.
# Available value: https, http
# Default value: https
protocol: https
# host controls host for qingstor.
# Default value: qingstor.com
host: qingstor.com
# port controls port for qingstor.
# Default value: 443
port: 443
# bucket_name is the bucket name for qingstor.
bucket_name: example_bucket
# access_key_id is the access_key_id for qingstor.
access_key_id: example_access_key_id
# secret_access_key is the secret_access_key for qingstor.
secret_access_key: example_secret_access_key
Endpoint qiniu

Can be used as source endpoint.

Qiniu is the object storage service provided by Qiniu.

qiniu endpoint has following options:

bucket_name: example_bucket
access_key: example_access_key
secret_key: example_secret_key
domain: example_domain
use_https: false
use_cdn_domains: false
Endpoint s3

Can be used as source endpoint.

S3 is the object storage service provided by AWS.

s3 endpoint has following options.

bucket_name: example_bucket
endpoint: example_endpoint
region: example_region
access_key_id: example_access_key_id
secret_access_key: example_secret_access_key
disable_ssl: false
use_accelerate: false
Endpoint upyun

Can be used as source endpoint.

upyun is the object storage service provided by Upyun.

upyun endpoint has following options.

bucket_name: example_bucket
operator: example_operator
password: example_password

Usage

Run

Run is the main command for qscamel. We can use this command to create or resume a task.

In order to create a new task, we can use:

qscamel run /path/to/task/file

In order to resume a task, we can use:

qscamel run /path/to/task/file

or

qscamel run task-name

When a new task created, we will calculate the sha256 checksum for it's content and save it to the database, and we will check if the content of the task file has been changed, if changed, qscamel will return an error. In other word, task can't be changed after created. If your need to update the task, please create a new one.

Delete

Delete can delete a task.

qscamel delete task-name
Status

Status will show the task status.

qscamel status
Clean

Clean will delete all the finished tasks.

qscamel clean
Version

Version will show the current version of qscamel.

qscamel version

Contributing

Please see Contributing Guidelines of this project before submitting patches.

LICENSE

The Apache License (Version 2.0, January 2004).

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis
fs
gcs
s3

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL