qscamel
qscamel is a command line tool to migrate data between different endpoint efficiently.
Features
-
Easy task management
-
Resume from where task stopped
-
Automatic retry mechanism
-
Concurrent migrating with goroutine pool
-
Support both copy and fetch migrating methods
-
Supoort data verify
-
Mutiple endpoint support
Quick start
Create a task file like following and save as example-task.yaml
:
name: example-task
type: copy
source:
type: fs
path: /path/to/source
destination:
type: qingstor
path: /path/to/destination
options:
bucket_name: example_bucket
access_key_id: example_access_key_id
secret_access_key: example_secret_access_key
Use qscamel:
qscamel run example-task.yaml
Have a cup of tea and you will see all file under /path/to/source
will be migrated to qingstor bucket example_bucket
with prefix /path/to/destination
.
See the detailed usage with qscamel -h
or qscamel --help
, and read the following docs.
Installation
Get the latest qscamel for Linux, macOS and Windows from releases
Configuration
qscamel has following config options:
# concurrency controls haw many goroutine run at the same time.
# if not set or set to 0, qscamel will use the numer of logic CPU * 100.
concurrency: 0
# log_level controls the log_level.
# Available value (from more to less): debug, info, warn, error, fatal, panic.
# Default value: info
log_level: info
# pid_file controls where the pid file will create.
# Default value: ~/.qscamel/qscamel.pid
pid_file: ~/.qscamel/qscamel.pid
# log_file controls where the log file will create.
# Default value: ~/.qscamel/qscamel.log
log_file: ~/.qscamel/qscamel.log
# database_file controls where the database folder will create.
# Default value: ~/.qscamel/db
database_file: ~/.qscamel/db
The default config will read from ~/.qscamel/qscamel.yaml
, you can also specify the config path with flag -c
or --config
.
For example:
qscamel run example-task -c /path/to/config/file
Task
Task file will define a task, and the task has following options:
# name is the unique identifier for a task, qscamel will use it to distingush
# different tasks.
name: example-task
# type is the type for current task.
# Available value: copy, fetch
type: copy
# source is the source endpoint for current task.
source:
# type is the type for endpoint.
# Available value: aliyun, fs, gcs, qingstor, qiniu, s3, upyun.
type: fs
# path is the path for endpoint.
path: "/path/to/source"
# destination is the destination endpoint for current task.
destination:
# type is the type for endpoint.
# Available value: fs, qingstor.
type: qingstor
# path is the path for endpoint.
path: /aaa
# options is the options for differenty endpoint.
options:
bucket_name: example_bucket
access_key_id: example_access_key_id
secret_access_key: example_secret_access_key
# ignore_existing controls whether ignore exist file.
# If set to true,qscamel will check whether ignore exist file.
# If does exist and size, md5 match, this file will skip.
# If not, qscamel will do the task as defined.
ignore_existing: false
Endpoint aliyun
Can be used as source endpoint.
Aliyun is the object storage service provided by Alibaba.
aliyun endpoint has following options:
endpoint: example_endpoint
bucket_name: example_bucket
access_key_id: example_access_key_id
access_key_secret: example_access_key_secret
Endpoint fs
Can be used as source and destination endpoint.
There is no more config for fs endpoint.
Endpoint gcs
Can be used as source endpoint.
GCS(Google Cloud Storage) is the object storage service provided by Google.
gcs endpoint has following options:
api_key: example_api_key
bucket_name: exmaple_bukcet
Endpoint qingstor
Can be used as source and destination endpoint.
qingstor endpoint has following options:
# protocol controls protocol for qingstor.
# Available value: https, http
# Default value: https
protocol: https
# host controls host for qingstor.
# Default value: qingstor.com
host: qingstor.com
# port controls port for qingstor.
# Default value: 443
port: 443
# zone controls zone for qingstor.
# This will auto detected, no need to set.
zone: pek3b
# bucket_name is the bucket name for qingstor.
bucket_name: example_bucket
# access_key_id is the access_key_id for qingstor.
access_key_id: example_access_key_id
# secret_access_key is the secret_access_key for qingstor.
secret_access_key: example_secret_access_key
# storage class is the storage class used for qingstor.
# Available value: STANDARD, STANDARD_IA
# Default value: STANDARD
storage_class: STANDARD
# multipart boundary size controls when qingstor will use multipart
# unit is Byte ,when file size is bigger then this value, qscamel
# will use multipart API.
# Available value: 1 ~ 5368709120
# Default value: 67108864
multipart_boundary_size: 67108864
Endpoint qiniu
Can be used as source endpoint.
Qiniu is the object storage service provided by Qiniu.
qiniu endpoint has following options:
bucket_name: example_bucket
access_key: example_access_key
secret_key: example_secret_key
domain: example_domain
use_https: false
use_cdn_domains: false
Endpoint s3
Can be used as source endpoint.
S3 is the object storage service provided by AWS.
s3 endpoint has following options.
bucket_name: example_bucket
endpoint: example_endpoint
region: example_region
access_key_id: example_access_key_id
secret_access_key: example_secret_access_key
disable_ssl: false
use_accelerate: false
Endpoint upyun
Can be used as source endpoint.
upyun is the object storage service provided by Upyun.
upyun endpoint has following options.
bucket_name: example_bucket
operator: example_operator
password: example_password
Usage
Run
Run is the main command for qscamel. We can use this command to create or resume a task.
In order to create a new task, we can use:
qscamel run /path/to/task/file
In order to resume a task, we can use:
qscamel run /path/to/task/file
or
qscamel run task-name
When a new task created, we will calculate the sha256 checksum for it's content and save it to the database, and we will check if the content of the task file has been changed, if changed, qscamel will return an error. In other word, task can't be changed after created. If your need to update the task, please create a new one.
Delete
Delete can delete a task.
qscamel delete task-name
Status
Status will show the task status.
qscamel status
Clean
Clean will delete all the finished tasks.
qscamel clean
Version
Version will show the current version of qscamel.
qscamel version
Contributing
Please see Contributing Guidelines
of this project before submitting patches.
LICENSE
The Apache License (Version 2.0, January 2004).