s3packer

command module
v1.60.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 22, 2025 License: MIT Imports: 10 Imported by: 0

README

s3packer - A configurable profile-based S3 backup and upload tool.

CLI for Linux/MacOS supports Amazon S3 | Google Cloud Storage | Linode (Akamai) Object Storage | Oracle Cloud Object Storage


Go Version Go Report Card Repo License

Code Quality


Jetbrains_OSS Jetbrains_GoLand

Special thanks to JetBrains! s3packer is developed with help from JetBrains' Open Source program.


About

s3packer is an S3 / Object Storage upload and backup tool. It uses YAML-based configs that tell it what to upload, where to upload, how to name, and how to tag the objects. s3packer makes backup redundancy easier by using separate profiles for buckets and providers. Currently it supports AWS, OCI (Oracle Cloud), and Linode (Akamai).


Download

See the releases page...


Service Provider Support

s3packer supports AWS S3, Google Cloud Storage, Oracle Cloud Object Storage (OCI), and Linode (Akamai) Object Storage.

See the example profiles:


How to Use

To start a session with an existing profile, just type in the following command:

$ s3packer --profile="myprofile.yaml"

Creating a new Profile

s3packer can create a base profile to help get you started. To create one, use the --create flag:

$ s3packer --create="my-new-profile.yaml"

Setting up a Profile

s3packer profiles are written in YAML. To set one up, you just need to fill out a few fields before you can get started.

Version
Version: 6

Provider

Tell s3packer which service you're using

PROVIDER Acceptable Values Required Description
Use aws, google, linode, oci Y name of service provider you will be using
Provider:
  Use: aws

Each provider needs their own special fields to configure.
SEE: docs/general_config.md

Bucket

Tell s3packer where the bucket is and whether to create it

BUCKET Acceptable Values Default Required Description
Create boolean false F Whether s3packer should create the bucket if it is not found
Name any string Y The name of the bucket
Region any string Y The region that the bucket is or will be in, e.g. eu-north-1
Bucket:
  Create: true
  Name: "deep-freeze"
  Region: "eu-north-1"

Options

s3packer's configurable options

OPTIONS Acceptable Values Default Required Description
MaxUploads any integer 1 N The number of simultaneous uploads, at least 1.
FollowSymlinks boolean false N Whether to follow symlinks under dirs provided
WalkDirs boolean true N Whether s3packer will walk subdirectories of dirs provided
OverwriteObjects always, never never N Whether overwrite objects that already exist in the bucket
Options:
  MaxUploads: 1
  FollowSymlinks: false
  WalkDirs: true
  OverwriteObjects: "never"
MaxUploads (Experimental)

The suggested setting for MaxUploads is 1.

Be careful when you set MaxUploads as some services struggle with anything more than 1. The notable exception being AWS which seems fine with 50-100 -- though, whether its faster to have a high MaxUploads value depends on your upload job.

Large files can be broken up into many parts which are then simultaneously uploaded. s3packer uses default SDK values for part count, part size, and the large file threshold values, unless otherwise called out.

An example of this would be: If you specify a MaxUploads value of 5, and s3packer tries to upload 5 large files that are each split into 20 parts, then there would be 100 simultaneous uploads happening. If you specify a MaxUpload value of 50 and there are 50 large files each split into 20 parts, then you could potentially have as many as 1,000 simultaneous uploads.


Objects

s3packer's configurable options for object name and renaming

OBJECTS Acceptable Values Default Required Description
NamingType absolute, relative Y the method s3packer uses to name objects that it uploads
NamePrefix any string N The string that will be prefixed to the object's "file" name
PathPrefix any string N a string path that will be prefixed to the object's "file" name and "path" name
OmitRootDir boolean True N whether the relative root of a provided directory will be added to the objects path name
Objects:
  NamingType: absolute
  NamePrefix: backup-
  PathPrefix: /backups/april/2023
  OmitRootDir: true

NamingType
The default is relative.

  • relative: The key will be prepended with the relative path of the file on the local filesystem (individual files specified in the profile will always end up at the root of the bucket, plus the PathPrefix and then NamePrefix).
  • absolute: The key will be prepended with the absolute path of the file on the local filesystem.

NamePrefix
This is blank by default. Any value you put here will be added before the filename when it's uploaded to S3. Using something like weekly- will add that string to any file you're uploading, like weekly-log.log or weekly-2021-01-01.log.

PathPrefix
This is blank by default. Any value put here will be added before the file path when it's uploaded to S3. If you use something like /backups/monthly, the file will be uploaded to /backups/monthly/your-file.txt.


Files, Dirs

Tells s3packer what you want to upload. You can specify directories or individual files. When you specify a directory, s3packer will NOT traverse subdirectories, unless configured to. You must specify one or the other.

FILES Required Description
path Y the absolute path to the file that will be uploaded
DIRS Required Description
path Y the absolute path to the directory that will be uploaded
Files:
  - "/Users/forrest/docs/stocks/apple"
  - "/Users/jenny/docs/song_lyrics"
Dirs:
  - "/Users/forrest/docs/objJob-application-lawn-mower.pdf"
  - "/Users/forrest/docs/dr-pepper-recipe.txt"
  - "/Users/jenny/letters/from-forrest.docx"

Tags

Add tags to each uploaded object (if supported by the provider)

TAGS Acceptable Values Required Description
Key any value N key:value tag pair, will be converted to a string
Tags:
  Author: "Forrest Gump"
  Year: 1994

TagOptions

Options related to object tagging (dependent on whether the provider supports object tagging)

TAGOPTIONS Acceptable Values Default Required Description
OriginPath boolean False N Whether s3packer will tag the object with the original absolute path of the file
ChecksumSHA256 boolean False N Whether s3packer will tag the object with the sha256 checksum of the file as uploaded
Tagging:
  OriginPath: true
  ChecksumSHA256: false

Note on Checksum Tagging
Some providers have checksum validation on objects to verify that uploads are completed correctly. This checksum is calculated separately from that process and is only for your reference.


Logging

Options for logging output

LOGGING Acceptable Values Default Required Description
Level 1-5 2 N The severity level a log message must be to output to the console or file
Console boolean True N Whether logging message will be output to stdout.
File boolean False N Whether logging output will be written to a file. Output is structured in JSON format.
Logfile path "/var/log/s3p.log" N The name of the file that output logging will be appended to.
Logging:
  Level: 3
  Console: true
  File: true
  Logfile: "/var/log/backup.log"

Notes on Level
Level is set to 2 (WARN) by default. The setting is by severity, with 1 being least severe (INFO) and 5 being most severe (PANIC).


Things to Keep in Mind...

Individual Files

If you’re uploading individual files, just remember that the prefix will be added to the start of each filename and they’ll be uploaded right to the root of the bucket (unless you specify custom PathPrefix. Note that if you have multiple files with the same name (like if you have five ‘log.log’ files from different directories), they could be overwritten as you upload.

Directories

When you’re uploading directories, when WalkDirs is set to true, then all the subdirectories and files will be uploaded to the bucket as well. Processing directories with a large number of files can take some time as the checksums are calculated and each directory entry is read.


Issues & Suggestions

If you run into problems, errors, or have feature suggestions, it would be great if you took the time to open a new issue on GitHub.


Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL