xy3

README

xy3

xy3 is born out of my need to create S3 backups while using XYplorer. Here are the XYplorer's file associations that I use:

|"Stream and extract from S3" s3>"xy3.exe" "download" --stream-and-extract
|"Download from S3" s3>"xy3.exe" "download"
|"Delete from S3" s3>"xy3.exe" "remove"
|"Compress and upload to S3" \>"xy3.exe" "upload" -b "bucket-name" -k "<curfolder>/"
|"Upload to S3" *>"xy3.exe" "upload" -b "bucket-name" -k "<curfolder>/"

Setup

go get github.com/nguyengg/xy3

CLI

xy3 exists as a CLI that I use with XYplorer on a daily basis.

# Uploading a file will generate a local .s3 (JSON) file that stores metadata about how to retrieve the file.
# For example, this command will create doc.txt.s3 and log.zip.s3.
xy3 up -b "bucket-name" -k "key-prefix/" --expected-bucket-owner "1234" doc.txt log.zip

# Downloading from the JSON .s3 files will create unique names to prevent duplicates.
# For example, since doc.txt and log.zip still exist, this command will create doc-1.txt and log-1.zip.
xy3 down doc.txt.s3 log.zip.s3

# To remove both local and remote files, use this command.
xy3 remove doc.txt.s3 log.zip.s3

Go module

This module implements a lot of useful libraries for dealing with S3 io operations.

Implements io.ReadSeeker, io.ReaderAt, and io.WriterTo using S3 ranged GetObject

See s3reader.

Implements io.Writer and io.ReaderFrom to upload to S3

See s3writer.

Zip compress and extract

You can use github.com/nguyengg/xy3/zipper directly to ZIP-compress directories and extract them. See zipper for more information.

Documentation

Index

• func CopyBufferWithContext(ctx context.Context, dst io.Writer, src io.Reader, buf []byte) (written int64, err error)
• func MkExclDir(parent, stem string) (name string, err error)
• func OpenExclFile(parent, stem, ext string) (file *os.File, err error)
• func StemAndExt(path string) (stem, ext string)

Functions

func CopyBufferWithContext

func CopyBufferWithContext(ctx context.Context, dst io.Writer, src io.Reader, buf []byte) (written int64, err error)

CopyBufferWithContext is a custom implementation of io.CopyBuffer that is cancellable via context.

Similar to io.CopyBuffer, if buf is nil, a new buffer of size 32*1024 is created. Unlike io.CopyBuffer, it does not matter if src implements io.WriterTo or dst implements io.ReaderFrom because those interfaces do not support context.

The context is checked for done status after every write. As a result, having too small a buffer may introduce too much overhead, while having a very large buffer may cause context cancellation to have a delayed effect.

func MkExclDir

func MkExclDir(parent, stem string) (name string, err error)

MkExclDir creates a new child directory that did not exist prior to this invocation.

Stem is the desired name of the directory. The actual directory that is created might have numeric suffixes such as stem-1, stem-2, etc. The return value "name" is the actual path to the newly created directory.

The directory is created with perm `0755`.

func OpenExclFile

func OpenExclFile(parent, stem, ext string) (file *os.File, err error)

OpenExclFile creates a new file for writing with the condition that the file did not exist prior to this call.

The first argument is the parent directory of the file to be created. The second argument is the stem of the file, the third the extension. For example, the stem of "hello-world.txt" is "hello-world", its ext ".txt". But with "hello-world.txt.s3", filepath.Ext will think ".s3" is the ext while this method allows you to choose ".txt.s3" a s extension instead. If you use ".txt.s3" as extension, the naming is more natural: it will be "hello-world-1.txt.s3" or "hello-world-2.txt.s3" instead of "hello-world.txt-1.s3". See StemAndExt for a variant of filepath.Ext that allows up to 4 characters to be counted as ext.

The file is opened with flag `os.O_RDWR|os.O_CREATE|os.O_EXCL` and permission `0666`. Caller is responsible for closing the file upon a successful return. See MkExclDir for a dir equivalent.

func StemAndExt

func StemAndExt(path string) (stem, ext string)

StemAndExt is a variant of filepath.Ext that allows extended extension to be detected while also returning the stem.

For example, `filepath.Ext("file.tar.gz")` would return ".gz", but `xy3.StemAndExt("file.tar.gz")` would return ".tar.gz" for the extension, "file" for the stem. This is useful when passed to OpenExclFile: "file-1.tar.gz" is more natural than "file.tar-1.gz".

StemAndExt will only accept file extensions of 5 characters or less, so if there is no `.` in the last 6 characters, the returned ext will be empty string unlike filepath.Ext which will keep searching until the last path separator or `.` is found. That means longer extensions such as ".jfif-tbnl" or ".turbot" will not be found by StemAndExt but can be found by filepath.Ext.