README
¶
grsync — golang rsync wrapper
Quick & Dirty fork of ranchers grsync for my own purposes.
Tested with rsync version 3.2.3 protocol version 31
Changes made:
- Add sshpass-Wrapper to allow passing ssh-password to rsync (sshsync needs to be installed duh, see example)
Archive-Flag is not set forcefully- Changed Progress to reflect current rsync progress format (not a great solution currently, some more parsing & better regex would be nice)
- Allow disabling the creation of directory if remote target is used ( see #7)
- Rsync: Added ListOnly-Param (see examples)
- Updated deps
Repository contains some helpful tools:
- raw rsync wrapper
- rsync task — wrapper which provide important information about rsync task: progress, remain items, total items and speed
Task wrapper usage
package main
import (
"fmt"
"github.com/ByteSizedMarius/grsync"
"time"
)
func main() {
task := grsync.NewTask(
"/local/source",
"remote@target::destination",
true, // use sshpass
false, // create directory
grsync.RsyncOptions{ // optimized for speed in my specific use-case, no guarantees
Inplace: true,
Partial: true,
HumanReadable: true,
NumericIDs: true,
Rsh: "ssh -T -c aes128-ctr -o Compression=no -x",
PasswordFile: "/local/password/file", // contains password that will be used by sshpass
RsyncBinaryPath: "/usr/bin/rsync",
},
)
go func() {
for {
state := task.State()
fmt.Printf(
"progress: %d %% / rem. %s / tot. %s / sp. %s \n",
state.Progress,
state.TimeRemaining,
state.DownloadedTotal,
state.Speed,
)
<-time.After(time.Second * 5)
}
}()
if err := task.Run(); err != nil {
panic(err)
}
fmt.Println("well done")
fmt.Println(task.Log())
}
Output:
progress: 0 % / rem. / tot. / sp.
progress: 0 % / rem. 0:22:27 / tot. 342.82M / sp. 108.93MB/s
progress: 0 % / rem. 0:22:16 / tot. 918.52M / sp. 109.41MB/s
progress: 0 % / rem. 0:22:18 / tot. 1.49G / sp. 108.85MB/s
progress: 1 % / rem. 0:22:16 / tot. 2.06G / sp. 108.58MB/s
progress: 1 % / rem. 0:22:16 / tot. 2.63G / sp. 108.19MB/s
progress: 2 % / rem. 0:22:10 / tot. 3.20G / sp. 108.21MB/s
File list:
package main
import (
"fmt"
"github.com/ByteSizedMarius/grsync"
)
func main() {
task := grsync.NewTask(
"remote@target::destination", // directory to list the files of
"",
true,
false,
grsync.RsyncOptions{
HumanReadable: true,
ListOnly: true,
Rsh: "ssh -T -c aes128-ctr -o Compression=no -x",
PasswordFile: "/root/pass",
RsyncBinaryPath: "/usr/bin/rsync",
},
)
if err := task.Run(); err != nil {
panic(err)
}
for _, file := range task.GetFileList() {
fmt.Printf("Permissions: %s, Size: %s, Date: %s, Time: %s, Name: %s\n", file[0], file[1], file[2], file[3], file[4])
}
fmt.Println("done")
}
Output:
Permissions: drwxrwxrwx, Size: 56, Date: 2023/10/07, Time: 13:19:08, Name: .
Permissions: -rwxrwxrwx, Size: 402.65G, Date: 2023/10/08, Time: 00:32:49, Name: File1
Permissions: -rwxrwxrwx, Size: 150.66G, Date: 2023/09/27, Time: 20:53:59, Name: File2
Permissions: drwxrwxrwx, Size: 68, Date: 2023/10/08, Time: 01:13:13, Name: Dir1
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Rsync ¶
type Rsync struct {
Source string
Destination string
CreateDir bool
// contains filtered or unexported fields
}
Rsync is wrapper under rsync
func NewRsync ¶
func NewRsync(source, destination string, useSshPass, createDir bool, options RsyncOptions) (*Rsync, error)
NewRsync returns task with described options If useSSHpass is true, then the password will be read from the options.PasswordFile file and passed to the rsync command using sshpass. sshpass needs to be available. If createDir is set to true, the destination will be created if it does not exist.
func (Rsync) StderrPipe ¶
func (r Rsync) StderrPipe() (io.ReadCloser, error)
StderrPipe returns a pipe that will be connected to the command's standard error when the command starts.
func (Rsync) StdoutPipe ¶
func (r Rsync) StdoutPipe() (io.ReadCloser, error)
StdoutPipe returns a pipe that will be connected to the command's standard output when the command starts.
type RsyncOptions ¶
type RsyncOptions struct {
// RsyncBinaryPath is a path to the rsync binary; by default just `rsync`
RsyncBinaryPath string
// RsyncPath specify the rsync to run on remote machine, e.g `--rsync-path="cd /a/b && rsync"`
RsyncPath string
// Verbose increase verbosity
Verbose bool
// Quet suppress non-error messages
Quiet bool
// Checksum skip based on checksum, not mod-time & size
Checksum bool
// Archve is archive mode; equals -rlptgoD (no -H,-A,-X)
Archive bool
// Recurse into directories
Recursive bool
// Relative option to use relative path names
Relative bool
// NoImliedDirs don't send implied dirs with --relative
NoImpliedDirs bool
// Update skip files that are newer on the receiver
Update bool
// Inplace update destination files in-place
Inplace bool
// Append data onto shorter files
Append bool
// AppendVerify --append w/old data in file checksum
AppendVerify bool
// Dirs transfer directories without recursing
Dirs bool
// Links copy symlinks as symlinks
Links bool
// CopyLinks transform symlink into referent file/dir
CopyLinks bool
// CopyUnsafeLinks only "unsafe" symlinks are transformed
CopyUnsafeLinks bool
// SafeLinks ignore symlinks that point outside the tree
SafeLinks bool
// CopyDirLinks transform symlink to dir into referent dir
CopyDirLinks bool
// KeepDirLinks treat symlinked dir on receiver as dir
KeepDirLinks bool
// HardLinks preserve hard links
HardLinks bool
// Perms preserve permissions
Perms bool
// NoPerms preserve permissions
NoPerms bool
// Executability preserve executability
Executability bool
// CHMOD affect file and/or directory permissions
CHMOD os.FileMode
// Acls preserve ACLs (implies -p)
ACLs bool
// XAttrs preserve extended attributes
XAttrs bool
// Owner preserve owner (super-user only)
Owner bool
// NoOwner prevent copying owner information to destination
NoOwner bool
// Group preserve group
Group bool
// NoGroup prevent copying group information to destination
NoGroup bool
// Devices preserve device files (super-user only)
Devices bool
// Specials preserve special files
Specials bool
// Times preserve modification times
Times bool
// NoTimes prevent copying modification times
NoTimes bool
// omit directories from --times
OmitDirTimes bool
// Super receiver attempts super-user activities
Super bool
// FakeSuper store/recover privileged attrs using xattrs
FakeSuper bool
// Sparce handle sparse files efficiently
Sparse bool
// DryRun perform a trial run with no changes made
DryRun bool
// WholeFile copy files whole (w/o delta-xfer algorithm)
WholeFile bool
// OneFileSystem don't cross filesystem boundaries
OneFileSystem bool
// BlockSize block-size=SIZE force a fixed checksum block-size
BlockSize int
// Rsh -rsh=COMMAND specify the remote shell to use
Rsh string
// Existing skip creating new files on receiver
Existing bool
// IgnoreExisting skip updating files that exist on receiver
IgnoreExisting bool
// RemoveSourceFiles sender removes synchronized files (non-dir)
RemoveSourceFiles bool
// Delete delete extraneous files from dest dirs
Delete bool
// DeleteBefore receiver deletes before transfer, not during
DeleteBefore bool
// DeleteDuring receiver deletes during the transfer
DeleteDuring bool
// DeleteDelay find deletions during, delete after
DeleteDelay bool
// DeleteAfter receiver deletes after transfer, not during
DeleteAfter bool
// DeleteExcluded also delete excluded files from dest dirs
DeleteExcluded bool
// IgnoreErrors delete even if there are I/O errors
IgnoreErrors bool
// Force deletion of dirs even if not empty
Force bool
// MaxDelete max-delete=NUM don't delete more than NUM files
MaxDelete int
// MaxSize max-size=SIZE don't transfer any file larger than SIZE
MaxSize int
// MinSize don't transfer any file smaller than SIZE
MinSize int
// Partial keep partially transferred files
Partial bool
// PartialDir partial-dir=DIR
PartialDir string
// DelayUpdates put all updated files into place at end
DelayUpdates bool
// PruneEmptyDirs prune empty directory chains from file-list
PruneEmptyDirs bool
// NumericIDs don't map uid/gid values by user/group name
NumericIDs bool
// Timeout: timeout=SECONDS set I/O timeout in seconds
Timeout int
// Contimeout: contimeout=SECONDS set daemon connection timeout in seconds
Contimeout int
// IgnoreTimes don't skip files that match size and time
IgnoreTimes bool
// SizeOnly skip files that match in size
SizeOnly bool
// ModifyWindow modify-window=NUM compare mod-times with reduced accuracy
ModifyWindow bool
// TempDir temp-dir=DIR create temporary files in directory DIR
TempDir string
// Fuzzy find similar file for basis if no dest file
Fuzzy bool
// CompareDest compare-dest=DIR also compare received files relative to DIR
CompareDest string
// CopyDest copy-dest=DIR ... and include copies of unchanged files
CopyDest string
// LinkDest link-dest=DIR hardlink to files in DIR when unchanged
LinkDest string
// Compress file data during the transfer
Compress bool
// CompressLevel explicitly set compression level
CompressLevel int
// SkipCompress skip-compress=LIST skip compressing files with suffix in LIST
SkipCompress []string
// CVSExclude auto-ignore files in the same way CVS does
CVSExclude bool
// Stats give some file-transfer stats
Stats bool
// HumanReadable output numbers in a human-readable format
HumanReadable bool
// Progress show progress during transfer
Progress bool
// Read daemon-access password from FILE
PasswordFile string
// limit socket I/O bandwidth
BandwidthLimit int
// Info
Info string
// Exclude --exclude="", exclude remote paths.
Exclude []string
// Include --include="", include remote paths.
Include []string
// Filter --filter="", include filter rule.
Filter string
// Chown --chown="", chown on receipt.
Chown string
// ListOnly --list-only, list the files instead of copying them.
ListOnly bool
// ipv4
IPv4 bool
// ipv6
IPv6 bool
//out-format
OutFormat bool
}
RsyncOptions for rsync
type State ¶
type State struct {
TimeRemaining string `json:"remain"` // Time Remaining (hh:mm:ss)
DownloadedTotal string `json:"total"` // Amount of downloaded Data in unknown unit
Speed string `json:"speed"` // Speed of download in unknown unit
Progress int `json:"progress"` // Progress in percent (0-100)
}
State contains information about rsync process
type Task ¶
type Task struct {
// contains filtered or unexported fields
}
Task is high-level API under rsync
func NewTask ¶
func NewTask(source, destination string, useSshPass, createDir bool, rsyncOptions RsyncOptions) (*Task, error)
NewTask returns new rsync task
func (*Task) GetFileList ¶
GetFileList is a helper function that returns a partially parsed list of files if RsyncOptions.ListOnly is true. The Information is returned as a slice of slices of strings in the following format: Index Value 0 Permissions 1 Size in Bytes 2 Date 3 Time 4 Name
Source Files
Click to show internal directories.
Click to hide internal directories.