scp

README

Secure Copy Protocol implemented in Go

Overview

Production-ready Secure Copy Protocol (aka: SCP) implemented in Go with well documentation and neat dependency.

Introduction

Secure Copy Protocol uses Secure Shell (SSH) to transfer files between host on a network.

There is no RFC that defines the specifics of the protocol. This package simply implements SCP against the OpenSSH's scp tool, thus you can directly transfer files to/from *uinx system within your Go code, as long as the remote host has OpenSSH installed.

Features

• Copy file from local to remote.
• Copy file from remote to local.
• Copy from buffer to remote file. (e.g: copy from bytes.Reader)
• Copy from remote file to buffer. (e.g: copy to os.Stdout)
• Recursively copy directory from local to remote.
• Recursively copy directory from remote to local.
• Set permission bits for transferred files.
• Set timeout/context for transfer.
• Preserve the permission bits and modification time at transfer.
• No resources leak. (e.g: goroutine, file descriptor)
• Low memory consuming for transferring huge files.
• TODO:
  • Transfer speed limit.
  • Performance benchmark/optimization for lots of small files.
• Won't support:
  • Copy file from remote to remote.

Install

go get github.com/povsister/scp

Example usage

This package leverages golang.org/x/crypto/ssh to establish a SSH connection to remote host.

Error handling are omitted in examples!

Copy a file to remote

// Build a SSH config from username/password
sshConf := scp.NewSSHConfigFromPassword("username", "password")

// Build a SSH config from private key
privPEM, err := ioutil.ReadFile("/path/to/privateKey")
// without passphrase
sshConf, err := scp.NewSSHConfigFromPrivateKey("username", privPEM)
// with passphrase
sshConf, err := scp.NewSSHConfigFromPrivateKey("username", privPEM, passphrase)

// Dial SSH to "my.server.com:22".
// If your SSH server does not listen on 22, simply suffix the address with port.
// e.g: "my.server.com:1234"
scpClient, err := scp.NewClient("my.server.com", sshConf, &scp.ClientOption{})

// Build a SCP client based on existing "golang.org/x/crypto/ssh.Client"
scpClient, err := scp.NewClientFromExistingSSH(existingSSHClient, &scp.ClientOption{})

defer scpClient.Close()


// Do the file transfer without timeout/context
err = scpClient.CopyFileToRemote("/path/to/local/file", "/path/at/remote", &scp.FileTransferOption{})

// Do the file copy with timeout, context and file properties preserved.
// Note that the context and timeout will both take effect.
fo := &scp.FileTransferOption{
    Context: yourCotext,
    Timeout: 30 * time.Second, 
    PreserveProp: true,
}
err = scpClient.CopyFileToRemote("/path/to/local/file", "/path/at/remote", fo)

Copy a file from remote

// Copy the file from remote and save it as "/path/to/local/file".
err = scpClient.CopyFileFromRemote("/path/to/remote/file", "/path/to/local/file", &scp.FileTransferOption{})

// Copy the remote file and print it in Stdout.
err = scpClient.CopyFromRemote("/path/to/remote/file", os.Stdout, &scp.FileTransferOption{})

Copy from buffer to remote as a file

// From buffer
buffer := []byte("something excited")
reader := bytes.NewReader(buffer)

// From fd
// Note that its YOUR responsibility to CLOSE the fd after transfer.
reader, err := os.Open("/path/to/local/file")
defer reader.Close()


// Note that the reader must implement "KnownSize" interface except os.File
// For the content length must be provided before transfer.
// The last part of remote location will be used as file name at remote.
err := scpClient.CopyToRemote(reader, "/path/to/remote/file", &scp.FileTransferOption{})

Recursively copy a directory to remote

// recursively copy to remote
err := scpClient.CopyDirToRemote("/path/to/local/dir", "/path/to/remote/dir", &scp.DirTransferOption{})

// recursively copy to remote with timeout, context and file properties.
// Note that the context and timeout will both take effect.
do := &scp.DirTransferOption{
    Context: yourContext,
    Timeout: 10 * time.Minute,
    PreserveProp: true,
}
err:= scpClient.CopyDirToRemote("/path/to/local/dir", "/path/to/remote/dir", do)

// recursively copy directory contents to remote. Does not create the
// source directory on the target side, like `scp -r /path/to/dir/* ...`
do := &scp.DirTransferOption{
    ContentOnly: true,
}
err := scpClient.CopyDirToRemote("/path/to/local/dir", "/path/to/remote/dir", do)

Recursively copy a directory from remote

// recursively copy from remote.
// The content of remote dir will be save under "/path/to/local".
err := scpClient.CopyDirFromRemote("/path/to/remote/dir", "/path/to/local", &scp.DirTransferOption{})

Progress

// Defines an anonymous function for ReceivePassthrough that conf the
// Writer attribute of PassthroughCopy to track and display the upload 
// progress. The new Writer writes data to the original writer, calculates
// the upload progress, and prints it as a percentage.

type Progress struct {
    TotalSize int64
    Writed int
}

func (p *Progress) Draw(n) () {
    p.Writed = n
    progress := float64(p.writed) / float64(p.TotalSize) * 100
    fmt.Printf("progress: %.2f%%\n", progress)
}

func (p *Progress) Close() error {
    return nil
}

var bar *Progress
do := &scp.DirTransferOption{
    ObserverCallback: func(oet scp.ObserveEventType, ti scp.TransferInfo) {
      if oet == scp.ObserveEventStart {
        bar = &Progress{TotalSize: ti.TotalSize()}
        return
      }
      bar.Draw(ti.TransferredSize())
    },
}
err = scpClient.CopyDirToRemote("/path/to/local/dir", "/path/to/remote/dir", do)

Something you need to know

SCP is a light-weighted protocol which implements file transfer only. It does not support advanced features like: directory listing, resume from break-point.

So, it's commonly used for transferring some small-size, temporary files. If you heavily depend on the file transfer, you may consider using SFTP instead.

Another thing you may notice is that I didn't put context.Context as the first argument in function signature. Instead, it's located in TransferOption. This is intentional because it makes the API also light-weighted.

License

MIT License

Documentation

Overview

Production-ready Secure Copy Protocol (aka: SCP) implemented in Go with well documentation and neat dependency.

Index

• Variables
• func NewSSHConfigFromPassword(username, password string) *ssh.ClientConfig
• func NewSSHConfigFromPrivateKey(username string, privPEM []byte, passphrase ...string) (cfg *ssh.ClientConfig, err error)
• type Client
  • func NewClient(serverAddr string, sshCfg *ssh.ClientConfig, scpOpt *ClientOption) (*Client, error)
  • func NewClientFromExistingSSH(existing *ssh.Client, scpOpt *ClientOption) (*Client, error)
  • func (c *Client) CopyDirFromRemote(remoteDir, localDir string, opt *DirTransferOption) error
  • func (c *Client) CopyDirToRemote(localDir string, remoteDir string, opt *DirTransferOption) error
  • func (c *Client) CopyFileFromRemote(remoteFile, localFile string, opt *FileTransferOption) error
  • func (c *Client) CopyFileToRemote(localFile string, remoteLoc string, opt *FileTransferOption) error
  • func (c *Client) CopyFromRemote(remoteFile string, buffer io.Writer, opt *FileTransferOption) error
  • func (c *Client) CopyToRemote(reader io.Reader, remoteTarget string, opt *FileTransferOption) error
• type ClientOption
• type DirTransferOption
• type FileTransferOption
• type KnownSize
• type ObserveEventType
• type TransferInfo

Variables

var (
	// DefaultFilePerm holds default permission bits for transferred files.
	DefaultFilePerm = os.FileMode(0644)
	// DefaultDirPerm holds default permission bits for transferred directories.
	DefaultDirPerm = os.FileMode(0755)

	// ErrNoTransferOption indicate a non-nil TransferOption should be provided.
	ErrNoTransferOption = errors.New("scp: TransferOption is not provided")

	// DirectoryPreReads sets the num of pre-read files/directories for recursively transferring a directory.
	// Set it larger may speedup the transfer with lots of small files.
	// Do not set it too large or you will exceed the max open files limit.
	DirectoryPreReads = 10
)

var (
	// DebugMode controls the debug output.
	// If true, the debug information of remote scp server
	// will be printed in Stderr.
	DebugMode = false
)

var ErrInvalidWrite = errors.New("invalid write result")

var (
	// ErrNoClientOption indicates a non-nil ClientOption should be provided
	ErrNoClientOption = errors.New("scp: ClientOption is not provided")
)

var MAX_BUFFSIZE int64 = 1024 * 1024 // 1MB for small size transfer improvements

var MIN_BUFFSIZE int64 = 32 * 1024 // 32kB, the common buffsize

Functions

func NewSSHConfigFromPassword

func NewSSHConfigFromPassword(username, password string) *ssh.ClientConfig

NewSSHConfigFromPassword returns a *ssh.ClientConfig with ssh.Password AuthMethod and 3 seconds timeout for connecting the server.

It *insecurely* ignores server's host key validation.

func NewSSHConfigFromPrivateKey

func NewSSHConfigFromPrivateKey(username string, privPEM []byte, passphrase ...string) (cfg *ssh.ClientConfig, err error)

NewSSHConfigFromPrivateKey returns a *ssh.ClientConfig with ssh.PublicKey AuthMethod and 3 seconds timeout for connecting the server.

The passphrase is optional. If multiple passphrase are provided, only the first will be used.

If the private key is encrypted, it will return a ssh.PassphraseMissingError.

It *insecurely* ignores server's host key validation.

Types

type Client

type Client struct {
	// The underlying ssh client
	*ssh.Client
	// contains filtered or unexported fields
}

Client has the "golang.org/x/crypto/ssh/Client" embedded, so it can be used as normal SSH client with additional SCP features.

func NewClient

func NewClient(serverAddr string, sshCfg *ssh.ClientConfig, scpOpt *ClientOption) (*Client, error)

NewClient returns a SSH client with SCP capability.

The serverAddr should be in "host" or "host:port" format. If no port is supplied, the default port 22 will be used.

IPv6 serverAddr must be enclosed in square brackets, as in "[::1]" or "[::1]:22"

func NewClientFromExistingSSH

func NewClientFromExistingSSH(existing *ssh.Client, scpOpt *ClientOption) (*Client, error)

NewClientFromExistingSSH returns a SSH client with SCP capability. It reuse the existing SSH connection without dialing

func (*Client) CopyDirFromRemote

func (c *Client) CopyDirFromRemote(remoteDir, localDir string, opt *DirTransferOption) error

CopyDirFromRemote recursively copies a remote directory into local directory. The localDir must exist before copying.

For example:

• CopyDirFromRemote("/remote/dir1", "/local/dir2", &DirTransferOption{})
• Results: "remote/dir1/<contents>" -> "/local/dir2/<contents>"

func (*Client) CopyDirToRemote

func (c *Client) CopyDirToRemote(localDir string, remoteDir string, opt *DirTransferOption) error

CopyDirToRemote recursively copies a directory to remoteDir.

func (*Client) CopyFileFromRemote

func (c *Client) CopyFileFromRemote(remoteFile, localFile string, opt *FileTransferOption) error

CopyFileFromRemote copies a remoteFile as localFile.

If localFile does not exist, it will be automatically created. If localFile already exists, it will be truncated for writing. If localFile is a directory, the name of remoteFile will be used.

For example:

• CopyFileFromRemote("/remote/file1", "/local/fileNotExist", &FileTransferOption)

• Result: "/remote/file1" -> "/local/file2" The "fileNotExist" will be created.

• CopyFileFromRemote("/remote/file1", "/local/fileExist", &FileTransferOption)

• Result: "/remote/file1" -> "/local/fileExist" The "fileExist" will be truncated for writing.

• CopyFileFromRemote("/remote/file1", "/local/dir", &FileTransferOption)

• Result: "/remote/file1" -> "/local/dir/file1" The "file1" will be used as filename and stored under "/local/dir" directory. Note that "/local/dir" must exist in this case.

func (*Client) CopyFileToRemote

func (c *Client) CopyFileToRemote(localFile string, remoteLoc string, opt *FileTransferOption) error

CopyFileToRemote copies a local file to remote location. It will automatically close the file after read.

func (*Client) CopyFromRemote

func (c *Client) CopyFromRemote(remoteFile string, buffer io.Writer, opt *FileTransferOption) error

CopyFromRemote copies a remote file into buffer.

Note that "PreserveProp" and "Perm" option does not take effect in this case.

func (*Client) CopyToRemote

func (c *Client) CopyToRemote(reader io.Reader, remoteTarget string, opt *FileTransferOption) error

CopyToRemote copies content from reader to remoteTarget. The reader must implement "KnownSize" interface except *os.File.

Currently, it supports following readers:

• *os.File
• *strings.Reader
• *bytes.Reader

Note that the last part of remoteTarget will be used as filename if unspecified.

It's CALLER'S responsibility to CLOSE the file if an *os.File is supplied.

type ClientOption

type ClientOption struct {
	// Use sudo to run remote scp server.
	// Default: false.
	Sudo bool
	// The scp remote server executable file.
	// Default: "scp".
	//
	// If your scp command is not in the default path,
	// specify it as "/path/to/scp".
	RemoteBinary string
}

ClientOption contains several configurable options for SCP client.

type DirTransferOption

type DirTransferOption struct {
	// Context for the directory transfer.
	// Can be both set with Timeout.
	// Default: no context
	Context context.Context
	// Timeout for transferring the whole directory.
	// Can be both set with Context.
	// Default: 0 (means no timeout)
	Timeout time.Duration
	// Preserve modification times and modes from the original file/directory.
	// Default: false
	PreserveProp bool
	// Limits the used bandwidth, specified in Kbit/s.
	// Default: 0 (Means no limit)
	// TODO: not implemented yet
	SpeedLimit int64
	// ContentOnly skips creating the source directory on the receiving side, and
	// only transfers the source directory's contents.
	// Default: false
	ContentOnly bool
	// ObserverCallback is an anonymous function that is called during different events in a file transfer.
	// It is invoked at the start of the transfer with ObserveEventType set to ObserveEventStart,
	// at the end of the transfer with ObserveEventType set to ObserveEventEnd,
	// and periodically during the transfer with ObserveEventType set to ObserveEventTick.
	// The TransferInfo object passed to the function contains information about the transfer,
	// such as the file name, size, and the amount of data transferred so far.
	ObserverCallback func(ObserveEventType, TransferInfo)
}

DirTransferOption holds the transfer options for directory.

type FileTransferOption

type FileTransferOption struct {
	// Context for the file transfer.
	// Can be both set with Timeout.
	// Default: no context
	Context context.Context
	// Timeout for transferring the file.
	// Can be both set with Context.
	// Default: 0 (Means no timeout)
	Timeout time.Duration
	// The permission bits for transferred file.
	// Override "PreserveProp" if specified.
	// Default: 0644
	Perm os.FileMode
	// Preserve modification times and permission bits from the original file.
	// Only valid for file transfer.
	// Default: false
	PreserveProp bool
	// Limits the used bandwidth, specified in Kbit/s.
	// Default: 0 (Means no limit)
	// TODO: not implemented yet
	SpeedLimit int64
	// ObserverCallback is an anonymous function that is called during different events in a file transfer.
	// It is invoked at the start of the transfer with ObserveEventType set to ObserveEventStart,
	// at the end of the transfer with ObserveEventType set to ObserveEventEnd,
	// and periodically during the transfer with ObserveEventType set to ObserveEventTick.
	// The TransferInfo object passed to the function contains information about the transfer,
	// such as the file name, size, and the amount of data transferred so far.
	ObserverCallback func(ObserveEventType, TransferInfo)
}

FileTransferOption holds the transfer options for file.

type KnownSize

type KnownSize interface {
	// return num in bytes
	Size() int64
}

KnownSize is intended for reader whose size is already known before reading.

type ObserveEventType

type ObserveEventType string

const (
	ObserveEventStart ObserveEventType = "start"
	ObserveEventTick  ObserveEventType = "tick"
	ObserveEventEnd   ObserveEventType = "end"
)

type TransferInfo

type TransferInfo interface {
	// Name returns the name of the file being transferred.
	Name() string
	// Path returns the full path of the file in the system.
	Path() string
	// TotalSize returns the total size of the file in bytes.
	TotalSize() int64
	// TransferredSize returns the number of bytes that have been transferred so far.
	TransferredSize() int64
	// LastUpdate returns a time.Time object indicating when the transfer information was last updated.
	LastUpdate() time.Time
	// Err returns an error if any issue occurred during the transfer; otherwise, it returns nil.
	Err() error
}

TransferInfo provides detailed information about a file transfer operation, allowing access to key attributes such as the file name, path, total size, amount of data transferred, last update time, and any errors that may have occurred during the transfer.