sts

package module
v1.18.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 11, 2024 License: MIT Imports: 14 Imported by: 0

README

Site Transfer Software (STS)

STS is a utility used for transmitting data over wide-area networks with the following priorities:

  • In-order delivery
  • Confirmed transfer via hash validation
  • Efficient use of bandwidth using HTTP and gzip compression
  • Bandwidth sharing among configured groups of files to avoid starvation

Usage

$ sts -help
  -conf string
      Configuration file path
  -debug
      Log program flow
  -help
      Print the help message
  -loop
      Run in a loop, i.e. don't exit until interrupted
  -mode string
      Mode: "out", "in", "auto" (default "auto")

NOTE: Specifying -mode auto (or if -mode omitted since auto is the default) will use the configuration file to determine which mode(s) to run. If only an OUT block is present then it will run in out mode. If only an IN block is present then it will run in in mode. If both exist then both are run.

Example Configuration

Below is an example configuration file. The "outgoing" and "incoming" blocks do not have to be included in the same configuration file. In fact, by default if -conf is not used, STS will look in $STS_HOME (or $PWD if not defined) for sts.{mode}.yaml (or sts.yaml if running -mode auto).

# OUTGOING CONFIGURATION
OUT:
  dirs: # Outgoing directory configuration; relative to $STS_HOME or directory of configuration file if not absolute
    cache : .sts     # Used to store queue cache(s)
    out   : data/out # Directory to watch for files to send; appends "/{target name}"
    logs  : data/log # Root directory for logging
    logs-out  : out  # Appended to root logs dir for logging of files sent (DEFAULT: outgoing_to); appends /{target}
    logs-flow : flow # Appended to root logs for error/info/debug messages (DEFAULT: messages)
  sources: # Supports multiple sources where omitted entries will inherit from previous sources hierarchically
    - name          : ...   # Name of the source (used by receiver)
      out-dir       : /...  # Override the global output directory setting
      log-dir       : /...  # Override the global log directory setting
      threads       : 8     # Maximum number of concurrent HTTP connections
      bin-size      : 10MB  # The generally-desired size for a given HTTP request (BEFORE any compression)
      compress      : 4     # Use GZIP compression of level 0-9 (0 = no compression, 9 = best but slowest)
      min-age       : 15s   # How old a file must be before being added to the "outgoing" queue
      max-age       : 12h   # How old a file can be before getting logged as "stale" (remains in the queue)
      cache-age     : 24h   # Interval at which to check files in the cache
      scan-delay    : 30s   # How long to wait between scans of the outgoing directory
      timeout       : 30m   # The HTTP timeout for a single request
      stat-payload  : false # Whether or not to log each payload's throughput stats
      stat-interval : 5m    # How often to log throughput statistics
      poll-delay    : 5s    # How long to wait after file sent before final validation
      poll-interval : 1m    # How long to wait between polling requests
      poll-attempts : 10    # How many times to "poll" for the successful reception of a file before re-sending
      target: # Target-specific configuration
        name          : ...      # Name of the target
        http-host     : ...:1992 # Target host, including port
        # If target not setup for HTTPS, remove or comment out the line below:
        http-tls-cert : conf/server.pem  # Public server certificate path; relative to $STS_HOME or $PWD if not absolute
      tags: # Tags are for configuration based on file patterns (omitted attributes are inherited)
        - pattern   : DEFAULT # The file "tag" pattern
          priority  : 0       # Relative importance (higher the number, greater the importance)
          order     : fifo    # File order (fifo (first in, first out) or none)
          delete    : true    # Whether or not to delete files after reception confirmation
          method    : http    # Transfer method ("http", "disk", or "none")
      include-hidden : false  # Whether or not to include "hidden" files
      include: # Regular expression pattern(s) as white list for files to send
        - ...
      ignore: # Regular expression pattern(s) as black list for files NOT to send
        - ...
      group-by : ^([^\.]*)  # How to group files for bandwidth sharing


# INCOMING CONFIGURATION
IN:
  sources: # Only accept requests from sources identified in the list below
    - ...
  keys: # Only accept requests with one of the keys from the list below
    - ...
  dirs: # Incoming directory configuration; relative to $STS_HOME or $PWD if not absolute
    stage : data/stage # Directory to stage data as it comes in; appends "/{source name}"
    final : data/in    # Final location for incoming files; appends "/{source name}"
    logs  : data/log   # Root directory for logging
    logs-in   : in   # Appended to root logs dir for logging of files sent (DEFAULT: incoming_from); appends /{source}
    logs-flow : flow # Appended to root logs for error/info/debug messages (DEFAULT: messages)
  server: # Server configuration.
    http-port     : 1992 # What port to listen on
    # HTTPS can be disabled by removing or commenting out the following two lines:
    http-tls-cert : conf/server.pem # Public server certificate path; relative to $STS_HOME or $PWD if not absolute
    http-tls-key  : conf/server.key # Private server key; relative to $STS_HOME or $PWD if not absolute
    compress      : 4 # Use GZIP compression of level 0-9 (0 = no compression, 9 = best but slowest) on response payloads

Definitions

Go (aka Golang)

An open-source programming language developed by Google. STS is written solely in Go and compiled to a single executable.

Thread

A Go routine.

Queue Cache

A data store (currently kept in memory and cached in JSON format to disk after each scan) to manage files in the outgoing queue. It contains a file's path, size, and mod time. Its purpose is to keep track of files found but not fully sent and validated such that on a crash recovery, files can be appropriately resent without duplication.

Bin

A bin is the payload of data pushed via HTTP to the configured target. A bin is composed of N files or file parts up to the configured bin size (before compression).

Stage Area

Configured directory where files are received and reconstructed before being moved to the final configured location.

Start-up

If STS is configured to send, the first thing it will do is check the cache to see if any files are already in the queue from last run. If so, a request is made to the target to find out what "partials" exist and corresponding companion metadata is returned. From this, STS can determine which files are partially sent already and which ones need to be polled for validation. Some of these might have already been fully received and some not at all. STS will handle both of these cases to avoid duplicate transfers.

Following the outgoing recovery logic, four components are started for managing the outgoing flow: Watcher, Queue, Sender, and Validator. A similar set of components will be started for any additionally configured source + target. Each source + target has its own configuration block. In send mode STS will also use a configurable number of threads used for making concurrent HTTP requests to the configured target.

If STS is configured to run as a receiver, an HTTP Server is started and file reception is handed off to the Stager which handles file validation and moving to the "final" directory.

Logical Flow

  1. Source Watcher: Files found in configured watch directory are cached in memory (and on disk) after computing (in parallel) each file's MD5 and passed to the Queue.

If the program crashes unexpectedly, STS will use the queue cache to perform a recovery procedure on next run that will pick up where it left off without sending duplicate data.

  1. Source Queue: Files received from Watcher are sorted in order of configured priority and/or in-order delivery. It will output single file chunks no larger than the configured payload (aka "bin") size and will rotate between groups of similar files (based on configurable pattern match) of the same priority in order to avoid starvation.

  2. Source Sender: Does these activities in parallel:

    1. Construct "bin" until full (or until input file stream is stagnant for a second)
    2. POST "bin" to target HTTP server, optionally compressing the payload
    3. Log successfully sent files and pass these to the Validator

  3. Source Validator: Builds a batch of files to validate based on configurable time interval and makes a request to the target host to confirm files sent have been validated and finalized. Files that fail validation are removed from the cache such the Watcher will pick them up again. Files that pass validation are removed if configured to do so. After a configurable number of poll attempts do not yield success, files are treated the same as if they had failed validation originally.

  4. Target HTTP Server: Receives POSTed "bin" from source host and writes data and companion metadata file to configured Stager.

  5. Target Stager: As streams of file parts are received from the HTTP Server, they are efficiently written to their file counterparts with a ".part" extension to indicate the file is not yet complete. Once the last part is written (mutex locks are used to avoid conflict by multiple threads) the file is renamed to remove the previously added ".part" extension and its MD5 hash is computed to make sure it matches the one stored in the "companion" file (.cmp extension). If the file matches AND its predecessor (if there is one) has also been received, it is moved to the "final" directory.

Documentation

Index

Constants

View Source 
const (

	// OrderFIFO indicates first-in, first-out sorting
	OrderFIFO = "fifo"

	// OrderLIFO indicates last-in, first-out sorting
	OrderLIFO = "lifo"

	// OrderNone indicates ordering is not important
	OrderNone = "none"

	// OrderAlpha indicates alphabetic ordering
	OrderAlpha = ""
)
View Source 
const (
	// DefLog is the default logs directory name
	DefLog = "logs"

	// DefLogMsg is the default log messages directory name
	// (appended to "logs")
	DefLogMsg = "messages"

	// DefLogOut is the default outgoing log messages directory name
	// (appended to "logs")
	DefLogOut = "outgoing_to"

	// DefLogIn is the default incoming log messages directory name
	// (appended to "logs")
	DefLogIn = "incoming_from"

	// DefOut is the default outgoing data directory name
	DefOut = "outgoing_to"

	// DefCache is the default cache directory name
	DefCache = ".sts"

	// DefStage is the default stage directory name
	DefStage = "stage"

	// DefFinal is the default final directory name
	DefFinal = "incoming_from"

	// MethodHTTP indicates HTTP transfer method
	MethodHTTP = "http"

	// DefaultPort is the default TCP port used for HTTP communication
	DefaultPort = 1992
)
View Source 
const (
	// ConfirmNone is the indicator that a file has not been confirmed.
	ConfirmNone = 0

	// ConfirmFailed is the indicator that file confirmation failed.
	ConfirmFailed = 1

	// ConfirmPassed is the indicator that file confirmation succeeded.
	ConfirmPassed = 2

	// ConfirmWaiting is the indicator that file confirmation succeeded but its
	// predecessor has not been confirmed.
	ConfirmWaiting = 3
)

Variables

This section is empty.

Functions

func InitPaths 

func InitPaths(conf *Conf,
	join func(...string) string,
	mkpath func(*string, bool) error) (err error)

InitPaths will initialize all conf paths and use the provided root for resolving relative paths

func SetDefault 

func SetDefault(def map[*string]string)

SetDefault will take a map of string pointers and set the values to the corresponding string in the map if it is empty

Types

type Binnable 

type Binnable interface {
	Sendable
	GetNextAlloc() (int64, int64)
	AddAlloc(int64)
	IsAllocated() bool
}

Binnable is the interface a file must implement to have a chunk be part of a sendable bin

type Binned 

type Binned interface {
	GetName() string
	GetRenamed() string
	GetPrev() string
	GetFileTime() time.Time
	GetFileHash() string
	GetFileSize() int64
	// GetSendSize will almost always be the same as GetFileSize.  The one
	// exception is for files recovered where some portion of the file was
	// sent earlier.
	GetSendSize() int64
	GetSlice() (int64, int64)
}

Binned is the interface for a single file chunk that is part of a payload

type ByteRange 

type ByteRange struct {
	Beg int64 `json:"b"`
	End int64 `json:"e"`
}

ByteRange is the JSON-encodable struct used by the Partial for tracking a a single byte range

type Cached 

type Cached interface {
	Hashed
	IsDone() bool
}

Cached is the interface a cached file must implement

type ClientConf 

type ClientConf struct {
	Dirs    *ClientDirs   `yaml:"dirs" json:"dirs"`
	Sources []*SourceConf `yaml:"sources" json:"sources"`
}

ClientConf is the struct for housing all outgoing configuration

func (*ClientConf) UnmarshalJSON 

func (conf *ClientConf) UnmarshalJSON(data []byte) (err error)

UnmarshalJSON implements the Unmarshaler interface for handling custom member(s) https://golang.org/pkg/encoding/json/#Unmarshal

func (*ClientConf) UnmarshalYAML 

func (conf *ClientConf) UnmarshalYAML(
	unmarshal func(interface{}) error,
) (err error)

UnmarshalYAML implements the Unmarshaler interface for handling custom member(s) https://godoc.org/gopkg.in/yaml.v2

type ClientDirs 

type ClientDirs struct {
	Out       string `yaml:"out" json:"out"`
	OutFollow bool   `yaml:"out-follow" json:"out-follow"`
	Log       string `yaml:"logs" json:"logs"`
	LogOut    string `yaml:"logs-out" json:"logs-out"`
	LogMsg    string `yaml:"logs-flow" json:"logs-flow"`
	Cache     string `yaml:"cache" json:"cache"`
}

ClientDirs is the struct for managing the outgoing directory configuration items

type ClientManager 

type ClientManager interface {
	GetClientStatus(clientID, clientName, clientOS string) (ClientStatus, error)
	GetClientConf(clientID string) (*ClientConf, error)
	SetClientConfReceived(clientID string, when time.Time) error
	SetClientState(clientID string, state ClientState) error
}

ClientManager is responsible for disseminating and collecting information about a client (sender)

type ClientState 

type ClientState struct {
	When      string                 `json:"when"`
	Version   string                 `json:"vers"`
	GoVersion string                 `json:"go"`
	BuildTime string                 `json:"built"`
	IsActive  bool                   `json:"active"`
	IPAddrs   []string               `json:"ip"`
	Messages  []string               `json:"msg"`
	Sources   map[string]SourceState `json:"sources"`
}

ClientState is the outer structure for a runner to communicate to the server its current state

type ClientStatus 

type ClientStatus uint

ClientStatus uses bitmasking to represent the status of a client 

const (
	// ClientIsDisabled indicates that transfer is to not occur
	ClientIsDisabled ClientStatus = 1 << iota

	// ClientIsApproved indicates that transfer should be enabled
	ClientIsApproved

	// ClientHasUpdatedConfiguration indicates, well, the updated configuration
	// should be retrieved
	ClientHasUpdatedConfiguration
)

type Conf 

type Conf struct {
	AgentKey string      `yaml:"stackimpact" json:"stackimpact"`
	Client   *ClientConf `yaml:"OUT" json:"out"`
	Server   *ServerConf `yaml:"IN" json:"in"`
}

Conf is the outer struct for decoding a YAML config file

func NewConf 

func NewConf(path string) (*Conf, error)

NewConf returns a parsed Conf reference based on the provided conf file path

type DBConf 

type DBConf struct {
	Host          string `yaml:"host" json:"host"`
	Port          uint   `yaml:"port" json:"port"`
	User          string `yaml:"user" json:"user"`
	Pass          string `yaml:"pass" json:"pass"`
	Name          string `yaml:"name" json:"name"`
	ClientsTable  string `yaml:"table-clients" json:"table-clients"`
	DatasetsTable string `yaml:"table-datasets" json:"table-datasets"`
}

DBConf is the struct for defining a database connection

type DecodeClientID 

type DecodeClientID func(clientID string) (key, uid string)

DecodeClientID converts a composite client ID into its original key and unique ID

type DecodePartials 

type DecodePartials func(r io.Reader) ([]*Partial, error)

DecodePartials decodes the input reader into a slice of Partial instance pointers

type Dispatcher 

type Dispatcher interface {
	Send(string) error
}

Dispatcher is the interface for broadcasting messages

type EncodeClientID 

type EncodeClientID func(key, uid string) (clientID string)

EncodeClientID creates a decodable composite ID from a key and unique ID

type File 

type File interface {
	GetPath() string
	GetName() string
	GetSize() int64
	GetTime() time.Time
	GetMeta() []byte
}

File is the most basic interface for a File object

type FileCache 

type FileCache interface {
	Iterate(func(Cached) bool)
	Get(string) Cached
	Add(Hashed)
	Done(name string, whileLocked func(Cached))
	Reset(string)
	Remove(string)
	Persist() error
}

FileCache is the interface for caching a collection of files

type FileQueue 

type FileQueue interface {
	Push([]Hashed)
	Pop() Sendable
}

FileQueue is the interface for getting files in the proper order

type FileSource 

type FileSource interface {
	Scan(func(File) bool) ([]File, time.Time, error)
	GetOpener() Open
	Remove(File) error
	Sync(File) (File, error)
	IsNotExist(error) bool
}

FileSource is the interface for reading and deleting from a store of file objects on the client (i.e. sending) side

type GateKeeper 

type GateKeeper interface {
	Recover()
	CleanNow()
	Prune(time.Duration)
	Ready() bool
	Scan(version string) ([]byte, error)
	Prepare(request []Binned)
	Receive(*Partial, io.Reader) error
	Received([]Binned) (nRecvd int)
	GetFileStatus(relPath string, sent time.Time) int
	Stop(bool)
}

GateKeeper is the interface for managing the "putting away" of files received on the server

type GateKeeperFactory 

type GateKeeperFactory func(source string) GateKeeper

GateKeeperFactory creates GateKeeper instances

type HTTPServer 

type HTTPServer struct {
	Host                     string  `yaml:"http-host" json:"http-host"`
	Port                     int     `yaml:"http-port" json:"http-port"`
	PathPrefix               string  `yaml:"http-path-prefix" json:"http-path-prefix"`
	TLSCertPath              string  `yaml:"http-tls-cert" json:"http-tls-cert"`
	TLSKeyPath               string  `yaml:"http-tls-key" json:"http-tls-key"`
	Compression              int     `yaml:"compress" json:"compress"`
	ChanceOfSimulatedFailure float64 `yaml:"chance-of-simulated-failure" json:"chance-of-simulated-failure"`
}

HTTPServer is the struct for managing the incoming HTTP host

type Hashed 

type Hashed interface {
	File
	GetHash() string
}

Hashed is the interface a file must implement to include a signature

type Logger 

type Logger interface {
	Debug(...interface{})
	Info(...interface{})
	Error(...interface{})
	Recent(int) []string
}

Logger is the generic logging interface

type MappingConf 

type MappingConf struct {
	Pattern  *regexp.Regexp
	Template string
}

MappingConf is the struct for indicating what path pattern should be mapped to a different target name based on the associated template string

func (*MappingConf) MarshalJSON 

func (m *MappingConf) MarshalJSON() ([]byte, error)

MarshalJSON implements Marshaler interface for handling custom member(s) https://golang.org/pkg/encoding/json/#Marshal

func (*MappingConf) UnmarshalJSON 

func (m *MappingConf) UnmarshalJSON(data []byte) (err error)

UnmarshalJSON implements the Unmarshaler interface for handling custom member(s) https://golang.org/pkg/encoding/json/#Unmarshal

func (*MappingConf) UnmarshalYAML 

func (m *MappingConf) UnmarshalYAML(unmarshal func(interface{}) error) (err error)

UnmarshalYAML implements the Unmarshaler interface for handling custom member(s): https://godoc.org/gopkg.in/yaml.v2

type Open 

type Open func(File) (Readable, error)

Open is a generic function for creating a Readable handle to a file

type Partial 

type Partial struct {
	Name    string           `json:"path"`
	Renamed string           `json:"renamed"`
	Prev    string           `json:"prev"`
	Time    marshal.NanoTime `json:"time"`
	Size    int64            `json:"size"`
	Hash    string           `json:"hash"`
	Source  string           `json:"src"`
	Parts   []*ByteRange     `json:"parts"`
}

Partial is the JSON-encodable struct containing metadata for a single file on the receiving end

type Payload 

type Payload interface {
	Add(Binnable) bool
	Remove(Binned)
	IsFull() bool
	Split(nParts int) Payload
	GetSize() int64
	GetParts() []Binned
	EncodeHeader() ([]byte, error)
	GetEncoder() io.ReadCloser
	GetStarted() time.Time
	GetCompleted() time.Time
}

Payload is the interface to a slice of Binnables that can be read for transmission

type PayloadDecoder 

type PayloadDecoder interface {
	GetParts() []Binned
	Next() (io.Reader, bool)
}

PayloadDecoder is the interface on the receiving side to a sent payload

type PayloadDecoderFactory 

type PayloadDecoderFactory func(
	metaLen int, pathSep string, payload io.Reader) (PayloadDecoder, error)

PayloadDecoderFactory creates a decoder instance to be used to parse a multipart byte stream with the metadata at the beginning

type PayloadFactory 

type PayloadFactory func(int64, Open, Rename) Payload

PayloadFactory creates a Payload instance based on the input max size (in bytes) and Open function

type Pollable 

type Pollable interface {
	Sent
	GetPrev() string // Needed in case the file needs to be resent
	GetStarted() time.Time
}

Pollable is the interface a file must implement to be polled by the client for whether or not the server received the file successfully

type Polled 

type Polled interface {
	Pollable
	NotFound() bool
	Waiting() bool
	Failed() bool
	Received() bool
}

Polled is the interface a file must implement as a response to being polled

type Queue 

type Queue struct {
	Region string `yaml:"aws-region" json:"aws-region"`
	Name   string `yaml:"name" json:"name"`
}

Queue is the struct for managing an AWS queue resource

type Readable 

type Readable interface {
	io.Reader
	io.Seeker
	io.Closer
}

Readable is a wrapper for being able to seek, read, and close a file generically

type ReceiveLogger 

type ReceiveLogger interface {
	Parse(
		handler func(name, renamed, hash string, size int64, t time.Time) bool,
		after time.Time,
		before time.Time) bool
	Received(file Received)
	WasReceived(
		name, hash string,
		after time.Time,
		before time.Time) bool
}

ReceiveLogger is the interface for logging on the incoming side

type Received 

type Received interface {
	GetName() string
	GetRenamed() string
	GetSize() int64
	GetHash() string
}

Received is the interface a file must implement to be logged as received

type RecentFile 

type RecentFile struct {
	Name   string    `json:"name"`
	Rename string    `json:"rename"`
	Path   string    `json:"path"`
	Hash   string    `json:"hash"`
	Size   int64     `json:"size"`
	Time   time.Time `json:"time"`
	Sent   bool      `json:"sent"`
}

RecentFile is the data structure that encapsulates the state of a file used to communicate client state to the server

type Recover 

type Recover func() ([]*Partial, error)

Recover is the function type for determining what partial files need to be sent since previous run

type RecoverTransmission 

type RecoverTransmission func(Payload) (int, error)

RecoverTransmission is the function type for querying the server about a failed payload to determine which parts if any were successfully received in order to avoid duplication

type Recovered 

type Recovered interface {
	Hashed
	GetPrev() string
	GetSendSize() int64
	Allocate(int64) (int64, int64)
	IsAllocated() bool
}

Recovered is the interface a file must implement to be recovered after a partial send

type Rename 

type Rename func(File) string

Rename is a generic function for generating a new name for a file

type RequestValidator 

type RequestValidator func(source, key string) bool

RequestValidator validates an incoming request

type SendLogger 

type SendLogger interface {
	Sent(Sent)
	WasSent(name, hash string, after time.Time, before time.Time) bool
}

SendLogger is the interface for logging on the sending side

type Sendable 

type Sendable interface {
	Hashed
	GetPrev() string
	GetSlice() (int64, int64)
	GetSendSize() int64
}

Sendable is the interface a file must implement to be sent by the client

type Sent 

type Sent interface {
	GetName() string
	GetSize() int64
	GetHash() string
	TimeMs() int64
}

Sent is the interface a file must implement to be recorded as sent

type ServerConf 

type ServerConf struct {
	SourceControl *DBConf     `yaml:"control" json:"control"`
	Sources       []string    `yaml:"sources" json:"sources"`
	Keys          []string    `yaml:"keys" json:"keys"`
	Dirs          *ServerDirs `yaml:"dirs" json:"dirs"`
	Server        *HTTPServer `yaml:"server" json:"server"`
	Queue         *Queue      `yaml:"queue" json:"queue"`
	PermitLogBuf  bool        `yaml:"log-buffering" json:"log-buffering"`
}

ServerConf is the struct for housing all incoming configuration

type ServerDirs 

type ServerDirs struct {
	Log    string `yaml:"logs" json:"logs"`
	LogIn  string `yaml:"logs-in" json:"logs-in"`
	LogMsg string `yaml:"logs-flow" json:"logs-flow"`
	Stage  string `yaml:"stage" json:"stage"`
	Final  string `yaml:"final" json:"final"`
	Serve  string `yaml:"serve" json:"serve"`
}

ServerDirs is the struct for managing the incoming directory configuration items

type SourceConf 

type SourceConf struct {
	Name          string
	OutDir        string
	LogDir        string
	Threads       int
	CacheAge      time.Duration
	MinAge        time.Duration
	MaxAge        time.Duration
	ScanDelay     time.Duration
	Timeout       time.Duration
	Compression   int
	StatInterval  time.Duration
	PollDelay     time.Duration
	PollInterval  time.Duration
	PollAttempts  int
	Target        *TargetConf
	Rename        []*MappingConf
	Tags          []*TagConf
	BinSize       units.Base2Bytes
	StatPayload   bool
	GroupBy       *regexp.Regexp
	IncludeHidden bool
	Include       []*regexp.Regexp
	Ignore        []*regexp.Regexp
	ErrorBackoff  float64
	// contains filtered or unexported fields
}

SourceConf is the struct for managing the configuration of an outgoing client source

func (*SourceConf) GenMappingVars 

func (c *SourceConf) GenMappingVars() map[string]string

func (*SourceConf) MarshalJSON 

func (ss *SourceConf) MarshalJSON() ([]byte, error)

MarshalJSON implements Marshaler interface for handling custom member(s) https://golang.org/pkg/encoding/json/#Marshal

func (*SourceConf) UnmarshalJSON 

func (ss *SourceConf) UnmarshalJSON(data []byte) (err error)

UnmarshalJSON implements the Unmarshaler interface for handling custom member(s) https://golang.org/pkg/encoding/json/#Unmarshal

func (*SourceConf) UnmarshalYAML 

func (ss *SourceConf) UnmarshalYAML(
	unmarshal func(interface{}) error,
) (err error)

UnmarshalYAML implements the Unmarshaler interface for handling custom member(s) https://godoc.org/gopkg.in/yaml.v2

type SourceState 

type SourceState struct {
	FoundCount  int64         `json:"foundCnt"`
	FoundSize   int64         `json:"foundSize"`
	QueueCount  int64         `json:"qCnt"`
	QueueSize   int64         `json:"qSize"`
	SentCount   int64         `json:"sentCnt"`
	SentSize    int64         `json:"sentSize"`
	Throughput  Throughput    `json:"throughput"`
	RecentFiles []*RecentFile `json:"files"`
}

SourceState is the structure used to communicate to the server the current state of a single running source (an outgoing "client")

type TagConf 

type TagConf struct {
	Priority    int
	Method      string
	Order       string
	Pattern     *regexp.Regexp
	Delete      bool
	LastDelay   time.Duration
	DeleteDelay time.Duration
	// contains filtered or unexported fields
}

TagConf is the struct for managing configuration options for "tags" (groups) of files

func (*TagConf) MarshalJSON 

func (t *TagConf) MarshalJSON() ([]byte, error)

MarshalJSON implements Marshaler interface for handling custom member(s) https://golang.org/pkg/encoding/json/#Marshal

func (*TagConf) UnmarshalJSON 

func (t *TagConf) UnmarshalJSON(data []byte) (err error)

UnmarshalJSON implements the Unmarshaler interface for handling custom member(s) https://golang.org/pkg/encoding/json/#Unmarshal

func (*TagConf) UnmarshalYAML 

func (t *TagConf) UnmarshalYAML(unmarshal func(interface{}) error) (err error)

UnmarshalYAML implements the Unmarshaler interface for handling custom member(s) https://godoc.org/gopkg.in/yaml.v2

type TargetConf 

type TargetConf struct {
	Name          string `yaml:"name" json:"name"`
	Key           string `yaml:"key" json:"key"`
	Host          string `yaml:"http-host" json:"http-host"`
	PathPrefix    string `yaml:"http-path-prefix" json:"http-path-prefix"`
	TLSCertPath   string `yaml:"http-tls-cert" json:"http-tls-cert"`
	TLSCertBase64 string `yaml:"http-tls-cert-encoded" json:"http-tls-cert-encoded"`
}

TargetConf houses the configuration for the target host for a given source

func (*TargetConf) ParseHost 

func (t *TargetConf) ParseHost() (host string, port int, err error)

ParseHost returns the hostname and port split out or uses default port

type Throughput 

type Throughput struct {
	Bytes   int64   `json:"bytes"`
	Seconds float64 `json:"secs"`
	PctIdle float64 `json:"idle"`
}

Throughput is the structure used to keep track of bandwidth usage

type Translate 

type Translate func(string) string

Translate is a general function type for converting one string to another

type Transmit 

type Transmit func(Payload) (int, error)

Transmit is the function type for actually sending payload over the wire (or whatever). It returns some representation of how much was successfully transmitted and any error encountered.

type Validate 

type Validate func([]Pollable) ([]Polled, error)

Validate is the function type for validating files sent by the client were successfully received by the server

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.