util

Documentation

Overview

Package util has various helper functions used by components of ratchet.

Index

• func CSVString(v interface{}) string
• func DeleteS3Objects(client *s3.S3, bucket string, objKeys []string) (*s3.DeleteObjectsOutput, error)
• func ExecuteSQLQuery(db *sql.DB, query string) error
• func GetDataFromSQLQuery(db *sql.DB, query string, batchSize int, structDest interface{}) (chan data.JSON, error)
• func GetS3Object(client *s3.S3, bucket, objKey string) (*s3.GetObjectOutput, error)
• func KillPipelineIfErr(err error, killChan chan error)
• func ListS3Objects(client *s3.S3, bucket, keyPrefix string) ([]string, error)
• func SQLInsertData(db *sql.DB, d data.JSON, tableName string, onDupKeyUpdate bool, ...) error
• type CSVWriter
  • func NewCSVWriter(w io.Writer) *CSVWriter
  • func (w *CSVWriter) Error() error
  • func (w *CSVWriter) Flush()
  • func (w *CSVWriter) Write(record []string) (err error)
  • func (w *CSVWriter) WriteAll(records [][]string) (err error)
• type Timer
  • func StartTimer() (t *Timer)
  • func (t *Timer) Duration() time.Duration
  • func (t *Timer) Stop() *Timer
  • func (t *Timer) Stopped() bool
  • func (t *Timer) String() string

Functions

func CSVString

func CSVString(v interface{}) string

func DeleteS3Objects

func DeleteS3Objects(client *s3.S3, bucket string, objKeys []string) (*s3.DeleteObjectsOutput, error)

DeleteS3Objects deletes the objects specified by the given object keys

func ExecuteSQLQuery

func ExecuteSQLQuery(db *sql.DB, query string) error

func GetDataFromSQLQuery

func GetDataFromSQLQuery(db *sql.DB, query string, batchSize int, structDest interface{}) (chan data.JSON, error)

GetDataFromSQLQuery is a util function that, given a properly intialized sql.DB and a valid SQL query, will handle executing the query and getting back data.JSON objects. This function is asynch, and data.JSON should be received on the return data channel. If there was a problem setting up the query, then an error will also be returned immediately. It is also possible for errors to occur during execution as data is retrieved from the query. If this happens, the object returned will be a JSON object in the form of {"Error": "description"}.

func GetS3Object

func GetS3Object(client *s3.S3, bucket, objKey string) (*s3.GetObjectOutput, error)

GetS3Object returns the object output for the given object key

func KillPipelineIfErr

func KillPipelineIfErr(err error, killChan chan error)

KillPipelineIfErr is an error-checking helper.

func ListS3Objects

func ListS3Objects(client *s3.S3, bucket, keyPrefix string) ([]string, error)

ListS3Objects returns all object keys matching the given prefix. Note that delimiter is set to "/". See http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html

func SQLInsertData

func SQLInsertData(db *sql.DB, d data.JSON, tableName string, onDupKeyUpdate bool, onDupKeyFields []string, batchSize int) error

SQLInsertData abstracts building and executing a SQL INSERT statement for the given Data object.

Note that the Data must be a valid JSON object (or an array of valid objects all with the same keys), where the keys are column names and the the values are SQL values to be inserted into those columns.

Types

type CSVWriter

type CSVWriter struct {
	Comma   rune
	UseCRLF bool

	AlwaysEncapsulate bool
	QuoteEscape       string
	// contains filtered or unexported fields
}

func NewCSVWriter

func NewCSVWriter(w io.Writer) *CSVWriter

func (*CSVWriter) Error

func (w *CSVWriter) Error() error

func (*CSVWriter) Flush

func (w *CSVWriter) Flush()

func (*CSVWriter) Write

func (w *CSVWriter) Write(record []string) (err error)

func (*CSVWriter) WriteAll

func (w *CSVWriter) WriteAll(records [][]string) (err error)

type Timer

type Timer struct {
	// contains filtered or unexported fields
}

Timer is a basic mechanism for measuring execution time.

func StartTimer

func StartTimer() (t *Timer)

StartTimer returns a new Timer that's already "started".

func (*Timer) Duration

func (t *Timer) Duration() time.Duration

Duration returns either the total executino duration (if Timer stopped) or the duration until time.Now() if timer is still running.

func (*Timer) Stop

func (t *Timer) Stop() *Timer

Stop sets the end time for the Timer and returns itself.

func (*Timer) Stopped

func (t *Timer) Stopped() bool

Stopped returns true if Stop() has been called on the timer.

func (*Timer) String

func (t *Timer) String() string