archiver

package module
v0.0.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 18, 2018 License: GPL-3.0 Imports: 23 Imported by: 1

README

RapidPro Archiver Build Status codecov Go Report Card

Simple service for archiving messages, runs and sessions to S3 from the RapidPro database.

Usage

Archives RapidPro flows, msgs and sessions to S3

Usage of archiver:
  -aws-access-key-id string
    	the access key id to use when authenticating S3 (default "missing_aws_access_key_id")
  -aws-secret-access-key string
    	the secret access key id to use when authenticating S3 (default "missing_aws_secret_access_key")
  -db string
    	the connection string for our database (default "postgres://localhost/temba?sslmode=disable")
  -debug-conf
    	print where config values are coming from
  -help
    	print usage information
  -log-level string
    	the log level, one of error, warn, info, debug (default "info")
  -s3-bucket string
    	the S3 bucket we will write archives to (default "dl-temba-archives")
  -s3-disable-ssl
    	whether we disable SSL when accessing S3. Should always be set to False unless you're hosting an S3 compatible service within a secure internal network
  -s3-endpoint string
    	the S3 endpoint we will write archives to (default "https://s3.amazonaws.com")
  -s3-force-path-style
    	whether we force S3 path style. Should generally need to default to False unless you're hosting an S3 compatible service
  -s3-region string
    	the S3 region we will write archives to (default "us-east-1")
  -sentry-dsn string
    	the sentry configuration to log errors to, if any

Environment variables:
                  ARCHIVER_AWS_ACCESS_KEY_ID - string
              ARCHIVER_AWS_SECRET_ACCESS_KEY - string
                                 ARCHIVER_DB - string
                          ARCHIVER_LOG_LEVEL - string
                          ARCHIVER_S3_BUCKET - string
                     ARCHIVER_S3_DISABLE_SSL - bool
                        ARCHIVER_S3_ENDPOINT - string
                ARCHIVER_S3_FORCE_PATH_STYLE - bool
                          ARCHIVER_S3_REGION - string
                         ARCHIVER_SENTRY_DSN - string

Documentation

Index

Constants

View Source 
const (
	// RunType for run archives
	RunType = ArchiveType("run")

	// MessageType for message archives
	MessageType = ArchiveType("message")

	// SessionType for session archives
	SessionType = ArchiveType("session")
)
View Source 
const (
	// DayPeriod id the period of a day (24 hours) from archive start date
	DayPeriod = ArchivePeriod("D")

	// MonthPeriod is the period of a month from archive start date
	MonthPeriod = ArchivePeriod("M")
)

Variables

This section is empty.

Functions

func BuildMonthlyArchive added in v0.0.6 

func BuildMonthlyArchive(ctx context.Context, conf *Config, s3Client s3iface.S3API, archives []*Archive, month *Archive, now time.Time, org Org, archiveType ArchiveType) error

BuildMonthlyArchive builds a monthly archive from the files present on S3

func CreateArchiveFile added in v0.0.6 

func CreateArchiveFile(ctx context.Context, db *sqlx.DB, archive *Archive, archivePath string) error

CreateArchiveFile is responsible for writing an archive file for the passed in archive from our database

func DeleteArchiveFile added in v0.0.6 

func DeleteArchiveFile(archive *Archive) error

DeleteArchiveFile removes our own disk archive file

func EnsureTempArchiveDirectory added in v0.0.6 

func EnsureTempArchiveDirectory(path string) error

EnsureTempArchiveDirectory checks that we can write to our archive directory, creating it first if needbe

func GetS3File added in v0.0.6 

func GetS3File(ctx context.Context, s3Client s3iface.S3API, fileURL string) (io.ReadCloser, error)

GetS3File return an io.ReadCloser for the passed in bucket and path

func NewS3Client added in v0.0.6 

func NewS3Client(config *Config) (s3iface.S3API, error)

NewS3Client creates a new s3 client from the passed in config, testing it as necessary

func PutS3File added in v0.0.6 

func PutS3File(s3Client s3iface.S3API, bucket string, path string, contentType string, contentEncoding string, filename string, md5 string) (string, error)

PutS3File writes the passed in file to the bucket with the passed in content type

func TestS3 added in v0.0.6 

func TestS3(s3Client s3iface.S3API, bucket string) error

TestS3 tests whether the passed in s3 client is properly configured and the passed in bucket is accessible

func UploadArchive 

func UploadArchive(ctx context.Context, s3Client s3iface.S3API, bucket string, archive *Archive) error

UploadArchive uploads the passed archive file to S3

func WithAcceptEncoding added in v0.0.6 

func WithAcceptEncoding(e string) request.Option

func WriteArchiveToDB 

func WriteArchiveToDB(ctx context.Context, db *sqlx.DB, archive *Archive) error

WriteArchiveToDB write an archive to the Database

Types

type Archive added in v0.0.6 

type Archive struct {
	ID          int         `db:"id"`
	ArchiveType ArchiveType `db:"archive_type"`
	OrgID       int         `db:"org_id"`
	CreatedOn   time.Time   `db:"created_on"`

	StartDate time.Time     `db:"start_date"`
	Period    ArchivePeriod `db:"period"`

	RecordCount int    `db:"record_count"`
	Size        int64  `db:"size"`
	Hash        string `db:"hash"`
	URL         string `db:"url"`
	BuildTime   int    `db:"build_time"`

	IsPurged bool `db:"is_purged"`
	Rollup   *int `db:"rollup_id"`

	Org         Org
	ArchiveFile string
	Dailies     []*Archive
}

Archive represents the model for an archive

func ArchiveOrg added in v0.0.6 

func ArchiveOrg(ctx context.Context, now time.Time, config *Config, db *sqlx.DB, s3Client s3iface.S3API, org Org, archiveType ArchiveType) ([]*Archive, error)

ArchiveOrg looks for any missing archives for the passed in org, creating and uploading them as necessary, returning the created archives

func BuildOrgDailyArchives added in v0.0.6 

func BuildOrgDailyArchives(ctx context.Context, now time.Time, config *Config, db *sqlx.DB, s3Client s3iface.S3API, org Org, archiveType ArchiveType) ([]*Archive, error)

BuildOrgDailyArchives builds all the montly archives for the passid in org

func BuildOrgMonthlyArchives added in v0.0.6 

func BuildOrgMonthlyArchives(ctx context.Context, now time.Time, config *Config, db *sqlx.DB, s3Client s3iface.S3API, org Org, archiveType ArchiveType) ([]*Archive, error)

BuildOrgMonthlyArchives builds all the montly archives for the passid in org

func GetCurrentArchives added in v0.0.6 

func GetCurrentArchives(ctx context.Context, db *sqlx.DB, org Org, archiveType ArchiveType) ([]*Archive, error)

GetCurrentArchives returns all the current archives for the passed in org and record type

func GetMissingDayArchives added in v0.0.6 

func GetMissingDayArchives(archives []*Archive, now time.Time, org Org, archiveType ArchiveType) ([]*Archive, error)

GetMissingDayArchives calculates what archives need to be generated for the passed in org this is calculated per day

func GetMissingMonthArchives added in v0.0.6 

func GetMissingMonthArchives(archives []*Archive, now time.Time, org Org, archiveType ArchiveType) ([]*Archive, error)

GetMissingMonthArchives gets which archives need to be rolled up in the set that has been passed in

type ArchivePeriod added in v0.0.6 

type ArchivePeriod string

ArchivePeriod is the period of data in the archive

type ArchiveType 

type ArchiveType string

ArchiveType is the type for the archives

type Config 

type Config struct {
	DB        string `help:"the connection string for our database"`
	LogLevel  string `help:"the log level, one of error, warn, info, debug"`
	SentryDSN string `help:"the sentry configuration to log errors to, if any"`

	S3Endpoint       string `help:"the S3 endpoint we will write archives to"`
	S3Region         string `help:"the S3 region we will write archives to"`
	S3Bucket         string `help:"the S3 bucket we will write archives to"`
	S3DisableSSL     bool   `` /* 158-byte string literal not displayed */
	S3ForcePathStyle bool   `` /* 127-byte string literal not displayed */

	AWSAccessKeyID     string `help:"the access key id to use when authenticating S3"`
	AWSSecretAccessKey string `help:"the secret access key id to use when authenticating S3"`

	TempDir           string `help:"directory where temporary archive files are written"`
	DeleteAfterUpload bool   `help:"whether we should delete temporary archive file"`
	UploadToS3        bool   `help:"whether we should upload archive to S3"`

	ArchiveMessages bool `help:"whether we should archive messages"`
	ArchiveRuns     bool `help:"whether we should archive runs"`
}

Config is our top level configuration object

func NewConfig 

func NewConfig() *Config

NewConfig returns a new default configuration object

type Org added in v0.0.6 

type Org struct {
	ID         int       `db:"id"`
	Name       string    `db:"name"`
	CreatedOn  time.Time `db:"created_on"`
	IsAnon     bool      `db:"is_anon"`
	ActiveDays int
}

Org represents the model for an org

func GetActiveOrgs 

func GetActiveOrgs(ctx context.Context, db *sqlx.DB) ([]Org, error)

GetActiveOrgs returns the active organizations sorted by id

Source Files

View all Source files

Directories

Path Synopsis
cmd
rp-archiver

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.