archiver

package module
v1.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 8, 2019 License: GPL-3.0 Imports: 24 Imported by: 0

README

RapidPro Archiver Build Status codecov Go Report Card

Simple service for archiving messages and runs to S3 from the RapidPro database.

Usage

Archives RapidPro runs and msgs to S3

Usage of archiver:
  -archive-messages
    	whether we should archive messages (default true)
  -archive-runs
    	whether we should archive runs (default true)
  -aws-access-key-id string
    	the access key id to use when authenticating S3 (default "missing_aws_access_key_id")
  -aws-secret-access-key string
    	the secret access key id to use when authenticating S3 (default "missing_aws_secret_access_key")
  -db string
    	the connection string for our database (default "postgres://localhost/archiver_test?sslmode=disable")
  -debug-conf
    	print where config values are coming from
  -delete
    	whether to delete messages and runs from the db after archival (default false)
  -help
    	print usage information
  -keep-files
    	whether we should keep local archive files after upload (default false)
  -log-level string
    	the log level, one of error, warn, info, debug (default "info")
  -s3-bucket string
    	the S3 bucket we will write archives to (default "dl-archiver-test")
  -s3-disable-ssl
    	whether we disable SSL when accessing S3. Should always be set to False unless you're hosting an S3 compatible service within a secure internal network
  -s3-endpoint string
    	the S3 endpoint we will write archives to (default "https://s3.amazonaws.com")
  -s3-force-path-style
    	whether we force S3 path style. Should generally need to default to False unless you're hosting an S3 compatible service
  -s3-region string
    	the S3 region we will write archives to (default "us-east-1")
  -sentry-dsn string
    	the sentry configuration to log errors to, if any
  -temp-dir string
    	directory where temporary archive files are written (default "/tmp")
  -upload-to-s3
    	whether we should upload archive to S3 (default true)

Environment variables:
                   ARCHIVER_ARCHIVE_MESSAGES - bool
                       ARCHIVER_ARCHIVE_RUNS - bool
                  ARCHIVER_AWS_ACCESS_KEY_ID - string
              ARCHIVER_AWS_SECRET_ACCESS_KEY - string
                                 ARCHIVER_DB - string
                             ARCHIVER_DELETE - bool
                         ARCHIVER_KEEP_FILES - bool
                          ARCHIVER_LOG_LEVEL - string
                          ARCHIVER_S3_BUCKET - string
                     ARCHIVER_S3_DISABLE_SSL - bool
                        ARCHIVER_S3_ENDPOINT - string
                ARCHIVER_S3_FORCE_PATH_STYLE - bool
                          ARCHIVER_S3_REGION - string
                         ARCHIVER_SENTRY_DSN - string
                           ARCHIVER_TEMP_DIR - string
                       ARCHIVER_UPLOAD_TO_S3 - bool

Documentation

Index

Constants

View Source 
const (
	// RunType for run archives
	RunType = ArchiveType("run")

	// MessageType for message archives
	MessageType = ArchiveType("message")

	// SessionType for session archives
	SessionType = ArchiveType("session")
)
View Source 
const (
	// DayPeriod id the period of a day (24 hours) from archive start date
	DayPeriod = ArchivePeriod("D")

	// MonthPeriod is the period of a month from archive start date
	MonthPeriod = ArchivePeriod("M")
)

Variables

This section is empty.

Functions

func ArchiveOrg added in v0.0.6 

func ArchiveOrg(ctx context.Context, now time.Time, config *Config, db *sqlx.DB, s3Client s3iface.S3API, org Org, archiveType ArchiveType) ([]*Archive, []*Archive, error)

ArchiveOrg looks for any missing archives for the passed in org, creating and uploading them as necessary, returning the created archives

func BuildRollupArchive added in v0.0.9 

func BuildRollupArchive(ctx context.Context, db *sqlx.DB, conf *Config, s3Client s3iface.S3API, monthlyArchive *Archive, now time.Time, org Org, archiveType ArchiveType) error

BuildRollupArchive builds a monthly archive from the files present on S3

func CreateArchiveFile added in v0.0.6 

func CreateArchiveFile(ctx context.Context, db *sqlx.DB, archive *Archive, archivePath string) error

CreateArchiveFile is responsible for writing an archive file for the passed in archive from our database

func DeleteArchiveFile added in v0.0.6 

func DeleteArchiveFile(archive *Archive) error

DeleteArchiveFile removes our own disk archive file

func DeleteArchivedMessages added in v0.0.11 

func DeleteArchivedMessages(ctx context.Context, config *Config, db *sqlx.DB, s3Client s3iface.S3API, archive *Archive) error

DeleteArchivedMessages takes the passed in archive, verifies the S3 file is still present (and correct), then selects all the messages in the archive date range, and if equal or fewer than the number archived, deletes them 100 at a time

Upon completion it updates the needs_deletion flag on the archive

func DeleteArchivedRuns added in v0.0.18 

func DeleteArchivedRuns(ctx context.Context, config *Config, db *sqlx.DB, s3Client s3iface.S3API, archive *Archive) error

DeleteArchivedRuns takes the passed in archive, verifies the S3 file is still present (and correct), then selects all the runs in the archive date range, and if equal or fewer than the number archived, deletes them 100 at a time

Upon completion it updates the needs_deletion flag on the archive

func EnsureTempArchiveDirectory added in v0.0.6 

func EnsureTempArchiveDirectory(path string) error

EnsureTempArchiveDirectory checks that we can write to our archive directory, creating it first if needbe

func GetCurrentArchiveCount added in v0.0.11 

func GetCurrentArchiveCount(ctx context.Context, db *sqlx.DB, org Org, archiveType ArchiveType) (int, error)

GetCurrentArchiveCount returns all the current archives for the passed in org and record type

func GetS3File added in v0.0.6 

func GetS3File(ctx context.Context, s3Client s3iface.S3API, fileURL string) (io.ReadCloser, error)

GetS3File return an io.ReadCloser for the passed in bucket and path

func GetS3FileETAG added in v0.0.11 

func GetS3FileETAG(ctx context.Context, s3Client s3iface.S3API, fileURL string) (string, error)

GetS3FileETAG returns the ETAG hash for the passed in file

func NewS3Client added in v0.0.6 

func NewS3Client(config *Config) (s3iface.S3API, error)

NewS3Client creates a new s3 client from the passed in config, testing it as necessary

func TestS3 added in v0.0.6 

func TestS3(s3Client s3iface.S3API, bucket string) error

TestS3 tests whether the passed in s3 client is properly configured and the passed in bucket is accessible

func UploadArchive 

func UploadArchive(ctx context.Context, s3Client s3iface.S3API, bucket string, archive *Archive) error

UploadArchive uploads the passed archive file to S3

func UploadToS3 added in v0.0.17 

func UploadToS3(ctx context.Context, s3Client s3iface.S3API, bucket string, path string, archive *Archive) error

UploadToS3 writes the passed in archive

func WriteArchiveToDB 

func WriteArchiveToDB(ctx context.Context, db *sqlx.DB, archive *Archive) error

WriteArchiveToDB write an archive to the Database

Types

type Archive added in v0.0.6 

type Archive struct {
	ID          int         `db:"id"`
	ArchiveType ArchiveType `db:"archive_type"`
	OrgID       int         `db:"org_id"`
	CreatedOn   time.Time   `db:"created_on"`

	StartDate time.Time     `db:"start_date"`
	Period    ArchivePeriod `db:"period"`

	RecordCount int    `db:"record_count"`
	Size        int64  `db:"size"`
	Hash        string `db:"hash"`
	URL         string `db:"url"`
	BuildTime   int    `db:"build_time"`

	NeedsDeletion bool       `db:"needs_deletion"`
	DeletedOn     *time.Time `db:"deleted_date"`
	Rollup        *int       `db:"rollup_id"`

	Org         Org
	ArchiveFile string
	Dailies     []*Archive
}

Archive represents the model for an archive

func CreateOrgArchives added in v0.0.9 

func CreateOrgArchives(ctx context.Context, now time.Time, config *Config, db *sqlx.DB, s3Client s3iface.S3API, org Org, archiveType ArchiveType) ([]*Archive, error)

CreateOrgArchives builds all the missing archives for the passed in org

func DeleteArchivedOrgRecords added in v0.0.11 

func DeleteArchivedOrgRecords(ctx context.Context, now time.Time, config *Config, db *sqlx.DB, s3Client s3iface.S3API, org Org, archiveType ArchiveType) ([]*Archive, error)

DeleteArchivedOrgRecords deletes all the records for the passeg in org based on archives already created

func GetArchivesNeedingDeletion added in v0.0.11 

func GetArchivesNeedingDeletion(ctx context.Context, db *sqlx.DB, org Org, archiveType ArchiveType) ([]*Archive, error)

GetArchivesNeedingDeletion returns all the archives which need to be deleted

func GetCurrentArchives added in v0.0.6 

func GetCurrentArchives(ctx context.Context, db *sqlx.DB, org Org, archiveType ArchiveType) ([]*Archive, error)

GetCurrentArchives returns all the current archives for the passed in org and record type

func GetDailyArchivesForDateRange added in v0.0.11 

func GetDailyArchivesForDateRange(ctx context.Context, db *sqlx.DB, org Org, archiveType ArchiveType, startDate time.Time, endDate time.Time) ([]*Archive, error)

GetDailyArchivesForDateRange returns all the current archives for the passed in org and record type and date range

func GetMissingDailyArchives added in v0.0.11 

func GetMissingDailyArchives(ctx context.Context, db *sqlx.DB, now time.Time, org Org, archiveType ArchiveType) ([]*Archive, error)

GetMissingDailyArchives calculates what archives need to be generated for the passed in org this is calculated per day

func GetMissingDailyArchivesForDateRange added in v0.0.11 

func GetMissingDailyArchivesForDateRange(ctx context.Context, db *sqlx.DB, startDate time.Time, endDate time.Time, org Org, archiveType ArchiveType) ([]*Archive, error)

GetMissingDailyArchivesForDateRange returns all them missing daily archives between the two passed in date ranges

func GetMissingMonthlyArchives added in v0.0.11 

func GetMissingMonthlyArchives(ctx context.Context, db *sqlx.DB, now time.Time, org Org, archiveType ArchiveType) ([]*Archive, error)

GetMissingMonthlyArchives gets which montly archives are currently missing for this org

func RollupOrgArchives added in v0.0.9 

func RollupOrgArchives(ctx context.Context, now time.Time, config *Config, db *sqlx.DB, s3Client s3iface.S3API, org Org, archiveType ArchiveType) ([]*Archive, error)

RollupOrgArchives rolls up monthly archives from our daily archives

type ArchivePeriod added in v0.0.6 

type ArchivePeriod string

ArchivePeriod is the period of data in the archive

type ArchiveType 

type ArchiveType string

ArchiveType is the type for the archives

type Config 

type Config struct {
	DB        string `help:"the connection string for our database"`
	LogLevel  string `help:"the log level, one of error, warn, info, debug"`
	SentryDSN string `help:"the sentry configuration to log errors to, if any"`

	S3Endpoint       string `help:"the S3 endpoint we will write archives to"`
	S3Region         string `help:"the S3 region we will write archives to"`
	S3Bucket         string `help:"the S3 bucket we will write archives to"`
	S3DisableSSL     bool   `` /* 158-byte string literal not displayed */
	S3ForcePathStyle bool   `` /* 127-byte string literal not displayed */

	AWSAccessKeyID     string `help:"the access key id to use when authenticating S3"`
	AWSSecretAccessKey string `help:"the secret access key id to use when authenticating S3"`

	TempDir    string `help:"directory where temporary archive files are written"`
	KeepFiles  bool   `help:"whether we should keep local archive files after upload (default false)"`
	UploadToS3 bool   `help:"whether we should upload archive to S3"`

	ArchiveMessages bool `help:"whether we should archive messages"`
	ArchiveRuns     bool `help:"whether we should archive runs"`
	Delete          bool `help:"whether to delete messages and runs from the db after archival (default false)"`
}

Config is our top level configuration object

func NewConfig 

func NewConfig() *Config

NewConfig returns a new default configuration object

type Org added in v0.0.6 

type Org struct {
	ID         int       `db:"id"`
	Name       string    `db:"name"`
	CreatedOn  time.Time `db:"created_on"`
	IsAnon     bool      `db:"is_anon"`
	Language   *string   `db:"language"`
	ActiveDays int
}

Org represents the model for an org

func GetActiveOrgs 

func GetActiveOrgs(ctx context.Context, db *sqlx.DB) ([]Org, error)

GetActiveOrgs returns the active organizations sorted by id

Source Files

View all Source files

Directories

Path Synopsis
cmd
rp-archiver

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.