storage

package
v0.14.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 5, 2020 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package storage is a container of all of the database layer implementation. 

Package storage implements the database access layer.  The underlying database
is Postgres, and the tables are defined as such:

	CREATE TABLE aws_resources (
		id VARCHAR PRIMARY KEY,
		account_id VARCHAR NOT NULL,
		region VARCHAR NOT NULL,
		type VARCHAR NOT NULL,
		meta JSONB
	);

We use these simple tables to preserve uniqueness and so we can add columns for additional
metadata when needed without polluting the aws_events_ips_hostnames star table:

	CREATE TABLE aws_ips (
		ip INET PRIMARY KEY
	);

	CREATE TABLE aws_hostnames (
		hostname VARCHAR PRIMARY KEY
	);

Notice "PARTITION BY" below.  We're using built-in partitioning.
See https://blog.timescale.com/scaling-partitioning-data-postgresql-10-explained-cd48a712a9a1/

	CREATE TABLE aws_events_ips_hostnames (
		ts TIMESTAMP NOT NULL,
		is_public BOOLEAN NOT NULL,
		is_join BOOLEAN NOT NULL,
		aws_resources_id VARCHAR NOT NULL,
		FOREIGN KEY (aws_resources_id) REFERENCES aws_resources (id),
		aws_ips_ip INET NOT NULL,
		FOREIGN KEY (aws_ips_ip) REFERENCES aws_ips (ip),
		aws_hostnames_hostname VARCHAR,
		FOREIGN KEY (aws_hostnames_hostname) REFERENCES aws_hostnames (hostname))
	PARTITION BY
		RANGE (
			ts
	);

And we'll make sure there's an index right away:

	CREATE INDEX IF NOT EXISTS aws_events_ips_hostnames_aws_ips_ip_ts_idx ON aws_events_ips_hostnames USING BTREE (aws_ips_ip, ts);

Also, some good advice to follow:  https://www.vividcortex.com/blog/2015/09/22/common-pitfalls-go/

The underlying code is careful to create new partition tables and indices when necessary.  Future updates to the implementation
would require creation of new tables, done by either a database admin or here in code, perhaps in the Init function.

Index

Constants

View Source 
const (
	// EmptySchemaVersion Version of database schema that cleans the database completely. Use cautiously!
	EmptySchemaVersion uint = 0
	// MinimumSchemaVersion Lowest version of database schema current code is able to handle
	MinimumSchemaVersion uint = 1
	// DualWriteSchemaVersion Lowest version of database schema that supports dual-writes
	DualWriteSchemaVersion uint = 2
)

Variables

This section is empty.

Functions

func NewStorageMigrator added in v0.13.0 

func NewStorageMigrator(sourcePath string, db *sql.DB) (domain.StorageMigrator, error)

NewStorageMigrator constructs an instance of StorageMigrator implemented by psql driver + file migration back-end

Types

type DB added in v0.2.0 

type DB struct {
	// contains filtered or unexported fields
}

DB represents a convenient database abstraction layer

func (*DB) DeletePartitions added in v0.4.0 

func (db *DB) DeletePartitions(ctx context.Context, name string) error

DeletePartitions deletes partitions by name.

func (*DB) FetchAll added in v0.9.0 

func (db *DB) FetchAll(ctx context.Context, when time.Time, count uint, offset uint, typeFilter string) ([]domain.CloudAssetDetails, error)

FetchAll gets all the assets present at the specified time

func (*DB) FetchByHostname added in v0.2.0 

func (db *DB) FetchByHostname(ctx context.Context, when time.Time, hostname string) ([]domain.CloudAssetDetails, error)

FetchByHostname gets the assets who have hostname at the specified time

func (*DB) FetchByIP added in v0.2.0 

func (db *DB) FetchByIP(ctx context.Context, when time.Time, ipAddress string) ([]domain.CloudAssetDetails, error)

FetchByIP gets the assets who have IP address at the specified time

func (*DB) GeneratePartition added in v0.2.0 

func (db *DB) GeneratePartition(ctx context.Context, begin time.Time, days int) error

GeneratePartition finds the latest partition, and generate the next partition based on the previous partition's time range

func (*DB) GetPartitions added in v0.3.0 

func (db *DB) GetPartitions(ctx context.Context) ([]domain.Partition, error)

GetPartitions fetches the created partitions and gets each record count in the database

func (*DB) GetSchemaVersion added in v0.13.0 

func (db *DB) GetSchemaVersion(ctx context.Context) (uint, error)

GetSchemaVersion retrieves the current version of database schema

func (*DB) Init added in v0.2.0 

func (db *DB) Init(ctx context.Context, host string, port uint16, user string, password string, dbname string, partitionTTL int) error

Init initializes a connection to a Postgres database according to the environment variables POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DATABASE

func (*DB) MigrateSchemaDown added in v0.13.0 

func (db *DB) MigrateSchemaDown(ctx context.Context) (uint, error)

MigrateSchemaDown performs a database schema rollback one version down

func (*DB) MigrateSchemaToVersion added in v0.13.0 

func (db *DB) MigrateSchemaToVersion(ctx context.Context, version uint) error

MigrateSchemaToVersion performs one or more database migrations to bring schema to the specified version

func (*DB) MigrateSchemaUp added in v0.13.0 

func (db *DB) MigrateSchemaUp(ctx context.Context) (uint, error)

MigrateSchemaUp performs a database schema migration one version up

func (*DB) Store added in v0.2.0 

func (db *DB) Store(ctx context.Context, cloudAssetChanges domain.CloudAssetChanges) error

Store an implementation of the Storage interface that records to a database

func (*DB) StoreV2 added in v0.14.0 

func (db *DB) StoreV2(ctx context.Context, cloudAssetChanges domain.CloudAssetChanges) error

StoreV2 Currently public to allow testing separately.Storage interface implementation that records to a database using new schema, to replace Store in the future.

type PostgresConfig added in v0.2.0 

type PostgresConfig struct {
	Hostname         string
	Port             uint16
	Username         string
	Password         string
	DatabaseName     string
	PartitionTTL     int
	MinSchemaVersion uint
	MigrationsPath   string
}

PostgresConfig contains the Postgres database configuration arguments

func (*PostgresConfig) Name added in v0.2.0 

func (c *PostgresConfig) Name() string

Name is used by the settings library to replace the default naming convention.

type PostgresConfigComponent added in v0.2.0 

type PostgresConfigComponent struct{}

PostgresConfigComponent satisfies the settings library Component API, and may be used by the settings.NewComponent function.

func NewPostgresComponent added in v0.10.0 

func NewPostgresComponent() *PostgresConfigComponent

NewPostgresComponent generates a PostgresConfigComponent

func (*PostgresConfigComponent) New added in v0.2.0 

func (*PostgresConfigComponent) New(ctx context.Context, c *PostgresConfig) (*DB, error)

New constructs a DB from a config.

func (*PostgresConfigComponent) Settings added in v0.2.0 

func (*PostgresConfigComponent) Settings() *PostgresConfig

Settings populates a set of defaults if none are provided via config.

type PostgresReadConfig added in v0.10.0 

type PostgresReadConfig struct {
	Hostname     string
	Port         uint16
	Username     string
	Password     string
	DatabaseName string
	PartitionTTL int
}

PostgresReadConfig contains the Postgres database configuration arguments for the ReadReplica

func (*PostgresReadConfig) Name added in v0.10.0 

func (c *PostgresReadConfig) Name() string

Name is used by the settings library to replace the default naming convention.

type PostgresReadConfigComponent added in v0.10.0 

type PostgresReadConfigComponent struct{}

PostgresReadConfigComponent satisfies the settings library Component API, and may be used by the settings.NewComponent function.

func NewPostgresReadComponent added in v0.10.0 

func NewPostgresReadComponent() *PostgresReadConfigComponent

NewPostgresReadComponent generates a new PostgresReadConfigComponent

func (*PostgresReadConfigComponent) New added in v0.10.0 

func (*PostgresReadConfigComponent) New(ctx context.Context, c *PostgresReadConfig) (*DB, error)

New - Unlike Master, Replica DB has no scripts and does not attempt to create the database as it can not by definition

func (*PostgresReadConfigComponent) Settings added in v0.10.0 

func (*PostgresReadConfigComponent) Settings() *PostgresReadConfig

Settings populates a set of defaults if none are provided via config.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.