storage

Documentation

Overview

Package storage is a container of all of the database layer implementation.

Package storage implements the database access layer.  The underlying database
is Postgres, and the tables are defined as such:

	CREATE TABLE aws_resources (
		id VARCHAR PRIMARY KEY,
		account_id VARCHAR NOT NULL,
		region VARCHAR NOT NULL,
		type VARCHAR NOT NULL,
		meta JSONB
	);

We use these simple tables to preserve uniqueness and so we can add columns for additional
metadata when needed without polluting the aws_events_ips_hostnames star table:

	CREATE TABLE aws_ips (
		ip INET PRIMARY KEY
	);

	CREATE TABLE aws_hostnames (
		hostname VARCHAR PRIMARY KEY
	);

Notice "PARTITION BY" below.  We're using built-in partitioning.
See https://blog.timescale.com/scaling-partitioning-data-postgresql-10-explained-cd48a712a9a1/

	CREATE TABLE aws_events_ips_hostnames (
		ts TIMESTAMP NOT NULL,
		is_public BOOLEAN NOT NULL,
		is_join BOOLEAN NOT NULL,
		aws_resources_id VARCHAR NOT NULL,
		FOREIGN KEY (aws_resources_id) REFERENCES aws_resources (id),
		aws_ips_ip INET NOT NULL,
		FOREIGN KEY (aws_ips_ip) REFERENCES aws_ips (ip),
		aws_hostnames_hostname VARCHAR,
		FOREIGN KEY (aws_hostnames_hostname) REFERENCES aws_hostnames (hostname))
	PARTITION BY
		RANGE (
			ts
	);

And we'll make sure there's an index right away:

	CREATE INDEX IF NOT EXISTS aws_events_ips_hostnames_aws_ips_ip_ts_idx ON aws_events_ips_hostnames USING BTREE (aws_ips_ip, ts);

Also, some good advice to follow:  https://www.vividcortex.com/blog/2015/09/22/common-pitfalls-go/

The underlying code is careful to create new partition tables and indices when necessary.  Future updates to the implementation
would require creation of new tables, done by either a database admin or here in code, perhaps in the Init function.

Index

• type DB
  • func (db *DB) DeletePartitions(ctx context.Context, days int) (int, error)
  • func (db *DB) FetchByHostname(ctx context.Context, when time.Time, hostname string) ([]domain.CloudAssetDetails, error)
  • func (db *DB) FetchByIP(ctx context.Context, when time.Time, ipAddress string) ([]domain.CloudAssetDetails, error)
  • func (db *DB) GeneratePartition(ctx context.Context, begin time.Time, days int) error
  • func (db *DB) GetPartitions(ctx context.Context) ([]domain.Partition, error)
  • func (db *DB) Init(ctx context.Context, postgresConfig *PostgresConfig) error
  • func (db *DB) RunScript(ctx context.Context, name string) error
  • func (db *DB) Store(ctx context.Context, cloudAssetChanges domain.CloudAssetChanges) error
• type PostgresConfig
  • func (c *PostgresConfig) Name() string
• type PostgresConfigComponent
  • func (*PostgresConfigComponent) New(ctx context.Context, c *PostgresConfig) (*DB, error)
  • func (*PostgresConfigComponent) Settings() *PostgresConfig

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB represents a convenient database abstraction layer

func (*DB) DeletePartitions

func (db *DB) DeletePartitions(ctx context.Context, days int) (int, error)

DeletePartitions deletes partitions older than the specified number of days

func (*DB) FetchByHostname

func (db *DB) FetchByHostname(ctx context.Context, when time.Time, hostname string) ([]domain.CloudAssetDetails, error)

FetchByHostname gets the assets who have hostname at the specified time

func (*DB) FetchByIP

func (db *DB) FetchByIP(ctx context.Context, when time.Time, ipAddress string) ([]domain.CloudAssetDetails, error)

FetchByIP gets the assets who have IP address at the specified time

func (*DB) GeneratePartition

func (db *DB) GeneratePartition(ctx context.Context, begin time.Time, days int) error

GeneratePartition finds the latest partition, and generate the next partition based on the previous partition's time range

func (*DB) GetPartitions

func (db *DB) GetPartitions(ctx context.Context) ([]domain.Partition, error)

GetPartitions fetches the created partitions and gets each record count in the database

func (*DB) Init

func (db *DB) Init(ctx context.Context, postgresConfig *PostgresConfig) error

Init initializes a connection to a Postgres database according to the environment variables POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DATABASE

func (*DB) RunScript

func (db *DB) RunScript(ctx context.Context, name string) error

RunScript executes a SQL script from disk against the database.

func (*DB) Store

func (db *DB) Store(ctx context.Context, cloudAssetChanges domain.CloudAssetChanges) error

Store an implementation of the Storage interface that records to a database

type PostgresConfig

type PostgresConfig struct {
	Hostname     string
	Port         string
	Username     string
	Password     string
	DatabaseName string
	PartitionTTL int
}

PostgresConfig contains the Postgres database configuration arguments

func (*PostgresConfig) Name

func (c *PostgresConfig) Name() string

Name is used by the settings library to replace the default naming convention.

type PostgresConfigComponent

type PostgresConfigComponent struct{}

PostgresConfigComponent satisfies the settings library Component API, and may be used by the settings.NewComponent function.

func (*PostgresConfigComponent) New

func (*PostgresConfigComponent) New(ctx context.Context, c *PostgresConfig) (*DB, error)

New constructs a DB from a config.

func (*PostgresConfigComponent) Settings

func (*PostgresConfigComponent) Settings() *PostgresConfig

Settings populates a set of defaults if none are provided via config.