storage

package
v0.23.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 13, 2020 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package storage is a container of all of the database layer implementation. 

Package storage implements the database access layer.  The underlying database
is Postgres, and the tables are defined as such:

	CREATE TABLE aws_resources (
		id VARCHAR PRIMARY KEY,
		account_id VARCHAR NOT NULL,
		region VARCHAR NOT NULL,
		type VARCHAR NOT NULL,
		meta JSONB
	);

We use these simple tables to preserve uniqueness and so we can add columns for additional
metadata when needed without polluting the aws_events_ips_hostnames star table:

	CREATE TABLE aws_ips (
		ip INET PRIMARY KEY
	);

	CREATE TABLE aws_hostnames (
		hostname VARCHAR PRIMARY KEY
	);

Notice "PARTITION BY" below.  We're using built-in partitioning.
See https://blog.timescale.com/scaling-partitioning-data-postgresql-10-explained-cd48a712a9a1/

	CREATE TABLE aws_events_ips_hostnames (
		ts TIMESTAMP NOT NULL,
		is_public BOOLEAN NOT NULL,
		is_join BOOLEAN NOT NULL,
		aws_resources_id VARCHAR NOT NULL,
		FOREIGN KEY (aws_resources_id) REFERENCES aws_resources (id),
		aws_ips_ip INET NOT NULL,
		FOREIGN KEY (aws_ips_ip) REFERENCES aws_ips (ip),
		aws_hostnames_hostname VARCHAR,
		FOREIGN KEY (aws_hostnames_hostname) REFERENCES aws_hostnames (hostname))
	PARTITION BY
		RANGE (
			ts
	);

And we'll make sure there's an index right away:

	CREATE INDEX IF NOT EXISTS aws_events_ips_hostnames_aws_ips_ip_ts_idx ON aws_events_ips_hostnames USING BTREE (aws_ips_ip, ts);

Also, some good advice to follow:  https://www.vividcortex.com/blog/2015/09/22/common-pitfalls-go/

The underlying code is careful to create new partition tables and indices when necessary.  Future updates to the implementation
would require creation of new tables, done by either a database admin or here in code, perhaps in the Init function.

Index

Constants

View Source 
const (
	Primary connectionType = iota
	Replica
)

Database Connection types

View Source 
const (
	// EmptySchemaVersion Version of database schema that cleans the database completely. Use cautiously!
	EmptySchemaVersion uint = 0
	// MinimumSchemaVersion Lowest version of database schema current code is able to handle
	MinimumSchemaVersion uint = 1
	// M1SchemaVersion Version of database schema with performance optimizations (M1) that allows back-fill to work
	M1SchemaVersion uint = 2
	// DualWritesSchemaVersion Lowest version of database schema that supports dual-writes (legacy and M1)
	DualWritesSchemaVersion uint = 3
	// ReadsFromNewSchemaVersion Lowest version of database schema that supports reads from M1 schema
	ReadsFromNewSchemaVersion uint = 4
)

Variables

This section is empty.

Functions

This section is empty.

Types

type DB added in v0.2.0 

type DB struct {
	// contains filtered or unexported fields
}

DB represents a convenient database abstraction layer

func (*DB) BackFillEventsLocally added in v0.15.0 

func (db *DB) BackFillEventsLocally(ctx context.Context, from time.Time, to time.Time) error

BackFillEventsLocally launches the bulk back-fill process using local write handler

func (*DB) DeletePartitions added in v0.4.0 

func (db *DB) DeletePartitions(ctx context.Context, name string) error

DeletePartitions deletes partitions by name.

func (*DB) FetchAll added in v0.9.0 

func (db *DB) FetchAll(ctx context.Context, when time.Time, count uint, offset uint, typeFilter string) ([]domain.CloudAssetDetails, error)

FetchAll gets all the assets present at the specified time

func (*DB) FetchByHostname added in v0.2.0 

func (db *DB) FetchByHostname(ctx context.Context, when time.Time, hostname string) ([]domain.CloudAssetDetails, error)

FetchByHostname gets the assets who have hostname at the specified time

func (*DB) FetchByIP added in v0.2.0 

func (db *DB) FetchByIP(ctx context.Context, when time.Time, ipAddress string) ([]domain.CloudAssetDetails, error)

FetchByIP gets the assets who have IP address at the specified time

func (*DB) GeneratePartition added in v0.2.0 

func (db *DB) GeneratePartition(ctx context.Context, begin time.Time, days int) error

GeneratePartition finds the latest partition, and generate the next partition based on the previous partition's time range

func (*DB) GetPartitions added in v0.3.0 

func (db *DB) GetPartitions(ctx context.Context) ([]domain.Partition, error)

GetPartitions fetches the created partitions and gets each record count in the database

func (*DB) Init added in v0.2.0 

func (db *DB) Init(ctx context.Context, url string, partitionTTL int) error

Init initializes a connection to a Postgres database according to the environment variables POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DATABASE

func (*DB) Store added in v0.2.0 

func (db *DB) Store(ctx context.Context, cloudAssetChanges domain.CloudAssetChanges) error

Store an implementation of the Storage interface that records to a database

func (*DB) StoreV2 added in v0.14.0 

func (db *DB) StoreV2(ctx context.Context, cloudAssetChanges domain.CloudAssetChanges) error

StoreV2 Currently public to allow testing separately.Storage interface implementation that records to a database using new schema, to replace Store in the future.

type LocalDBExportHandler added in v0.15.0 

type LocalDBExportHandler struct {
	// contains filtered or unexported fields
}

LocalDBExportHandler implements local handler for individual events that stores them into v2 schema

func NewLocalExportHandler added in v0.15.0 

func NewLocalExportHandler(db *DB) *LocalDBExportHandler

NewLocalExportHandler returns local handler for individual events that stores them into v2 schema

func (*LocalDBExportHandler) Handle added in v0.15.0 

func (h *LocalDBExportHandler) Handle(changes domain.CloudAssetChanges) error

Handle writes individual events into v2 schema

type PostgresConfig added in v0.2.0 

type PostgresConfig struct {
	URL              string
	ReplicaURL       string
	PartitionTTL     int
	MinSchemaVersion uint
	MigrationsPath   string
}

PostgresConfig contains the Postgres database configuration arguments

func (*PostgresConfig) Name added in v0.2.0 

func (c *PostgresConfig) Name() string

Name is used by the settings library to replace the default naming convention.

type PostgresConfigComponent added in v0.2.0 

type PostgresConfigComponent struct{}

PostgresConfigComponent satisfies the settings library Component API, and may be used by the settings.NewComponent function.

func NewPostgresComponent added in v0.10.0 

func NewPostgresComponent() *PostgresConfigComponent

NewPostgresComponent generates a PostgresConfigComponent

func (*PostgresConfigComponent) New added in v0.2.0 

func (*PostgresConfigComponent) New(ctx context.Context, c *PostgresConfig, t connectionType) (*DB, error)

New constructs a DB from a config.

func (*PostgresConfigComponent) Settings added in v0.2.0 

func (*PostgresConfigComponent) Settings() *PostgresConfig

Settings populates a set of defaults if none are provided via config.

type SchemaManager added in v0.20.0 

type SchemaManager struct {
	DataSourceURL    string
	MigrationsSource string
	// contains filtered or unexported fields
}

SchemaManager is an abstraction layer for manipulating database schema backed by golang/migrate

func NewSchemaManager added in v0.20.0 

func NewSchemaManager(sourcePath string, datasourceURL string) (*SchemaManager, error)

NewSchemaManager generates a SchemaManager component based on settings

func (*SchemaManager) EnsureConnected added in v0.20.0 

func (sm *SchemaManager) EnsureConnected() error

EnsureConnected checks if the migrator database connection is functioning properly and replaces it if needed

func (*SchemaManager) ForceSchemaToVersion added in v0.20.0 

func (sm *SchemaManager) ForceSchemaToVersion(ctx context.Context, version uint) error

ForceSchemaToVersion sets the database schema to specified version without running any migrations and clears dirty flag

func (*SchemaManager) GetSchemaVersion added in v0.20.0 

func (sm *SchemaManager) GetSchemaVersion(ctx context.Context) (uint, error)

GetSchemaVersion retrieves the current version of database schema

func (*SchemaManager) MigrateSchemaDown added in v0.20.0 

func (sm *SchemaManager) MigrateSchemaDown(ctx context.Context) (uint, error)

MigrateSchemaDown performs a database schema rollback one version down

func (*SchemaManager) MigrateSchemaToVersion added in v0.20.0 

func (sm *SchemaManager) MigrateSchemaToVersion(ctx context.Context, version uint) error

MigrateSchemaToVersion performs one or more database migrations to bring schema to the specified version

func (*SchemaManager) MigrateSchemaUp added in v0.20.0 

func (sm *SchemaManager) MigrateSchemaUp(ctx context.Context) (uint, error)

MigrateSchemaUp performs a database schema migration one version up

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.