datastore

package module
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 8, 2023 License: MIT Imports: 31 Imported by: 14

README

go-datastore

Data layer using GORM for accessing models via a Database (MySQL, Postgres, SQLite, MongoDB)

Release Build Status Report codecov Mergify Status Go
Gitpod Ready-to-Code standard-readme compliant Makefile Included Sponsor Donate


Table of Contents


Installation

go-datastore requires a supported release of Go.

go get -u github.com/mrz1836/go-datastore

Documentation

View the generated documentation

GoDoc

Database model assumptions:

  • id If a unique ID is needed, the datastore expects id and uses _id internally for Mongo
  • metadata is an optional field for storing key->value JSON data

Custom array and object fields:

  • WithCustomFields(arrayFields, objectFields ) Use this method to add custom array or object fields (IE: metadata)

Custom methods for Mongo:

  • WithCustomMongoConditionProcessor() Use this method to add custom condition processing for custom object fields
  • WithCustomMongoIndexer() Use this method to add custom mongo indexes

Repository Features

This repository was created using MrZ's go-template

Built-in Features
Package Dependencies
Library Deployment

Releases are automatically created when you create a new git tag!

If you want to manually make releases, please install GoReleaser:

goreleaser for easy binary or library deployment to GitHub and can be installed:

  • using make: make install-releaser
  • using brew: brew install goreleaser

The .goreleaser.yml file is used to configure goreleaser.


Automatic releases via GitHub Actions from creating a new tag:

make tag version=1.2.3

Manual Releases (optional)

Use make release-snap to create a snapshot version of the release, and finally make release to ship to production (manually).


Makefile Commands

View all makefile commands

make help

List of all current commands:

all                           Runs multiple commands
clean                         Remove previous builds and any cached data
clean-mods                    Remove all the Go mod cache
coverage                      Shows the test coverage
diff                          Show the git diff
generate                      Runs the go generate command in the base of the repo
godocs                        Sync the latest tag with GoDocs
help                          Show this help message
install                       Install the application
install-all-contributors      Installs all contributors locally
install-go                    Install the application (Using Native Go)
install-releaser              Install the GoReleaser application
lint                          Run the golangci-lint application (install if not found)
release                       Full production release (creates release in GitHub)
release                       Runs common.release then runs godocs
release-snap                  Test the full release (build binaries)
release-test                  Full production test release (everything except deploy)
replace-version               Replaces the version in HTML/JS (pre-deploy)
tag                           Generate a new tag and push (tag version=0.0.0)
tag-remove                    Remove a tag if found (tag-remove version=0.0.0)
tag-update                    Update an existing tag to current commit (tag-update version=0.0.0)
test                          Runs lint and ALL tests
test-ci                       Runs all tests via CI (exports coverage)
test-ci-no-race               Runs all tests via CI (no race) (exports coverage)
test-ci-short                 Runs unit tests via CI (exports coverage)
test-no-lint                  Runs just tests
test-short                    Runs vet, lint and tests (excludes integration tests)
test-unit                     Runs tests and outputs coverage
uninstall                     Uninstall the application (and remove files)
update-contributors           Regenerates the contributors html/list
update-linter                 Update the golangci-lint package (macOS only)
vet                           Run the Go vet application

Examples & Tests

All unit tests and examples run via GitHub Actions and uses Go version 1.18.x. View the configuration file.


Run all tests (including integration tests)

make test

Run tests (excluding integration tests)

make test-short

Benchmarks

Run the Go benchmarks:

make bench

Code Standards

Read more about this Go project's code standards.


Usage

Checkout all the examples!


Contributing

View the contributing guidelines and follow the code of conduct.


How can I help?

All kinds of contributions are welcome 🙌! The most basic way to show your support is to star 🌟 the project, or to raise issues 💬. You can also support this project by becoming a sponsor on GitHub 👏 or by making a bitcoin donation to ensure this journey continues indefinitely! 🚀

Stars


Contributors ✨

Thank you to these wonderful people (emoji key):


Mr. Z
🚇 💻 🚧 🛡️
Siggi
🚇 💻 🛡️

This project follows the all-contributors specification.


License

License

Documentation

Overview

Package datastore is the database service abstraction layer

Index

Constants

View Source 
const (

	// SortDesc will sort descending
	SortDesc = "desc"

	// SortAsc will sort ascending
	SortAsc = "asc"
)

Defaults for library functionality

View Source 
const (
	Postgres = "postgres"
	JSON     = "JSON"
	JSONB    = "JSONB"
)

index creation constants

Variables

View Source 
var (
	// DateFields are standard known date fields
	DateFields = []string{dateCreatedAt, dateUpdatedAt, dateModifiedAt}
)
View Source 
var ErrDuplicateKey = errors.New("duplicate key")

ErrDuplicateKey error when a record is inserted and conflicts with an existing record

View Source 
var ErrNoResults = errors.New("no results found")

ErrNoResults error when no results are found

View Source 
var ErrNoSourceFound = errors.New("no source database found in all given configurations")

ErrNoSourceFound is when no source database is found in all given configurations

View Source 
var ErrUnknownCollection = errors.New("could not determine collection name from model")

ErrUnknownCollection is thrown when the collection can not be found using the model/name

View Source 
var ErrUnknownSQL = errors.New("unknown sql implementation")

ErrUnknownSQL is an error when using a SQL engine that is not known for indexes and migrations

View Source 
var ErrUnsupportedDriver = errors.New("sql driver unsupported")

ErrUnsupportedDriver is when the given SQL driver is not determined to be known or supported

View Source 
var ErrUnsupportedEngine = errors.New("unsupported datastore engine")

ErrUnsupportedEngine is used when the engine given is not a known datastore engine

View Source 
var SQLDatabases = []Engine{
	MySQL,
	PostgreSQL,
	SQLite,
}

SQLDatabases is the list of supported SQL databases (via GORM)

Functions

func GetModelBoolAttribute 

func GetModelBoolAttribute(model interface{}, attribute string) *bool

GetModelBoolAttribute the attribute from the model as a bool

func GetModelName 

func GetModelName(model interface{}) *string

GetModelName get the name of the model via reflection

func GetModelStringAttribute 

func GetModelStringAttribute(model interface{}, attribute string) *string

GetModelStringAttribute the attribute from the model as a string

func GetModelTableName 

func GetModelTableName(model interface{}) *string

GetModelTableName get the db table name of the model via reflection

func GetModelType 

func GetModelType(model interface{}) reflect.Type

GetModelType get the model type of the model interface via reflection

func GetModelUnset 

func GetModelUnset(model interface{}) map[string]bool

GetModelUnset gets any empty values on the model and makes sure the update actually unsets those values in the database, otherwise this never happens, and we cannot unset

func IsModelSlice 

func IsModelSlice(model interface{}) bool

IsModelSlice returns true if the given interface is a slice of models

func IsSQLEngine 

func IsSQLEngine(e Engine) bool

IsSQLEngine check whether the string already is in the slice

func MarshalQueryParams 

func MarshalQueryParams(m QueryParams) graphql.Marshaler

MarshalQueryParams will marshal the custom type

func StringInSlice 

func StringInSlice(a string, list []string) bool

StringInSlice check whether the string already is in the slice

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is the datastore client (configuration)

func (*Client) AutoMigrateDatabase 

func (c *Client) AutoMigrateDatabase(ctx context.Context, models ...interface{}) error

AutoMigrateDatabase will detect the engine and migrate as needed

func (*Client) Close 

func (c *Client) Close(ctx context.Context) error

Close will terminate (close) the datastore and any open connections

func (*Client) CreateInBatches 

func (c *Client) CreateInBatches(
	ctx context.Context,
	models interface{},
	batchSize int,
) error

CreateInBatches create all the models given in batches

func (*Client) CreateInBatchesMongo 

func (c *Client) CreateInBatchesMongo(
	ctx context.Context,
	models interface{},
	batchSize int,
) error

CreateInBatchesMongo insert multiple models vai bulk.Write

func (*Client) CustomWhere 

func (c *Client) CustomWhere(tx CustomWhereInterface, conditions map[string]interface{}, engine Engine) interface{}

CustomWhere add conditions

func (*Client) Debug 

func (c *Client) Debug(on bool)

Debug will set the debug flag

func (*Client) DebugLog 

func (c *Client) DebugLog(ctx context.Context, text string)

DebugLog will display verbose logs

func (*Client) Engine 

func (c *Client) Engine() Engine

Engine will return the client's engine

func (*Client) Execute 

func (c *Client) Execute(query string) *gorm.DB

Execute a SQL query

func (*Client) GetArrayFields 

func (c *Client) GetArrayFields() []string

GetArrayFields will return the array fields

func (*Client) GetDatabaseName 

func (c *Client) GetDatabaseName() string

GetDatabaseName will return the full database name for the given model name

func (*Client) GetModel 

func (c *Client) GetModel(
	ctx context.Context,
	model interface{},
	conditions map[string]interface{},
	timeout time.Duration,
	forceWriteDB bool,
) error

GetModel will get a model from the datastore

func (*Client) GetModelCount 

func (c *Client) GetModelCount(
	ctx context.Context,
	model interface{},
	conditions map[string]interface{},
	timeout time.Duration,
) (int64, error)

GetModelCount will return a count of the model matching conditions

func (*Client) GetModels 

func (c *Client) GetModels(
	ctx context.Context,
	models interface{},
	conditions map[string]interface{},
	queryParams *QueryParams,
	fieldResults interface{},
	timeout time.Duration,
) error

GetModels will return a slice of models based on the given conditions

func (*Client) GetModelsAggregate 

func (c *Client) GetModelsAggregate(ctx context.Context, models interface{},
	conditions map[string]interface{}, aggregateColumn string, timeout time.Duration) (map[string]interface{}, error)

GetModelsAggregate will return an aggregate count of the model matching conditions

func (*Client) GetMongoCollection 

func (c *Client) GetMongoCollection(
	collectionName string,
) *mongo.Collection

GetMongoCollection will get the mongo collection for the given tableName

func (*Client) GetMongoCollectionByTableName 

func (c *Client) GetMongoCollectionByTableName(
	tableName string,
) *mongo.Collection

GetMongoCollectionByTableName will get the mongo collection for the given tableName

func (*Client) GetMongoConditionProcessor 

func (c *Client) GetMongoConditionProcessor() func(conditions *map[string]interface{})

GetMongoConditionProcessor will return a custom mongo condition processor if set

func (*Client) GetMongoIndexer 

func (c *Client) GetMongoIndexer() func() map[string][]mongo.IndexModel

GetMongoIndexer will return a custom mongo condition indexer

func (*Client) GetObjectFields 

func (c *Client) GetObjectFields() []string

GetObjectFields will return the object fields

func (*Client) GetTableName 

func (c *Client) GetTableName(modelName string) string

GetTableName will return the full table name for the given model name

func (*Client) HasMigratedModel 

func (c *Client) HasMigratedModel(modelType string) bool

HasMigratedModel will return if the model type has been migrated

func (*Client) IncrementModel 

func (c *Client) IncrementModel(
	ctx context.Context,
	model interface{},
	fieldName string,
	increment int64,
) (newValue int64, err error)

IncrementModel will increment the given field atomically in the database and return the new value

func (*Client) IndexExists 

func (c *Client) IndexExists(tableName, indexName string) (bool, error)

IndexExists check whether the given index exists in the datastore

func (*Client) IndexMetadata 

func (c *Client) IndexMetadata(tableName, field string) error

IndexMetadata check and creates the metadata json index

func (*Client) IsAutoMigrate 

func (c *Client) IsAutoMigrate() bool

IsAutoMigrate returns whether auto migration is on

func (*Client) IsDebug 

func (c *Client) IsDebug() bool

IsDebug will return the debug flag (bool)

func (*Client) IsNewRelicEnabled 

func (c *Client) IsNewRelicEnabled() bool

IsNewRelicEnabled will return if new relic is enabled

func (*Client) NewTx 

func (c *Client) NewTx(ctx context.Context, fn func(*Transaction) error) error

NewTx will start a new datastore transaction

func (*Client) Raw 

func (c *Client) Raw(query string) *gorm.DB

Raw a raw SQL query

func (*Client) SaveModel 

func (c *Client) SaveModel(
	ctx context.Context,
	model interface{},
	tx *Transaction,
	newRecord, commitTx bool,
) error

SaveModel will take care of creating or updating a model (primary key based) (abstracting the database)

value is a pointer to the model

type ClientInterface 

type ClientInterface interface {
	GetterInterface
	StorageService
	Close(ctx context.Context) error
	Debug(on bool)
	DebugLog(ctx context.Context, text string)
	Engine() Engine
	IsAutoMigrate() bool
	IsDebug() bool
	IsNewRelicEnabled() bool
}

ClientInterface is the Datastore client interface

func NewClient 

func NewClient(ctx context.Context, opts ...ClientOps) (ClientInterface, error)

NewClient creates a new client for all Datastore functionality

If no options are given, it will use the defaultClientOptions() ctx may contain a NewRelic txn (or one will be created)

type ClientOps 

type ClientOps func(c *clientOptions)

ClientOps allow functional options to be supplied that overwrite default client options.

func WithAutoMigrate 

func WithAutoMigrate(migrateModels ...interface{}) ClientOps

WithAutoMigrate will enable auto migrate database mode (given models)

Pointers of structs (IE: &models.Xpub{})

func WithCustomFields 

func WithCustomFields(arrayFields []string, objectFields []string) ClientOps

WithCustomFields will add custom fields to the datastore

func WithCustomMongoConditionProcessor 

func WithCustomMongoConditionProcessor(f func(conditions *map[string]interface{})) ClientOps

WithCustomMongoConditionProcessor will add a custom mongo condition processor function

func WithCustomMongoIndexer 

func WithCustomMongoIndexer(f func() map[string][]mongo.IndexModel) ClientOps

WithCustomMongoIndexer will add a custom mongo index function (returns custom mongo indexes)

func WithDebugging 

func WithDebugging() ClientOps

WithDebugging will enable debugging mode

func WithLogger 

func WithLogger(customLogger zLogger.GormLoggerInterface) ClientOps

WithLogger will set the custom logger interface

func WithMongo 

func WithMongo(config *MongoDBConfig) ClientOps

WithMongo will set the datastore to use MongoDB

func WithMongoConnection 

func WithMongoConnection(database *mongo.Database, tablePrefix string) ClientOps

WithMongoConnection will set the datastore to use an existing Mongo database connection

func WithNewRelic 

func WithNewRelic() ClientOps

WithNewRelic will enable the NewRelic wrapper

func WithSQL 

func WithSQL(engine Engine, configs []*SQLConfig) ClientOps

WithSQL will load a datastore using either an SQL database config or existing connection

func WithSQLConnection 

func WithSQLConnection(engine Engine, sqlDB *sql.DB, tablePrefix string) ClientOps

WithSQLConnection will set the datastore to an existing connection for MySQL or PostgreSQL

func WithSQLite 

func WithSQLite(config *SQLiteConfig) ClientOps

WithSQLite will set the datastore to use SQLite

type CommonConfig 

type CommonConfig struct {
	Debug                 bool          `json:"debug" mapstructure:"debug"`                                       // flag for debugging sql queries in logs
	MaxConnectionIdleTime time.Duration `json:"max_connection_idle_time" mapstructure:"max_connection_idle_time"` // 360
	MaxConnectionTime     time.Duration `json:"max_connection_time" mapstructure:"max_connection_time"`           // 60
	MaxIdleConnections    int           `json:"max_idle_connections" mapstructure:"max_idle_connections"`         // 5
	MaxOpenConnections    int           `json:"max_open_connections" mapstructure:"max_open_connections"`         // 5
	TablePrefix           string        `json:"table_prefix" mapstructure:"table_prefix"`                         // pre_users (pre)
}

CommonConfig is the common configuration fields between engines

type CustomWhereInterface 

type CustomWhereInterface interface {
	Where(query interface{}, args ...interface{})
	// contains filtered or unexported methods
}

CustomWhereInterface is an interface for the CustomWhere clauses

type DatabaseLogWrapper 

type DatabaseLogWrapper struct {
	zLogger.GormLoggerInterface
}

DatabaseLogWrapper is a special wrapper for the GORM logger

func (*DatabaseLogWrapper) LogMode 

func (d *DatabaseLogWrapper) LogMode(level gLogger.LogLevel) gLogger.Interface

LogMode will set the log level/mode

type Engine 

type Engine string

Engine is the different engines that are supported (database) 

const (
	Empty      Engine = "empty"
	MongoDB    Engine = "mongodb"
	MySQL      Engine = "mysql"
	PostgreSQL Engine = "postgresql"
	SQLite     Engine = "sqlite"
)

Supported engines (databases)

func (Engine) IsEmpty 

func (e Engine) IsEmpty() bool

IsEmpty will return true if the datastore is not set

func (Engine) String 

func (e Engine) String() string

String is the string version of engine

type GetterInterface 

type GetterInterface interface {
	GetArrayFields() []string
	GetDatabaseName() string
	GetMongoCollection(collectionName string) *mongo.Collection
	GetMongoCollectionByTableName(tableName string) *mongo.Collection
	GetMongoConditionProcessor() func(conditions *map[string]interface{})
	GetMongoIndexer() func() map[string][]mongo.IndexModel
	GetObjectFields() []string
	GetTableName(modelName string) string
}

GetterInterface is the getter methods

type MongoDBConfig 

type MongoDBConfig struct {
	CommonConfig       `json:",inline" mapstructure:",squash"` // Common configuration
	DatabaseName       string                                  `json:"database_name" mapstructure:"database_name"` // The database name
	ExistingConnection *mongo.Database                         `json:"-" mapstructure:"-"`                         // Used for existing database connection
	Transactions       bool                                    `json:"transactions" mapstructure:"transactions"`   // If it has transactions
	URI                string                                  `json:"uri" mapstructure:"uri"`                     // The connection string URI
}

MongoDBConfig is the configuration for each MongoDB connection

type QueryParams 

type QueryParams struct {
	Page          int    `json:"page,omitempty"`
	PageSize      int    `json:"page_size,omitempty"`
	OrderByField  string `json:"order_by_field,omitempty"`
	SortDirection string `json:"sort_direction,omitempty"`
}

QueryParams object to use when limiting and sorting database query results

func UnmarshalQueryParams 

func UnmarshalQueryParams(v interface{}) (QueryParams, error)

UnmarshalQueryParams will unmarshal the custom type

type SQLConfig 

type SQLConfig struct {
	CommonConfig              `json:",inline" mapstructure:",squash"` // Common configuration
	Driver                    string                                  `json:"driver" mapstructure:"driver"`                                             // mysql or postgresql
	ExistingConnection        *sql.DB                                 `json:"-" mapstructure:"-"`                                                       // Used for existing database connection
	Host                      string                                  `json:"host" mapstructure:"host"`                                                 // database host IE: localhost
	Name                      string                                  `json:"name" mapstructure:"name"`                                                 // database-name
	Password                  string                                  `json:"password" mapstructure:"password" encrypted:"true"`                        // user-password
	Port                      string                                  `json:"port" mapstructure:"port"`                                                 // 3306
	Replica                   bool                                    `json:"replica" mapstructure:"replica"`                                           // True if it's a replica (Read-Only)
	SkipInitializeWithVersion bool                                    `json:"skip_initialize_with_version" mapstructure:"skip_initialize_with_version"` // Skip using MySQL in test mode
	TimeZone                  string                                  `json:"time_zone" mapstructure:"time_zone"`                                       // timezone (IE: Asia/Shanghai)
	TxTimeout                 time.Duration                           `json:"tx_timeout" mapstructure:"tx_timeout"`                                     // 5*time.Second
	User                      string                                  `json:"user" mapstructure:"user"`                                                 // database username
}

SQLConfig is the configuration for each SQL connection (mysql or postgresql)

type SQLiteConfig 

type SQLiteConfig struct {
	CommonConfig       `json:",inline" mapstructure:",squash"` // Common configuration
	DatabasePath       string                                  `json:"database_path" mapstructure:"database_path"` // Location of a permanent database file (if NOT set, uses temporary memory)
	ExistingConnection gorm.ConnPool                           `json:"-" mapstructure:"-"`                         // Used for existing database connection
	Shared             bool                                    `json:"shared" mapstructure:"shared"`               // Adds a shared param to the connection string
}

SQLiteConfig is the configuration for each SQLite connection

type StorageService 

type StorageService interface {
	AutoMigrateDatabase(ctx context.Context, models ...interface{}) error
	CreateInBatches(ctx context.Context, models interface{}, batchSize int) error
	CustomWhere(tx CustomWhereInterface, conditions map[string]interface{}, engine Engine) interface{}
	Execute(query string) *gorm.DB
	GetModel(ctx context.Context, model interface{}, conditions map[string]interface{},
		timeout time.Duration, forceWriteDB bool) error
	GetModels(ctx context.Context, models interface{}, conditions map[string]interface{}, queryParams *QueryParams,
		fieldResults interface{}, timeout time.Duration) error
	GetModelCount(ctx context.Context, model interface{}, conditions map[string]interface{},
		timeout time.Duration) (int64, error)
	GetModelsAggregate(ctx context.Context, models interface{}, conditions map[string]interface{},
		aggregateColumn string, timeout time.Duration) (map[string]interface{}, error)
	HasMigratedModel(modelType string) bool
	IncrementModel(ctx context.Context, model interface{},
		fieldName string, increment int64) (newValue int64, err error)
	IndexExists(tableName, indexName string) (bool, error)
	IndexMetadata(tableName, field string) error
	NewTx(ctx context.Context, fn func(*Transaction) error) error
	Raw(query string) *gorm.DB
	SaveModel(ctx context.Context, model interface{}, tx *Transaction, newRecord, commitTx bool) error
}

StorageService is the storage related methods

type Transaction 

type Transaction struct {
	// contains filtered or unexported fields
}

Transaction is the internal datastore transaction

func (*Transaction) CanCommit 

func (tx *Transaction) CanCommit() bool

CanCommit will return true if it can commit

func (*Transaction) Commit 

func (tx *Transaction) Commit() error

Commit will commit the transaction

func (*Transaction) Rollback 

func (tx *Transaction) Rollback() error

Rollback the transaction

Source Files

View all Source files

Directories

Path Synopsis
Package customTypes encapsulates our custom database field types
 Package customTypes encapsulates our custom database field types
Package nrgorm integrates New Relic into the GORM database layer
 Package nrgorm integrates New Relic into the GORM database layer

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.