README ¶
go-datastore
Data layer using GORM for accessing models via a Database (MySQL, Postgres, SQLite, MongoDB)
Table of Contents
Installation
go-datastore requires a supported release of Go.
go get -u github.com/mrz1836/go-datastore
Documentation
View the generated documentation
Database model assumptions:
id
If a unique ID is needed, the datastore expectsid
and uses_id
internally for Mongometadata
is an optional field for storingkey->value
JSON data
Custom array and object fields:
WithCustomFields(arrayFields, objectFields )
Use this method to add custom array or object fields (IE: metadata)
Custom methods for Mongo:
WithCustomMongoConditionProcessor()
Use this method to add custom condition processing for custom object fieldsWithCustomMongoIndexer()
Use this method to add custom mongo indexes
Repository Features
This repository was created using MrZ's go-template
Built-in Features
- Continuous integration via GitHub Actions
- Build automation via Make
- Dependency management using Go Modules
- Code formatting using gofumpt and linting with golangci-lint and yamllint
- Unit testing with testify, race detector, code coverage HTML report and Codecov report
- Releasing using GoReleaser on new Tag
- Dependency scanning and updating thanks to Dependabot and Nancy
- Security code analysis using CodeQL Action
- Automatic syndication to pkg.go.dev on every release
- Generic templates for Issues and Pull Requests in GitHub
- All standard GitHub files such as
LICENSE
,CONTRIBUTING.md
,CODE_OF_CONDUCT.md
, andSECURITY.md
- Code ownership configuration for GitHub
- All your ignore files for vs-code, docker and git
- Automatic sync for labels into GitHub using a predefined configuration
- Built-in powerful merging rules using Mergify
- Welcome new contributors on their first Pull-Request
- Follows the standard-readme specification
- Visual Studio Code configuration with Go
- (Optional) Slack, Discord or Twitter announcements on new GitHub Releases
- (Optional) Easily add contributors in any Issue or Pull-Request
Package Dependencies
Library Deployment
Releases are automatically created when you create a new git tag!
If you want to manually make releases, please install GoReleaser:
goreleaser for easy binary or library deployment to GitHub and can be installed:
- using make:
make install-releaser
- using brew:
brew install goreleaser
The .goreleaser.yml file is used to configure goreleaser.
Automatic Releases on Tag Creation (recommended)
Automatic releases via GitHub Actions from creating a new tag:
make tag version=1.2.3
Manual Releases (optional)
Use make release-snap
to create a snapshot version of the release, and finally make release
to ship to production (manually).
Makefile Commands
View all makefile
commands
make help
List of all current commands:
all Runs multiple commands
clean Remove previous builds and any cached data
clean-mods Remove all the Go mod cache
coverage Shows the test coverage
diff Show the git diff
generate Runs the go generate command in the base of the repo
godocs Sync the latest tag with GoDocs
help Show this help message
install Install the application
install-all-contributors Installs all contributors locally
install-go Install the application (Using Native Go)
install-releaser Install the GoReleaser application
lint Run the golangci-lint application (install if not found)
release Full production release (creates release in GitHub)
release Runs common.release then runs godocs
release-snap Test the full release (build binaries)
release-test Full production test release (everything except deploy)
replace-version Replaces the version in HTML/JS (pre-deploy)
tag Generate a new tag and push (tag version=0.0.0)
tag-remove Remove a tag if found (tag-remove version=0.0.0)
tag-update Update an existing tag to current commit (tag-update version=0.0.0)
test Runs lint and ALL tests
test-ci Runs all tests via CI (exports coverage)
test-ci-no-race Runs all tests via CI (no race) (exports coverage)
test-ci-short Runs unit tests via CI (exports coverage)
test-no-lint Runs just tests
test-short Runs vet, lint and tests (excludes integration tests)
test-unit Runs tests and outputs coverage
uninstall Uninstall the application (and remove files)
update-contributors Regenerates the contributors html/list
update-linter Update the golangci-lint package (macOS only)
vet Run the Go vet application
Examples & Tests
All unit tests and examples run via GitHub Actions and uses Go version 1.18.x. View the configuration file.
Run all tests (including integration tests)
make test
Run tests (excluding integration tests)
make test-short
Benchmarks
Run the Go benchmarks:
make bench
Code Standards
Read more about this Go project's code standards.
Usage
Checkout all the examples!
Contributing
View the contributing guidelines and follow the code of conduct.
How can I help?
All kinds of contributions are welcome 🙌! The most basic way to show your support is to star 🌟 the project, or to raise issues 💬. You can also support this project by becoming a sponsor on GitHub 👏 or by making a bitcoin donation to ensure this journey continues indefinitely! 🚀
Contributors ✨
Thank you to these wonderful people (emoji key):
Mr. Z 🚇 💻 🚧 🛡️ |
Siggi 🚇 💻 🛡️ |
This project follows the all-contributors specification.
License
Documentation ¶
Overview ¶
Package datastore is the database service abstraction layer
Index ¶
- Constants
- Variables
- func GetModelBoolAttribute(model interface{}, attribute string) *bool
- func GetModelName(model interface{}) *string
- func GetModelStringAttribute(model interface{}, attribute string) *string
- func GetModelTableName(model interface{}) *string
- func GetModelType(model interface{}) reflect.Type
- func GetModelUnset(model interface{}) map[string]bool
- func IsModelSlice(model interface{}) bool
- func IsSQLEngine(e Engine) bool
- func MarshalQueryParams(m QueryParams) graphql.Marshaler
- func StringInSlice(a string, list []string) bool
- type Client
- func (c *Client) AutoMigrateDatabase(ctx context.Context, models ...interface{}) error
- func (c *Client) Close(ctx context.Context) error
- func (c *Client) CreateInBatches(ctx context.Context, models interface{}, batchSize int) error
- func (c *Client) CreateInBatchesMongo(ctx context.Context, models interface{}, batchSize int) error
- func (c *Client) CustomWhere(tx CustomWhereInterface, conditions map[string]interface{}, engine Engine) interface{}
- func (c *Client) Debug(on bool)
- func (c *Client) DebugLog(ctx context.Context, text string)
- func (c *Client) Engine() Engine
- func (c *Client) Execute(query string) *gorm.DB
- func (c *Client) GetArrayFields() []string
- func (c *Client) GetDatabaseName() string
- func (c *Client) GetModel(ctx context.Context, model interface{}, conditions map[string]interface{}, ...) error
- func (c *Client) GetModelCount(ctx context.Context, model interface{}, conditions map[string]interface{}, ...) (int64, error)
- func (c *Client) GetModels(ctx context.Context, models interface{}, conditions map[string]interface{}, ...) error
- func (c *Client) GetModelsAggregate(ctx context.Context, models interface{}, conditions map[string]interface{}, ...) (map[string]interface{}, error)
- func (c *Client) GetMongoCollection(collectionName string) *mongo.Collection
- func (c *Client) GetMongoCollectionByTableName(tableName string) *mongo.Collection
- func (c *Client) GetMongoConditionProcessor() func(conditions *map[string]interface{})
- func (c *Client) GetMongoIndexer() func() map[string][]mongo.IndexModel
- func (c *Client) GetObjectFields() []string
- func (c *Client) GetTableName(modelName string) string
- func (c *Client) HasMigratedModel(modelType string) bool
- func (c *Client) IncrementModel(ctx context.Context, model interface{}, fieldName string, increment int64) (newValue int64, err error)
- func (c *Client) IndexExists(tableName, indexName string) (bool, error)
- func (c *Client) IndexMetadata(tableName, field string) error
- func (c *Client) IsAutoMigrate() bool
- func (c *Client) IsDebug() bool
- func (c *Client) IsNewRelicEnabled() bool
- func (c *Client) NewRawTx() (*Transaction, error)
- func (c *Client) NewTx(ctx context.Context, fn func(*Transaction) error) error
- func (c *Client) Raw(query string) *gorm.DB
- func (c *Client) SaveModel(ctx context.Context, model interface{}, tx *Transaction, ...) error
- type ClientInterface
- type ClientOps
- func WithAutoMigrate(migrateModels ...interface{}) ClientOps
- func WithCustomFields(arrayFields []string, objectFields []string) ClientOps
- func WithCustomMongoConditionProcessor(f func(conditions *map[string]interface{})) ClientOps
- func WithCustomMongoIndexer(f func() map[string][]mongo.IndexModel) ClientOps
- func WithDebugging() ClientOps
- func WithLogger(customLogger zLogger.GormLoggerInterface) ClientOps
- func WithMongo(config *MongoDBConfig) ClientOps
- func WithMongoConnection(database *mongo.Database, tablePrefix string) ClientOps
- func WithNewRelic() ClientOps
- func WithSQL(engine Engine, configs []*SQLConfig) ClientOps
- func WithSQLConnection(engine Engine, sqlDB *sql.DB, tablePrefix string) ClientOps
- func WithSQLite(config *SQLiteConfig) ClientOps
- type CommonConfig
- type CustomWhereInterface
- type DatabaseLogWrapper
- type Engine
- type GetterInterface
- type MongoDBConfig
- type QueryParams
- type SQLConfig
- type SQLiteConfig
- type StorageService
- type Transaction
Constants ¶
const ( // SortDesc will sort descending SortDesc = "desc" // SortAsc will sort ascending SortAsc = "asc" )
Defaults for library functionality
const ( Postgres = "postgres" JSON = "JSON" JSONB = "JSONB" )
index creation constants
Variables ¶
var ( // DateFields are standard known date fields DateFields = []string{dateCreatedAt, dateUpdatedAt, dateModifiedAt} )
var ErrDuplicateKey = errors.New("duplicate key")
ErrDuplicateKey error when a record is inserted and conflicts with an existing record
var ErrNoResults = errors.New("no results found")
ErrNoResults error when no results are found
var ErrNoSourceFound = errors.New("no source database found in all given configurations")
ErrNoSourceFound is when no source database is found in all given configurations
var ErrNotImplemented = errors.New("not implemented")
ErrNotImplemented is an error when a method is not implemented
var ErrUnknownCollection = errors.New("could not determine collection name from model")
ErrUnknownCollection is thrown when the collection can not be found using the model/name
var ErrUnknownSQL = errors.New("unknown sql implementation")
ErrUnknownSQL is an error when using a SQL engine that is not known for indexes and migrations
var ErrUnsupportedDriver = errors.New("sql driver unsupported")
ErrUnsupportedDriver is when the given SQL driver is not determined to be known or supported
var ErrUnsupportedEngine = errors.New("unsupported datastore engine")
ErrUnsupportedEngine is used when the engine given is not a known datastore engine
var SQLDatabases = []Engine{ MySQL, PostgreSQL, SQLite, }
SQLDatabases is the list of supported SQL databases (via GORM)
Functions ¶
func GetModelBoolAttribute ¶
GetModelBoolAttribute the attribute from the model as a bool
func GetModelName ¶
func GetModelName(model interface{}) *string
GetModelName get the name of the model via reflection
func GetModelStringAttribute ¶
GetModelStringAttribute the attribute from the model as a string
func GetModelTableName ¶
func GetModelTableName(model interface{}) *string
GetModelTableName get the db table name of the model via reflection
func GetModelType ¶
GetModelType get the model type of the model interface via reflection
func GetModelUnset ¶
GetModelUnset gets any empty values on the model and makes sure the update actually unsets those values in the database, otherwise this never happens, and we cannot unset
func IsModelSlice ¶
func IsModelSlice(model interface{}) bool
IsModelSlice returns true if the given interface is a slice of models
func IsSQLEngine ¶
IsSQLEngine check whether the string already is in the slice
func MarshalQueryParams ¶
func MarshalQueryParams(m QueryParams) graphql.Marshaler
MarshalQueryParams will marshal the custom type
func StringInSlice ¶
StringInSlice check whether the string already is in the slice
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is the datastore client (configuration)
func (*Client) AutoMigrateDatabase ¶
AutoMigrateDatabase will detect the engine and migrate as needed
func (*Client) CreateInBatches ¶
CreateInBatches create all the models given in batches
func (*Client) CreateInBatchesMongo ¶
func (c *Client) CreateInBatchesMongo( ctx context.Context, models interface{}, batchSize int, ) error
CreateInBatchesMongo insert multiple models vai bulk.Write
func (*Client) CustomWhere ¶
func (c *Client) CustomWhere(tx CustomWhereInterface, conditions map[string]interface{}, engine Engine) interface{}
CustomWhere add conditions
func (*Client) GetArrayFields ¶
GetArrayFields will return the array fields
func (*Client) GetDatabaseName ¶
GetDatabaseName will return the full database name for the given model name
func (*Client) GetModel ¶
func (c *Client) GetModel( ctx context.Context, model interface{}, conditions map[string]interface{}, timeout time.Duration, forceWriteDB bool, ) error
GetModel will get a model from the datastore
func (*Client) GetModelCount ¶
func (c *Client) GetModelCount( ctx context.Context, model interface{}, conditions map[string]interface{}, timeout time.Duration, ) (int64, error)
GetModelCount will return a count of the model matching conditions
func (*Client) GetModels ¶
func (c *Client) GetModels( ctx context.Context, models interface{}, conditions map[string]interface{}, queryParams *QueryParams, fieldResults interface{}, timeout time.Duration, ) error
GetModels will return a slice of models based on the given conditions
func (*Client) GetModelsAggregate ¶
func (c *Client) GetModelsAggregate(ctx context.Context, models interface{}, conditions map[string]interface{}, aggregateColumn string, timeout time.Duration) (map[string]interface{}, error)
GetModelsAggregate will return an aggregate count of the model matching conditions
func (*Client) GetMongoCollection ¶
func (c *Client) GetMongoCollection( collectionName string, ) *mongo.Collection
GetMongoCollection will get the mongo collection for the given tableName
func (*Client) GetMongoCollectionByTableName ¶
func (c *Client) GetMongoCollectionByTableName( tableName string, ) *mongo.Collection
GetMongoCollectionByTableName will get the mongo collection for the given tableName
func (*Client) GetMongoConditionProcessor ¶
GetMongoConditionProcessor will return a custom mongo condition processor if set
func (*Client) GetMongoIndexer ¶
func (c *Client) GetMongoIndexer() func() map[string][]mongo.IndexModel
GetMongoIndexer will return a custom mongo condition indexer
func (*Client) GetObjectFields ¶
GetObjectFields will return the object fields
func (*Client) GetTableName ¶
GetTableName will return the full table name for the given model name
func (*Client) HasMigratedModel ¶
HasMigratedModel will return if the model type has been migrated
func (*Client) IncrementModel ¶
func (c *Client) IncrementModel( ctx context.Context, model interface{}, fieldName string, increment int64, ) (newValue int64, err error)
IncrementModel will increment the given field atomically in the database and return the new value
func (*Client) IndexExists ¶
IndexExists check whether the given index exists in the datastore
func (*Client) IndexMetadata ¶
IndexMetadata check and creates the metadata json index
func (*Client) IsAutoMigrate ¶
IsAutoMigrate returns whether auto migration is on
func (*Client) IsNewRelicEnabled ¶
IsNewRelicEnabled will return if new relic is enabled
func (*Client) NewRawTx ¶ added in v0.4.0
func (c *Client) NewRawTx() (*Transaction, error)
NewRawTx will start a new datastore transaction
type ClientInterface ¶
type ClientInterface interface { GetterInterface StorageService Close(ctx context.Context) error Debug(on bool) DebugLog(ctx context.Context, text string) Engine() Engine IsAutoMigrate() bool IsDebug() bool IsNewRelicEnabled() bool }
ClientInterface is the Datastore client interface
type ClientOps ¶
type ClientOps func(c *clientOptions)
ClientOps allow functional options to be supplied that overwrite default client options.
func WithAutoMigrate ¶
func WithAutoMigrate(migrateModels ...interface{}) ClientOps
WithAutoMigrate will enable auto migrate database mode (given models)
Pointers of structs (IE: &models.Xpub{})
func WithCustomFields ¶
WithCustomFields will add custom fields to the datastore
func WithCustomMongoConditionProcessor ¶
WithCustomMongoConditionProcessor will add a custom mongo condition processor function
func WithCustomMongoIndexer ¶
func WithCustomMongoIndexer(f func() map[string][]mongo.IndexModel) ClientOps
WithCustomMongoIndexer will add a custom mongo index function (returns custom mongo indexes)
func WithLogger ¶
func WithLogger(customLogger zLogger.GormLoggerInterface) ClientOps
WithLogger will set the custom logger interface
func WithMongo ¶
func WithMongo(config *MongoDBConfig) ClientOps
WithMongo will set the datastore to use MongoDB
func WithMongoConnection ¶
WithMongoConnection will set the datastore to use an existing Mongo database connection
func WithSQL ¶
WithSQL will load a datastore using either an SQL database config or existing connection
func WithSQLConnection ¶
WithSQLConnection will set the datastore to an existing connection for MySQL or PostgreSQL
func WithSQLite ¶
func WithSQLite(config *SQLiteConfig) ClientOps
WithSQLite will set the datastore to use SQLite
type CommonConfig ¶
type CommonConfig struct { Debug bool `json:"debug" mapstructure:"debug"` // flag for debugging sql queries in logs MaxConnectionIdleTime time.Duration `json:"max_connection_idle_time" mapstructure:"max_connection_idle_time"` // 360 MaxConnectionTime time.Duration `json:"max_connection_time" mapstructure:"max_connection_time"` // 60 MaxIdleConnections int `json:"max_idle_connections" mapstructure:"max_idle_connections"` // 5 MaxOpenConnections int `json:"max_open_connections" mapstructure:"max_open_connections"` // 5 TablePrefix string `json:"table_prefix" mapstructure:"table_prefix"` // pre_users (pre) }
CommonConfig is the common configuration fields between engines
type CustomWhereInterface ¶
type CustomWhereInterface interface { Where(query interface{}, args ...interface{}) // contains filtered or unexported methods }
CustomWhereInterface is an interface for the CustomWhere clauses
type DatabaseLogWrapper ¶
type DatabaseLogWrapper struct {
zLogger.GormLoggerInterface
}
DatabaseLogWrapper is a special wrapper for the GORM logger
type Engine ¶
type Engine string
Engine is the different engines that are supported (database)
const ( Empty Engine = "empty" MongoDB Engine = "mongodb" MySQL Engine = "mysql" PostgreSQL Engine = "postgresql" SQLite Engine = "sqlite" )
Supported engines (databases)
type GetterInterface ¶
type GetterInterface interface { GetArrayFields() []string GetDatabaseName() string GetMongoCollection(collectionName string) *mongo.Collection GetMongoCollectionByTableName(tableName string) *mongo.Collection GetMongoConditionProcessor() func(conditions *map[string]interface{}) GetMongoIndexer() func() map[string][]mongo.IndexModel GetObjectFields() []string GetTableName(modelName string) string }
GetterInterface is the getter methods
type MongoDBConfig ¶
type MongoDBConfig struct { CommonConfig `json:",inline" mapstructure:",squash"` // Common configuration DatabaseName string `json:"database_name" mapstructure:"database_name"` // The database name ExistingConnection *mongo.Database `json:"-" mapstructure:"-"` // Used for existing database connection Transactions bool `json:"transactions" mapstructure:"transactions"` // If it has transactions URI string `json:"uri" mapstructure:"uri"` // The connection string URI }
MongoDBConfig is the configuration for each MongoDB connection
type QueryParams ¶
type QueryParams struct { Page int `json:"page,omitempty"` PageSize int `json:"page_size,omitempty"` OrderByField string `json:"order_by_field,omitempty"` SortDirection string `json:"sort_direction,omitempty"` }
QueryParams object to use when limiting and sorting database query results
func UnmarshalQueryParams ¶
func UnmarshalQueryParams(v interface{}) (QueryParams, error)
UnmarshalQueryParams will unmarshal the custom type
type SQLConfig ¶
type SQLConfig struct { CommonConfig `json:",inline" mapstructure:",squash"` // Common configuration Driver string `json:"driver" mapstructure:"driver"` // mysql or postgresql ExistingConnection *sql.DB `json:"-" mapstructure:"-"` // Used for existing database connection Host string `json:"host" mapstructure:"host"` // database host IE: localhost Name string `json:"name" mapstructure:"name"` // database-name Password string `json:"password" mapstructure:"password" encrypted:"true"` // user-password Port string `json:"port" mapstructure:"port"` // 3306 Replica bool `json:"replica" mapstructure:"replica"` // True if it's a replica (Read-Only) SkipInitializeWithVersion bool `json:"skip_initialize_with_version" mapstructure:"skip_initialize_with_version"` // Skip using MySQL in test mode TimeZone string `json:"time_zone" mapstructure:"time_zone"` // timezone (IE: Asia/Shanghai) TxTimeout time.Duration `json:"tx_timeout" mapstructure:"tx_timeout"` // 5*time.Second User string `json:"user" mapstructure:"user"` // database username }
SQLConfig is the configuration for each SQL connection (mysql or postgresql)
type SQLiteConfig ¶
type SQLiteConfig struct { CommonConfig `json:",inline" mapstructure:",squash"` // Common configuration DatabasePath string `json:"database_path" mapstructure:"database_path"` // Location of a permanent database file (if NOT set, uses temporary memory) ExistingConnection gorm.ConnPool `json:"-" mapstructure:"-"` // Used for existing database connection }
SQLiteConfig is the configuration for each SQLite connection
type StorageService ¶
type StorageService interface { AutoMigrateDatabase(ctx context.Context, models ...interface{}) error CreateInBatches(ctx context.Context, models interface{}, batchSize int) error CustomWhere(tx CustomWhereInterface, conditions map[string]interface{}, engine Engine) interface{} Execute(query string) *gorm.DB GetModel(ctx context.Context, model interface{}, conditions map[string]interface{}, timeout time.Duration, forceWriteDB bool) error GetModels(ctx context.Context, models interface{}, conditions map[string]interface{}, queryParams *QueryParams, fieldResults interface{}, timeout time.Duration) error GetModelCount(ctx context.Context, model interface{}, conditions map[string]interface{}, timeout time.Duration) (int64, error) GetModelsAggregate(ctx context.Context, models interface{}, conditions map[string]interface{}, aggregateColumn string, timeout time.Duration) (map[string]interface{}, error) HasMigratedModel(modelType string) bool IncrementModel(ctx context.Context, model interface{}, fieldName string, increment int64) (newValue int64, err error) IndexExists(tableName, indexName string) (bool, error) IndexMetadata(tableName, field string) error NewTx(ctx context.Context, fn func(*Transaction) error) error NewRawTx() (*Transaction, error) Raw(query string) *gorm.DB SaveModel(ctx context.Context, model interface{}, tx *Transaction, newRecord, commitTx bool) error }
StorageService is the storage related methods
type Transaction ¶
type Transaction struct {
// contains filtered or unexported fields
}
Transaction is the internal datastore transaction
func (*Transaction) CanCommit ¶
func (tx *Transaction) CanCommit() bool
CanCommit will return true if it can commit
func (*Transaction) Commit ¶
func (tx *Transaction) Commit() error
Commit will commit the transaction
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
Package customtypes encapsulates our custom database field types
|
Package customtypes encapsulates our custom database field types |
Package nrgorm integrates New Relic into the GORM database layer
|
Package nrgorm integrates New Relic into the GORM database layer |