datasource

README

Datasource

GoFr provides following features to ensure robust and observable interactions with various data sources:

1. Health Checks

A mechanism for a datasource to self-report its operational status. New datasources require implementing the HealthCheck() method with the signature:

HealthCheck() datasource.Health

This method should return the current health status of the datasource.

1. Retry Mechanism

GoFr attempts to re-establish connections if lost during application runtime. New datasources should be verified for built-in retry mechanisms. If absent, implement a mechanism for automatic reconnection.

1. Metrics

Datasources should expose relevant metrics for performance monitoring. The specific metrics to be implemented depend on the datasource type. Discussions are required to determine the appropriate metrics for each new datasource.

1. Logging

GoFr supports level-based logging with the PrettyPrint interface. New datasources should implement logging with the following levels:

• DEBUG: Logs connection attempts with critical details.
• INFO: Logs successful connection establishment.
• WARN: Logs connection retrying

Additional logs can be added to enhance debugging and improving user experience.

1. Tracing

GoFr supports tracing for all the datasouces, for example for SQL it traces the request using github.com/XSAM/otelsql. If any official package or any widely used package is not available we have to implement our own, but in scope of a different ISSUE.

All logs should include:

• Timestamp
• Request ID (Correlation ID)
• Time taken to execute the query
• Datasource name (consistent with other logs)

Implementing New Datasources

GoFr offers built-in support for popular datasources like SQL (MySQL, PostgreSQL, SQLite), Redis, and Pub/Sub (MQTT, Kafka, Google as backend). Including additional functionalities within the core GoFr binary would increase the application size unnecessarily.

Therefore, GoFr utilizes a pluggable approach for new datasources by separating implementation in the following way:

• Interface Definition:

  Create an interface with required methods within the datasource package. Register the interface with the container (similar to Mongo in https://github.com/tfogo/mongodb-go-tutorial).

• Method Registration:

  Create a method in gofr.go (similar to the existing one) that accepts the newly defined interface.

• Separate Repository:

  Develop a separate repository to implement the interface for the new datasource. This approach ensures that the new datasource dependency is only loaded when utilized, minimizing binary size for GoFr applications. It also empowers users to create custom implementations beyond the defaults provided by GoFr.

Supported Datasources

Datasource	Health-Check	Logs	Metrics	Traces	As Driver
MySQL	✅	✅	✅	✅
REDIS	✅	✅	✅	✅
PostgreSQL	✅	✅	✅	✅
MongoDB	✅	✅	✅		✅
SQLite	✅	✅	✅	✅
Cassandra	✅	✅	✅		✅

Documentation

Index

• Constants
• Variables
• type Cassandra
• type CassandraProvider
• type Datasource
• type ErrorDB
  • func (e ErrorDB) Error() string
  • func (e ErrorDB) StatusCode() int
  • func (e ErrorDB) WithStack() ErrorDB
• type File
• type FileSystem
• type FileSystemProvider
• type Health
• type Logger
• type Mongo
• type MongoProvider
• type RowReader

Constants

const (
	StatusUp   = "UP"
	StatusDown = "DOWN"
)

Variables

var (
	ErrFileClosed        = errors.New("File is closed")
	ErrOutOfRange        = errors.New("out of range")
	ErrTooLarge          = errors.New("too large")
	ErrFileNotFound      = os.ErrNotExist
	ErrFileExists        = os.ErrExist
	ErrDestinationExists = os.ErrExist
)

Types

type Cassandra

type Cassandra interface {
	// Query executes the query and binds the result into dest parameter.
	// Returns error if any error occurs while binding the result.
	// Can be used to single as well as multiple rows.
	// Accepts pointer to struct or slice as dest parameter for single and multiple rows retrieval respectively.
	//
	// Example:
	//
	//	// Get multiple rows with only one column
	//	   ids := make([]int, 0)
	//	   err := c.Query(&ids, "SELECT id FROM users")
	//
	//	// Get a single object from database
	//	   type user struct {
	//	   	ID    int
	//	   	Name string
	//	   }
	//	   u := user{}
	//	   err := c.Query(&u, "SELECT * FROM users WHERE id=?", 1)
	//
	//	// Get array of objects from multiple rows
	//	   type user struct {
	//	   	ID    int
	//	   	Name string `db:"name"`
	//	   }
	//	   users := []user{}
	//	   err := c.Query(&users, "SELECT * FROM users")
	Query(dest interface{}, stmt string, values ...interface{}) error

	// Exec executes the query without returning any rows.
	// Return error if any error occurs while executing the query.
	// Can be used to execute UPDATE or INSERT.
	//
	// Example:
	//
	//	// Without values
	//	   err := c.Exec("INSERT INTO users VALUES(1, 'John Doe')")
	//
	//	// With Values
	//	   id := 1
	//	   name := "John Doe"
	//	   err := c.Exec("INSERT INTO users VALUES(?, ?)", id, name)
	Exec(stmt string, values ...interface{}) error

	// ExecCAS executes a lightweight transaction (i.e. an UPDATE or INSERT statement containing an IF clause).
	// If the transaction fails because the existing values did not match, the previous values will be stored in dest.
	// Returns true if the query is applied otherwise false.
	// Returns false and error if any error occur while executing the query.
	// Accepts only pointer to struct and built-in types as the dest parameter.
	//
	// Example:
	//
	//	type user struct {
	//		ID    int
	//		Name string
	//	}
	//	u := user{}
	//	applied, err := c.ExecCAS(&ids, "INSERT INTO users VALUES(1, 'John Doe') IF NOT EXISTS")
	ExecCAS(dest interface{}, stmt string, values ...interface{}) (bool, error)
}

type CassandraProvider

type CassandraProvider interface {
	Cassandra

	// UseLogger sets the logger for the Cassandra client.
	UseLogger(logger interface{})

	// UseMetrics sets the metrics for the Cassandra client.
	UseMetrics(metrics interface{})

	// Connect establishes a connection to Cassandra and registers metrics using the provided configuration when the client was Created.
	Connect()
}

type Datasource

type Datasource interface {
	Register(config config.Config)
}

type ErrorDB

type ErrorDB struct {
	Err     error
	Message string
}

ErrorDB represents an error specific to database operations.

func (ErrorDB) Error

func (e ErrorDB) Error() string

func (ErrorDB) StatusCode

func (e ErrorDB) StatusCode() int

func (ErrorDB) WithStack

func (e ErrorDB) WithStack() ErrorDB

WithStack adds a stack trace to the Error.

type File

type File interface {
	io.Closer
	io.Reader
	io.ReaderAt
	io.Seeker
	io.Writer
	io.WriterAt

	ReadAll() (RowReader, error)
}

File represents a file in the filesystem.

type FileSystem

type FileSystem interface {
	// Create creates a file in the filesystem, returning the file and an
	// error, if any happens.
	Create(name string) (File, error)

	// Mkdir creates a directory in the filesystem, return an error if any
	// happens.
	Mkdir(name string, perm os.FileMode) error

	// MkdirAll creates a directory path and all parents that does not exist
	// yet.
	MkdirAll(path string, perm os.FileMode) error

	// Open opens a file, returning it or an error, if any happens.
	Open(name string) (File, error)

	// OpenFile opens a file using the given flags and the given mode.
	OpenFile(name string, flag int, perm os.FileMode) (File, error)

	// Remove removes a file identified by name, returning an error, if any
	// happens.
	Remove(name string) error

	// RemoveAll removes a directory path and any children it contains. It
	// does not fail if the path does not exist (return nil).
	RemoveAll(path string) error

	// Rename renames a file.
	Rename(oldname, newname string) error
}

FileSystem : Any simulated or real filesystem should implement this interface.

type FileSystemProvider

type FileSystemProvider interface {
	FileSystem

	// UseLogger sets the logger for the FileSystem client.
	UseLogger(logger interface{})

	// UseMetrics sets the metrics for the FileSystem client.
	UseMetrics(metrics interface{})

	// Connect establishes a connection to FileSystem and registers metrics using the provided configuration when the client was Created.
	Connect()
}

type Health

type Health struct {
	Status  string                 `json:"status,omitempty"`
	Details map[string]interface{} `json:"details,omitempty"`
}

type Logger

type Logger interface {
	Debug(args ...interface{})
	Debugf(format string, args ...interface{})
	Log(args ...interface{})
	Logf(format string, args ...interface{})
	Error(args ...interface{})
	Errorf(format string, args ...interface{})
	Warn(args ...interface{})
	Warnf(format string, args ...interface{})
}

Logger interface is used by datasource packages to log information about query execution. Developer Notes: Note that it's a reduced version of logging.Logger interface. We are not using that package to ensure that datasource package is not dependent on logging package. That way logging package should be easily able to import datasource package and provide a different "pretty" version for different log types defined here while avoiding the cyclical import issue. Idiomatically, interfaces should be defined by packages who are using it; unlike other languages. Also - accept interfaces, return concrete types.

type Mongo

type Mongo interface {
	// Find executes a query to find documents in a collection based on a filter and stores the results
	// into the provided results interface.
	Find(ctx context.Context, collection string, filter interface{}, results interface{}) error

	// FindOne executes a query to find a single document in a collection based on a filter and stores the result
	// into the provided result interface.
	FindOne(ctx context.Context, collection string, filter interface{}, result interface{}) error

	// InsertOne inserts a single document into a collection.
	// It returns the identifier of the inserted document and an error, if any.
	InsertOne(ctx context.Context, collection string, document interface{}) (interface{}, error)

	// InsertMany inserts multiple documents into a collection.
	// It returns the identifiers of the inserted documents and an error, if any.
	InsertMany(ctx context.Context, collection string, documents []interface{}) ([]interface{}, error)

	// DeleteOne deletes a single document from a collection based on a filter.
	// It returns the number of documents deleted and an error, if any.
	DeleteOne(ctx context.Context, collection string, filter interface{}) (int64, error)

	// DeleteMany deletes multiple documents from a collection based on a filter.
	// It returns the number of documents deleted and an error, if any.
	DeleteMany(ctx context.Context, collection string, filter interface{}) (int64, error)

	// UpdateByID updates a document in a collection by its ID.
	// It returns the number of documents updated and an error if any.
	UpdateByID(ctx context.Context, collection string, id interface{}, update interface{}) (int64, error)

	// UpdateOne updates a single document in a collection based on a filter.
	// It returns an error if any.
	UpdateOne(ctx context.Context, collection string, filter interface{}, update interface{}) error

	// UpdateMany updates multiple documents in a collection based on a filter.
	// It returns the number of documents updated and an error if any.
	UpdateMany(ctx context.Context, collection string, filter interface{}, update interface{}) (int64, error)

	// CountDocuments counts the number of documents in a collection based on a filter.
	// It returns the count and an error if any.
	CountDocuments(ctx context.Context, collection string, filter interface{}) (int64, error)

	// Drop an entire collection from the database.
	// It returns an error if any.
	Drop(ctx context.Context, collection string) error
}

Mongo is an interface representing a MongoDB database client with common CRUD operations.

type MongoProvider

type MongoProvider interface {
	Mongo

	// UseLogger sets the logger for the MongoDB client.
	UseLogger(logger interface{})

	// UseMetrics sets the metrics for the MongoDB client.
	UseMetrics(metrics interface{})

	// Connect establishes a connection to MongoDB and registers metrics using the provided configuration when the client was Created.
	Connect()
}

MongoProvider is an interface that extends Mongo with additional methods for logging, metrics, and connection management. Which is used for initializing datasource.

type RowReader

type RowReader interface {
	Next() bool
	Scan(interface{}) error
}