datasource

README

Datasource

GoFr provides following features to ensure robust and observable interactions with various data sources:

1. Health Checks

A mechanism for a datasource to self-report its operational status. New datasources require implementing the HealthCheck() method with the signature:

HealthCheck() datasource.Health

This method should return the current health status of the datasource.

1. Retry Mechanism

GoFr attempts to re-establish connections if lost during application runtime. New datasources should be verified for built-in retry mechanisms. If absent, implement a mechanism for automatic reconnection.

1. Metrics

Datasources should expose relevant metrics for performance monitoring. The specific metrics to be implemented depend on the datasource type. Discussions are required to determine the appropriate metrics for each new datasource.

1. Logging

GoFr supports level-based logging with the PrettyPrint interface. New datasources should implement logging with the following levels:

• DEBUG: Logs connection attempts with critical details.
• INFO: Logs successful connection establishment.
• WARN: Logs connection retrying

Additional logs can be added to enhance debugging and improving user experience.

1. Tracing

GoFr supports tracing for all the datasources, for example for SQL it traces the request using https://github.com/XSAM/otelsql. If any official package or any widely used package is not available we have to implement our own, but in scope of a different ISSUE.

All logs should include:

• Timestamp
• Request ID (Correlation ID)
• Time taken to execute the query
• Datasource name (consistent with other logs)

Implementing New Datasources

GoFr offers built-in support for popular datasources like SQL (MySQL, PostgreSQL, SQLite), Redis, and Pub/Sub (MQTT, Kafka, Google as backend). Including additional functionalities within the core GoFr binary would increase the application size unnecessarily.

Therefore, GoFr utilizes a pluggable approach for new datasources by separating implementation in the following way:

• Interface Definition:

  Create an interface with required methods within the datasource package. Register the interface with the container (similar to MongoDB in https://github.com/tfogo/mongodb-go-tutorial).

• Method Registration:

  Create a method in gofr.go (similar to the existing one) that accepts the newly defined interface.

• Separate Repository:

  Develop a separate repository to implement the interface for the new datasource. This approach ensures that the new datasource dependency is only loaded when utilized, minimizing binary size for GoFr applications. It also empowers users to create custom implementations beyond the defaults provided by GoFr.

Supported Datasources

Datasource	Health-Check	Logs	Metrics	Traces	As Driver
MySQL	✅	✅	✅	✅
Redis	✅	✅	✅	✅
PostgreSQL	✅	✅	✅	✅
MongoDB	✅	✅	✅	✅	✅
SQLite	✅	✅	✅	✅
BadgerDB	✅	✅	✅	✅	✅
Cassandra	✅	✅	✅	✅	✅
ClickHouse		✅	✅	✅	✅
FTP		✅			✅
SFTP		✅			✅
Solr		✅	✅	✅	✅
Dgraph	✅	✅	✅	✅
Azure Event Hub		✅	✅		✅
OpenTSDB	✅	✅		✅	✅

Documentation

Index

• Constants
• Variables
• type Datasource
• type ErrorDB
  • func (e ErrorDB) Error() string
  • func (ErrorDB) StatusCode() int
  • func (e ErrorDB) WithStack() ErrorDB
• type File
• type FileSystem
• type FileSystemProvider
• type Health
• type Logger
• type RowReader

Constants

const (
	StatusUp   = "UP"
	StatusDown = "DOWN"
)

Variables

var (
	ErrFileClosed        = errors.New("File is closed")
	ErrOutOfRange        = errors.New("out of range")
	ErrTooLarge          = errors.New("too large")
	ErrFileNotFound      = os.ErrNotExist
	ErrFileExists        = os.ErrExist
	ErrDestinationExists = os.ErrExist
)

Types

type Datasource

type Datasource interface {
	Register(config config.Config)
}

type ErrorDB

type ErrorDB struct {
	Err     error
	Message string
}

ErrorDB represents an error specific to database operations.

func (ErrorDB) Error

func (e ErrorDB) Error() string

func (ErrorDB) StatusCode

func (ErrorDB) StatusCode() int

func (ErrorDB) WithStack

func (e ErrorDB) WithStack() ErrorDB

WithStack adds a stack trace to the Error.

type File

type File interface {
	io.Closer
	io.Reader
	io.ReaderAt
	io.Seeker
	io.Writer
	io.WriterAt

	ReadAll() (RowReader, error)
}

File represents a file in the filesystem.

type FileSystem

type FileSystem interface {
	// Create creates a file in the filesystem, returning the file and an
	// error, if any happens.
	Create(name string) (File, error)

	// Mkdir creates a directory in the filesystem, return an error if any
	// happens.
	Mkdir(name string, perm os.FileMode) error

	// MkdirAll creates a directory path and all parents that does not exist
	// yet.
	MkdirAll(path string, perm os.FileMode) error

	// Open opens a file, returning it or an error, if any happens.
	Open(name string) (File, error)

	// OpenFile opens a file using the given flags and the given mode.
	OpenFile(name string, flag int, perm os.FileMode) (File, error)

	// Remove removes a file identified by name, returning an error, if any
	// happens.
	Remove(name string) error

	// RemoveAll removes a directory path and any children it contains. It
	// does not fail if the path does not exist (return nil).
	RemoveAll(path string) error

	// Rename renames a file.
	Rename(oldname, newname string) error
}

FileSystem : Any simulated or real filesystem should implement this interface.

type FileSystemProvider

type FileSystemProvider interface {
	FileSystem

	// UseLogger sets the logger for the FileSystem client.
	UseLogger(logger interface{})

	// UseMetrics sets the metrics for the FileSystem client.
	UseMetrics(metrics interface{})

	// Connect establishes a connection to FileSystem and registers metrics using the provided configuration when the client was Created.
	Connect()
}

type Health

type Health struct {
	Status  string                 `json:"status,omitempty"`
	Details map[string]interface{} `json:"details,omitempty"`
}

type Logger

type Logger interface {
	Debug(args ...interface{})
	Debugf(format string, args ...interface{})
	Info(args ...interface{})
	Infof(format string, args ...interface{})
	Error(args ...interface{})
	Errorf(format string, args ...interface{})
	Warn(args ...interface{})
	Warnf(format string, args ...interface{})
}

Logger interface is used by datasource packages to log information about query execution. Developer Notes: Note that it's a reduced version of logging.Logger interface. We are not using that package to ensure that datasource package is not dependent on logging package. That way logging package should be easily able to import datasource package and provide a different "pretty" version for different log types defined here while avoiding the cyclical import issue. Idiomatically, interfaces should be defined by packages who are using it; unlike other languages. Also - accept interfaces, return concrete types.

type RowReader

type RowReader interface {
	Next() bool
	Scan(interface{}) error
}