database

README

CROWler supported Database Managers

Although if I am working to add support for multiple database managers, the only one supported at the moment is PostgreSQL. The database manager is used to store the data collected by the CROWler. The database manager is also used to store the sources etc. of the CROWler.

Work in progress to add support for other database managers:

• SQLite
• MySQL / MariaDB

Documentation

Overview

Package database is responsible for handling the database setup, configuration and abstraction.

Package database is responsible for handling the database setup, configuration and abstraction.

Package database is responsible for handling the database setup, configuration and abstraction.

Package database is responsible for handling the database setup, configuration and abstraction.

Package database is responsible for handling the database setup, configuration and abstraction.

Package database is responsible for handling the database setup, configuration and abstraction.

Index

• Variables
• func DBSetup(db *sql.DB, dbms, dbname string)
• type Handler
  • func NewHandler(c cfg.Config) (Handler, error)
• type PostgresHandler
  • func (handler *PostgresHandler) Begin() (*sql.Tx, error)
  • func (handler *PostgresHandler) CheckConnection(c cfg.Config) error
  • func (handler *PostgresHandler) Close() error
  • func (handler *PostgresHandler) Commit(tx *sql.Tx) error
  • func (handler *PostgresHandler) ConfigForQuery()
  • func (handler *PostgresHandler) ConfigForWrite()
  • func (handler *PostgresHandler) Connect(c cfg.Config) error
  • func (handler *PostgresHandler) DBMS() string
  • func (handler *PostgresHandler) Exec(query string, args ...interface{}) (sql.Result, error)
  • func (handler *PostgresHandler) ExecuteQuery(query string, args ...interface{}) (*sql.Rows, error)
  • func (handler *PostgresHandler) Ping() error
  • func (handler *PostgresHandler) QueryRow(query string, args ...interface{}) *sql.Row
  • func (handler *PostgresHandler) Rollback(tx *sql.Tx) error
• type SQLiteHandler
  • func (handler *SQLiteHandler) Begin() (*sql.Tx, error)
  • func (handler *SQLiteHandler) CheckConnection(c cfg.Config) error
  • func (handler *SQLiteHandler) Close() error
  • func (handler *SQLiteHandler) Commit(tx *sql.Tx) error
  • func (handler *SQLiteHandler) Connect(c cfg.Config) error
  • func (handler *SQLiteHandler) DBMS() string
  • func (handler *SQLiteHandler) Exec(query string, args ...interface{}) (sql.Result, error)
  • func (handler *SQLiteHandler) ExecuteQuery(query string, args ...interface{}) (*sql.Rows, error)
  • func (handler *SQLiteHandler) Ping() error
  • func (handler *SQLiteHandler) QueryRow(query string, args ...interface{}) *sql.Row
  • func (handler *SQLiteHandler) Rollback(tx *sql.Tx) error
• type Source
• type TxHandler

Variables

var DefaultSourceCfgJSON = []byte(`{"config":"default"}`)

DefaultSourceCfgJSON is the default configuration for a source in JSON format.

var DefaultSourceCfgRaw = json.RawMessage(DefaultSourceCfgJSON)

DefaultSourceCfgRaw is the default configuration for a source.

Functions

func DBSetup

func DBSetup(db *sql.DB, dbms, dbname string)

DBSetup checks if the specified database exists and creates it if it doesn't. It takes a pointer to a sql.DB object, the DBMS type (e.g., PostgreSQL), and the database name as parameters. If the database doesn't exist, it executes an SQL file to create it.

Types

type Handler

type Handler interface {
	Connect(c cfg.Config) error
	Close() error
	Ping() error
	ExecuteQuery(query string, args ...interface{}) (*sql.Rows, error)
	Exec(query string, args ...interface{}) (sql.Result, error)
	DBMS() string
	Begin() (*sql.Tx, error)
	Commit(tx *sql.Tx) error
	Rollback(tx *sql.Tx) error
	QueryRow(query string, args ...interface{}) *sql.Row
	CheckConnection(c cfg.Config) error
}

Handler is the interface that wraps the basic methods to interact with the database.

func NewHandler

func NewHandler(c cfg.Config) (Handler, error)

NewHandler returns a new Handler based on the database type specified in the Config struct.

type PostgresHandler

type PostgresHandler struct {
	// contains filtered or unexported fields
}

PostgresHandler is the implementation of the DatabaseHandler interface

func (*PostgresHandler) Begin

func (handler *PostgresHandler) Begin() (*sql.Tx, error)

Begin starts a transaction

func (*PostgresHandler) CheckConnection

func (handler *PostgresHandler) CheckConnection(c cfg.Config) error

CheckConnection checks if the database connection is still alive

func (*PostgresHandler) Close

func (handler *PostgresHandler) Close() error

Close closes the database connection

func (*PostgresHandler) Commit

func (handler *PostgresHandler) Commit(tx *sql.Tx) error

Commit commits a transaction

func (*PostgresHandler) ConfigForQuery

func (handler *PostgresHandler) ConfigForQuery()

func (*PostgresHandler) ConfigForWrite

func (handler *PostgresHandler) ConfigForWrite()

func (*PostgresHandler) Connect

func (handler *PostgresHandler) Connect(c cfg.Config) error

Connect connects to the database

func (*PostgresHandler) DBMS

func (handler *PostgresHandler) DBMS() string

DBMS returns the database management system

func (*PostgresHandler) Exec

func (handler *PostgresHandler) Exec(query string, args ...interface{}) (sql.Result, error)

Exec executes a commands on the database

func (*PostgresHandler) ExecuteQuery

func (handler *PostgresHandler) ExecuteQuery(query string, args ...interface{}) (*sql.Rows, error)

ExecuteQuery executes a query and returns the result

func (*PostgresHandler) Ping

func (handler *PostgresHandler) Ping() error

Ping checks if the database connection is still alive

func (*PostgresHandler) QueryRow

func (handler *PostgresHandler) QueryRow(query string, args ...interface{}) *sql.Row

QueryRow executes a query that is expected to return at most one row. QueryRow always returns a non-nil value. Errors are deferred until Row's Scan method is called.

func (*PostgresHandler) Rollback

func (handler *PostgresHandler) Rollback(tx *sql.Tx) error

Rollback rolls back a transaction

type SQLiteHandler

type SQLiteHandler struct {
	// contains filtered or unexported fields
}

SQLiteHandler is the implementation of the DatabaseHandler interface

func (*SQLiteHandler) Begin

func (handler *SQLiteHandler) Begin() (*sql.Tx, error)

Begin starts a transaction

func (*SQLiteHandler) CheckConnection

func (handler *SQLiteHandler) CheckConnection(c cfg.Config) error

CheckConnection checks if the database connection is still alive

func (*SQLiteHandler) Close

func (handler *SQLiteHandler) Close() error

Close closes the database connection

func (*SQLiteHandler) Commit

func (handler *SQLiteHandler) Commit(tx *sql.Tx) error

Commit commits a transaction

func (*SQLiteHandler) Connect

func (handler *SQLiteHandler) Connect(c cfg.Config) error

Connect connects to an SQLite database

func (*SQLiteHandler) DBMS

func (handler *SQLiteHandler) DBMS() string

DBMS returns the database management system

func (*SQLiteHandler) Exec

func (handler *SQLiteHandler) Exec(query string, args ...interface{}) (sql.Result, error)

Exec executes a commands on the database

func (*SQLiteHandler) ExecuteQuery

func (handler *SQLiteHandler) ExecuteQuery(query string, args ...interface{}) (*sql.Rows, error)

ExecuteQuery executes a query and returns the result

func (*SQLiteHandler) Ping

func (handler *SQLiteHandler) Ping() error

Ping checks if the database connection is still alive

func (*SQLiteHandler) QueryRow

func (handler *SQLiteHandler) QueryRow(query string, args ...interface{}) *sql.Row

QueryRow executes a query that is expected to return at most one row. QueryRow always returns a non-nil value. Errors are deferred until Row's Scan method is called.

func (*SQLiteHandler) Rollback

func (handler *SQLiteHandler) Rollback(tx *sql.Tx) error

Rollback rolls back a transaction

type Source

type Source struct {
	// ID is the unique identifier of the source.
	ID uint64
	// Category (optional) is the category of the source.
	CategoryID uint64
	// Name (optional) is the name of the source.
	Name string
	// UsrID (optional) is the user identifier of the source.
	UsrID uint64
	// URL is the URL of the source.
	URL string
	// Restricted indicates whether the crawling has to be restricted to the source domain or not.
	Restricted uint
	// Flags represents additional flags associated with the source.
	Flags uint
	// Config is a JSON object containing the configuration for the source.
	Config *json.RawMessage // we use json.RawMessage to avoid unmarshalling the JSON object

	// The following fields are not stored in the database but are used internally.
	Status int
}

Source represents the structure of the Sources table for a record we have decided we need to crawl. Id represents the unique identifier of the source. URL represents the URL of the source. Restricted indicates whether the crawling has to be restricted to the source domain or not. Flags represents additional flags associated with the source.

type TxHandler

type TxHandler struct {
	sql.Tx
}