persistence

Documentation

Index

• Variables
• func CountUrlsWhere(ctx context.Context, db *sql.DB, where string, args ...interface{}) (int, error)
• func GetLatestTime(ctx context.Context, db *sql.DB, extractor types.Extractor) (*time.Time, error)
• func InitDb(ctx context.Context, c *config.AppConfig) (*sql.DB, error)
• func InsertDocument(ctx context.Context, db *sql.DB, row *types.DocumentRow) error
• func InsertUrl(ctx context.Context, db *sql.DB, row *types.UrlRow) error
• func InsertUrlMeta(ctx context.Context, db *sql.DB, rows ...types.UrlMetaRow) error
• func InsertVisit(ctx context.Context, db *sql.DB, row *types.VisitRow) error
• func OpenConnection(ctx context.Context, c *config.AppConfig) (*sql.DB, error)
• func UrlsById(ctx context.Context, db *sql.DB, ids ...string) ([]types.UrlDbEntity, error)

Variables

var MigrationsDir embed.FS

@note Initially visits had a unique index on `extractor_name, url_md5, visit_time`, however, this lead to duplicate visits. The visits were duplicated because some browsers will immport the history of other browsers, or in cases like the history trends chrome extension duplication is explicitly part of the goal. Thus, in order to minimize duplication visits are considered unique by url and unix timestamp.

Functions

func CountUrlsWhere

func CountUrlsWhere(ctx context.Context, db *sql.DB, where string, args ...interface{}) (int, error)

Count the number of urls that match the given where clause. URL meta is available in the where clause as well.

func GetLatestTime

func GetLatestTime(ctx context.Context, db *sql.DB, extractor types.Extractor) (*time.Time, error)

func InitDb

func InitDb(ctx context.Context, c *config.AppConfig) (*sql.DB, error)

Initialize the database. Create tables and indexes

func InsertDocument

func InsertDocument(ctx context.Context, db *sql.DB, row *types.DocumentRow) error

func InsertUrl

func InsertUrl(ctx context.Context, db *sql.DB, row *types.UrlRow) error

func InsertUrlMeta

func InsertUrlMeta(ctx context.Context, db *sql.DB, rows ...types.UrlMetaRow) error

func InsertVisit

func InsertVisit(ctx context.Context, db *sql.DB, row *types.VisitRow) error

func OpenConnection

func OpenConnection(ctx context.Context, c *config.AppConfig) (*sql.DB, error)

Open a connection to the database. Calling code should close the connection when done. @note It is assumed that the database is already initialized. Thus this may be less useful than `InitDB`

func UrlsById

func UrlsById(ctx context.Context, db *sql.DB, ids ...string) ([]types.UrlDbEntity, error)