sqlite

Documentation

Overview

This is the implementation of the store.URLDataStore interface for sqlite.

Use Factory() to make a new sqlite storage instance.

• You *must* call Open()
• The DB will be closed when the context passed to Open() is cancelled.
• Concurrent usage OK
• In-Memory DBs are supported
• The DB will be created if it doesn't exist

Index

• Constants
• Variables
• func DefaultOptions() sqliteOptions
• func Factory(filename string) store.Factory
• func InMemoryOptions() sqliteOptions
• type AccessMode
• type JournalMode
• type SqliteStore
  • func (s *SqliteStore) Clear() error
  • func (s *SqliteStore) Close() error
  • func (s *SqliteStore) Create() error
  • func (s SqliteStore) Fetch(url *nurl.URL) (*store.StoredUrlData, error)
  • func (s *SqliteStore) Maintain() error
  • func (s *SqliteStore) Open(ctx context.Context) error
  • func (s *SqliteStore) Store(uptr *store.StoredUrlData) (uint64, error)
• type SyncMode

Constants

const (
	FiveSecondDuration             = 5 * time.Second
	JournalModeWAL     JournalMode = "WAL"
	JournalModeMemory  JournalMode = "MEMORY"
	JournalModeOff     JournalMode = "OFF"
	BigCacheSize                   = 20000
	NormalCacheSize                = 2000 // This is actually the sqlite default
	SyncOff            SyncMode    = "OFF"
	SyncNormal         SyncMode    = "NORMAL"
	AccessModeRWC      AccessMode  = "rwc"
	AccessModeMemory   AccessMode  = "memory"
)

const (
	InMemoryDBName = ":memory:"
)

Variables

var (
	DefaultDatabase   = "scrape_data/scrape.db"
	ErrDatabaseExists = errors.New("database already exists")
	ErrIsInMemory     = errors.New("file path is in-memory DB (':memory:')")
)

var (
	ErrMappingNotFound    = errors.New("id mapping not found")
	ErrStoreNotOpen       = errors.New("store not opened for this dsn")
	ErrCantCreateDatabase = errors.New("can't create the database")
)

Functions

func DefaultOptions

func DefaultOptions() sqliteOptions

Returns an options set tuned for on-disk databases

func Factory

func Factory(filename string) store.Factory

Returns the factory function that will be used to instantiate the store. The factory function will guarantee that the preconditions are in place for the db and the instance is ready to use.

func InMemoryOptions

func InMemoryOptions() sqliteOptions

Returns an options set tuned for in-memory databases

Types

type AccessMode

type AccessMode string

type JournalMode

type JournalMode string

type SqliteStore

type SqliteStore struct {
	database.DBHandle[stmtIndex]
	// contains filtered or unexported fields
}

func (*SqliteStore) Clear

func (s *SqliteStore) Clear() error

Clear() will drop all tables and recreate them. This is a destructive operation. Clear uses the same query as Create(), so it will also re-create the database

func (*SqliteStore) Close

func (s *SqliteStore) Close() error

The underlying DB handle's close will be called when the context passed to Open() is cancelled

func (*SqliteStore) Create

func (s *SqliteStore) Create() error

When this is called, the path to the database must already exist.

func (SqliteStore) Fetch

func (s SqliteStore) Fetch(url *nurl.URL) (*store.StoredUrlData, error)

Fetch will return the stored data for requested URL, or nil if not found.

The returned result _may_ come from a different URL than the requested URL, if we've seen the passed URL before AND the page reported it's canonical url as being different than the requested URL.

In that case, the canonical version of the content will be returned, if we have it.

func (*SqliteStore) Maintain

func (s *SqliteStore) Maintain() error

func (*SqliteStore) Open

func (s *SqliteStore) Open(ctx context.Context) error

Opens the database, creating it if it doesn't exist. The passed contexts will be used for query preparation, and to close the database when the context is cancelled.

func (*SqliteStore) Store

func (s *SqliteStore) Store(uptr *store.StoredUrlData) (uint64, error)

Save the data for a URL. Returns a key for the stored URL (which you actually can't use for anything, so this interface may change)

type SyncMode

type SyncMode string