index

README

go-whosonfirst-sqlite-index

Go package for indexing SQLite databases using table constucts defined in the aaronland/go-sqlite/v2 package and records defined by the whosonfirst/go-whosonfirst-iterate/v2 package.

Documentation

Tools

$> make cli
go build -mod vendor -o bin/example cmd/example/main.go

example

$> ./bin/example -h
Usage of ./bin/example:
  -database-uri string
    	 (default "modernc://mem")
  -emitter-uri string
    	A valid whosonfirst/go-whosonfirst-iterate/v2 URI. Valid schemes are: directory://,featurecollection://,file://,filelist://,geojsonl://,null://,repo://. (default "repo://")
  -live-hard-die-fast
    	Enable various performance-related pragmas at the expense of possible (unlikely) database corruption (default true)
  -post-index
    	Enable post indexing callback function
  -timings
    	Display timings during and after indexing

For example:

$> ./bin/example -dsn 'modernc://cwd/test.db' /usr/local/data/sfomuseum-data-architecture/
2021/02/18 11:34:58 time to index paths (1) 403.514656ms

$> sqlite3  test.db 
SQLite version 3.28.0 2019-04-15 14:49:49
Enter ".help" for usage hints.
sqlite> .tables
example

sqlite> SELECT COUNT(id) FROM example;
12751

See also

• https://github.com/aaronland/go-sqlite
• https://github.com/whosonfirst/go-whosonfirst-iterate

Documentation

Overview

package index provides methods for indexing SQLite databases using table constucts defined in the `aaronland/go-sqlite` package and records defined by the `whosonfirst/go-whosonfirst-iterate/v2` package.

Index

• type SQLiteIndexer
  • func NewSQLiteIndexer(opts *SQLiteIndexerOptions) (*SQLiteIndexer, error)
  • func (idx *SQLiteIndexer) IndexPaths(ctx context.Context, iterator_uri string, uris []string) error
  • func (idx *SQLiteIndexer) IndexURIs(ctx context.Context, iterator_uri string, uris ...string) error
• type SQLiteIndexerLoadRecordFunc
• type SQLiteIndexerOptions
• type SQLiteIndexerPostIndexFunc

Types

type SQLiteIndexer

type SQLiteIndexer struct {

	// Timings is a boolean flag indicating whether timings (time to index records) should be recorded)
	Timings bool
	// Logger is a `log.Logger` instance
	Logger *log.Logger
	// contains filtered or unexported fields
}

SQLiteIndexer is a struct that provides methods for indexing records in one or more SQLite database tables

func NewSQLiteIndexer

func NewSQLiteIndexer(opts *SQLiteIndexerOptions) (*SQLiteIndexer, error)

NewSQLiteInder returns a `SQLiteIndexer` configured with 'opts'.

func (*SQLiteIndexer) IndexPaths

func (idx *SQLiteIndexer) IndexPaths(ctx context.Context, iterator_uri string, uris []string) error

IndexPaths is deprecated and has been superseded by the `IndexURIs` method.

func (*SQLiteIndexer) IndexURIs

func (idx *SQLiteIndexer) IndexURIs(ctx context.Context, iterator_uri string, uris ...string) error

IndexURIs will index records returned by the `whosonfirst/go-whosonfirst-iterate` instance for 'uris',

type SQLiteIndexerLoadRecordFunc

type SQLiteIndexerLoadRecordFunc func(context.Context, string, io.ReadSeeker, ...interface{}) (interface{}, error)

SQLiteIndexerLoadRecordFunc is a custom `whosonfirst/go-whosonfirst-iterate/v2` callback function to be invoked for each record processed by the `IndexURIs` method.

type SQLiteIndexerOptions

type SQLiteIndexerOptions struct {
	// DB is the `aaronland/go-sqlite.Database` instance that records will be indexed in.
	DB sqlite.Database
	// Tables is the list of `aaronland/go-sqlite.Table` instances that records will be indexed in.
	Tables []sqlite.Table
	// LoadRecordFunc is a custom `whosonfirst/go-whosonfirst-iterate/v2` callback function to be invoked
	// for each record processed by	the `IndexURIs`	method.
	LoadRecordFunc SQLiteIndexerLoadRecordFunc
	// PostIndexFunc is an optional custom function to invoke after a record has been indexed.
	PostIndexFunc SQLiteIndexerPostIndexFunc
}

SQLiteIndexerOptions

type SQLiteIndexerPostIndexFunc

type SQLiteIndexerPostIndexFunc func(context.Context, sqlite.Database, []sqlite.Table, interface{}) error

SQLiteIndexerPostIndexFunc is a custom function to invoke after a record has been indexed.