spelunker

package module
v0.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 30, 2024 License: BSD-3-Clause Imports: 13 Imported by: 6

README

go-whosonfirst-spelunker

Go package implementing a common interface for Who's On First "spelunker"-ing.

Documentation

Documentation is incomplete at this time.

Important

This is work in progress and you should expect things to change, break or simply not work yet.

Motivation

This is a refactoring of both the whosonfirst/whosonfirst-www-spelunker and whosonfirst/go-whosonfirst-browser packages.

Specifically, the former (whosonfirst-www-spelunker) is written in Python and has a sufficiently complex set of requirements that spinning up a new instance is difficult. By rewriting the spelunker tool in Go the hope is to eliminate or at least minimize these external requirements and to make it easier to deploy the spelunker to "serverless" environments like AWS Lambda or Function URLs. The latter (go-whosonfirst-browser) has developed a sufficiently large and complex code base that starting from scratch and simply copying, and adapting, existing functionality seemed easier than trying to refactor everything.

A note about versioning

Currently this package is unversioned reflecting the fact that it is still in flux. The rate of change is slowing down and will eventually be assigned version numbers less than 1.x for as long as it takes to produce the initial "minimal viable (and working)" Spelunker implementations. These versions (0.x.y) should not be considered to be backwards compatible with each other and are expected to change as the first stable interface is settled, specifically if and whether it will contain spatial functions.

Once a decision has been reached on that matter and everything is proven to work this package (and all the related packages, discussed below) will be bumped up to a "version 2.x" release, skipping version 1.x altogether, reflecting the fact that the original Python version of the Spelunker is "version 1" and that this code base is meaningfully different.

After the "v2" release this package (and related packages) will follow the standard Go convention of incrementing version numbers if and when there are changes to the underlying Spelunker interface.

Structure

There are three "classes" of go-whosonfirst-spelunker packages:

go-whosonfirst-spelunker

That would be the package that you are looking at right now. It defines the Spelunker interface defining the minimal methods required for a Spelunker application be it a command-line application, a web application or something else.

This package does not export any working implementations of the Spelunker interface. It simply defines the interface and other associated types.

go-whosonfirst-spelunker-httpd

The whosonfirst/go-whosonfirst-spelunker-httpd package provides libraries for implementing a web-based spelunker service. While it does define a working cmd/server tool demonstrating how those libraries can be used, like the go-whosonfirst-spelunker package it does not export any working implementations of the Spelunker interface.

The idea is to separate the interaction details and the mechanics of a web application from the details of how data is stored or queried from any given database containing Who's On First records.

The server itself can be run and will serve requests because its default database is the NullSpelunker implementation but since that implementation simply returns "Not implemented" for every method in the Spelunker interface it probably won't be of much use.

go-whosonfirst-spelunker-{DATABASE}

These are packages that implement the Spelunker interface for a particular database engine. Current implementations are:

go-whosonfirst-spelunker-opensearch

The whosonfirst/go-whosonfirst-spelunker-opensearch package implements the Spelunker interface using an OpenSearch document store, for example data indexed by the whosonfirst/go-whosonfirst-opensearch package.

It imports both the go-whosonfirst-spelunker and go-whosonfirst-spelunker-httpd and exports local instances of the web-based server (httpd).

Set up and example(s) to be written...

go-whosonfirst-spelunker-sql

The whosonfirst/go-whosonfirst-spelunker-sqlite package implements the Spelunker interface using a Go database/sql relational database source, for example SQLite databases produced by the whosonfirst/go-whosonfirst-sqlite-features-index package.

It imports both the go-whosonfirst-spelunker and go-whosonfirst-spelunker-httpd and exports local instances of the "spelunker" command-line tool and web-based server. For example, to create a database for use by the SQLite implementation of the Spelunker interface:

$> cd /usr/local/whosonfirst/go-whosonfirst-sqlite-features-index
$> ./bin/wof-sqlite-index-features-mattn \
	-timings \
	-database-uri mattn:///usr/local/data/ca.db \
	-spelunker-tables \
	-index-alt geojson \
	/usr/local/data/whosonfirst-data-admin-ca

And then to use that database with a local (go-whosonfirst-spelunker-sql) instance of server code exported by the go-whosonfirst-spelunker-httpd package:

$> cd /usr/local/whosonfirst/go-whosonfirst-spelunker-sql
$> ./bin/server \
	-server-uri http://localhost:8080 \
	-spelunker-uri sql://sqlite3?dsn=file:/usr/local/data/ca.db

This is what the code for the server tool looks like (with error handling omitted for the sake of brevity):

package main

import (
        "context"
        "log/slog"

        _ "github.com/mattn/go-sqlite3"
        "github.com/whosonfirst/go-whosonfirst-spelunker-httpd/app/server"
        _ "github.com/whosonfirst/go-whosonfirst-spelunker-sql"
)

func main() {
        ctx := context.Background()
        logger := slog.Default()
        server.Run(ctx, logger)
}

So far this package has only been tested with SQLite databases and probably contains some SQLite-specific syntax. The hope is that database engine specifics can be handled in conditionals in the go-whosonfirst-spelunker-sql package itself leaving consumers none the wiser.

go-whosonfirst-spelunker-sqlite

This package builds on the whosonfirst/go-whosonfirst-spelunker-sql and the whosonfirst/go-whosonfirst-spelunker-httpd packages but also imports @psanford 's sqlite3vfs packages to enable the use of SQLite databases hosted on remote servers.

It modifies the default go-whosonfirst-spelunker-httpd options and flags and then launches the spelunker server using the RunWithOptions method. For example (with error handling omitted for the sake of brevity):

import (
	"context"
	"fmt"
	"github.com/psanford/sqlite3vfs"
	"github.com/psanford/sqlite3vfshttp"
	"log/slog"
	"net/http"
	"net/url"
	"os"
	"path/filepath"

	_ "github.com/mattn/go-sqlite3"
	"github.com/whosonfirst/go-whosonfirst-spelunker-httpd/app/server"
	_ "github.com/whosonfirst/go-whosonfirst-spelunker-sql"
)

func main() {

	ctx := context.Background()
	logger := slog.Default()

	fs := server.DefaultFlagSet()

	opts, _ := server.RunOptionsFromFlagSet(ctx, fs)

	is_vfs, vfs_uri, _ := checkVFS(opts.SpelunkerURI)

	if is_vfs {
		opts.SpelunkerURI = vfs_uri
	}

	server.RunWithOptions(ctx, opts, logger)
}

func checkVFS(spelunker_uri string) (bool, string, error) {

	u, _ := url.Parse(spelunker_uri)

	q := u.Query()

	if !q.Has("vfs") {
		return false, spelunker_uri, nil
	}

	vfs_url := q.Get("vfs")

	vfs := sqlite3vfshttp.HttpVFS{
		URL: vfs_url,
		// Consult cmd/server/main.go for roundTripper; it has been
		// excluded here for the sake of brevity
		RoundTripper: &roundTripper{
			referer:   q.Get("vfs-referer"),
			userAgent: q.Get("vfs-user-agent"),
		},
	}

	sqlite3vfs.RegisterVFS("httpvfs", &vfs)

	dsn := "spelunker.db?vfs=httpvfs&mode=ro"
	q.Set("dsn", dsn)
	q.Del("vfs")

	u.RawQuery = q.Encode()
	return true, u.String(), nil
}

Note: In practice this (querying a SQLite database over HTTP) doesn't really work in a Spelunker context. Specifically, it works for simple atomic queries but the moment the application starts to do multiple overlapping queries in the same session/context there are database locks and everything times out. Maybe I am doing something wrong? I would love to know what and how to fix it if that's the case since this is a super-compelling deployment strategy. Until then it should probably best be understood as a reference implementation only.

See also

Documentation

Index

Constants

View Source 
const COUNTRY_FILTER_SCHEME string = "country"
View Source 
const IS_CURRENT_FILTER_SCHEME string = "iscurrent"
View Source 
const IS_DEPRECATED_FILTER_SCHEME string = "isdeprecated"
View Source 
const PLACETYPE_FILTER_SCHEME string = "placetype"

Variables

View Source 
var ErrNotFound = errors.New("Not found")
View Source 
var ErrNotImplemented = errors.New("Not implemented")

Functions

func RegisterSpelunker 

func RegisterSpelunker(ctx context.Context, scheme string, init_func SpelunkerInitializationFunc) error

RegisterSpelunker registers 'scheme' as a key pointing to 'init_func' in an internal lookup table used to create new `Spelunker` instances by the `NewSpelunker` method.

func Schemes 

func Schemes() []string

Schemes returns the list of schemes that have been registered.

Types

type Concordance 

type Concordance struct {
	MachineTag
	// contains filtered or unexported fields
}

func NewConcordanceFromString 

func NewConcordanceFromString(str_concordance string) (*Concordance, error)

func NewConcordanceFromTriple 

func NewConcordanceFromTriple(namespace string, predicate string, value any) *Concordance

func (*Concordance) Namespace 

func (c *Concordance) Namespace() string

func (*Concordance) Predicate 

func (c *Concordance) Predicate() string

func (*Concordance) String 

func (c *Concordance) String() string

func (*Concordance) Value 

func (c *Concordance) Value() any

type CountryFilter 

type CountryFilter struct {
	Filter
	// contains filtered or unexported fields
}

func (*CountryFilter) Scheme 

func (f *CountryFilter) Scheme() string

func (*CountryFilter) Value 

func (f *CountryFilter) Value() any

type Facet 

type Facet struct {
	Property string `json:"property"`
}

func NewFacet 

func NewFacet(p string) *Facet

func (*Facet) String 

func (f *Facet) String() string

type FacetCount 

type FacetCount struct {
	Key   string `json:"key"`
	Count int64  `json:"count"`
}

type Faceting 

type Faceting struct {
	Facet   *Facet        `json:"facet"`
	Results []*FacetCount `json:"results"`
}

type Facets 

type Facets map[string]int64

type Filter 

type Filter interface {
	Scheme() string
	Value() any
}

func NewCountryFilter 

func NewCountryFilter(ctx context.Context, uri string) (Filter, error)

func NewCountryFilterFromString 

func NewCountryFilterFromString(ctx context.Context, code string) (Filter, error)

func NewIsCurrentFilter 

func NewIsCurrentFilter(ctx context.Context, uri string) (Filter, error)

func NewIsCurrentFilterFromString 

func NewIsCurrentFilterFromString(ctx context.Context, name string) (Filter, error)

func NewIsDeprecatedFilter 

func NewIsDeprecatedFilter(ctx context.Context, uri string) (Filter, error)

func NewIsDeprecatedFilterFromString 

func NewIsDeprecatedFilterFromString(ctx context.Context, name string) (Filter, error)

func NewPlacetypeFilter 

func NewPlacetypeFilter(ctx context.Context, uri string) (Filter, error)

func NewPlacetypeFilterFromString 

func NewPlacetypeFilterFromString(ctx context.Context, name string) (Filter, error)

func NewTagFilter 

func NewTagFilter(ctx context.Context, uri string) (Filter, error)

func NewTagFilterFromString 

func NewTagFilterFromString(ctx context.Context, t string) (Filter, error)

type IsCurrentFilter 

type IsCurrentFilter struct {
	Filter
	// contains filtered or unexported fields
}

func (*IsCurrentFilter) Scheme 

func (f *IsCurrentFilter) Scheme() string

func (*IsCurrentFilter) Value 

func (f *IsCurrentFilter) Value() any

type IsDeprecatedFilter 

type IsDeprecatedFilter struct {
	Filter
	// contains filtered or unexported fields
}

func (*IsDeprecatedFilter) Scheme 

func (f *IsDeprecatedFilter) Scheme() string

func (*IsDeprecatedFilter) Value 

func (f *IsDeprecatedFilter) Value() any

type Language 

type Language struct {
}

type MachineTag 

type MachineTag interface {
	Namespace() string
	Predicate() string
	Value() any
	String() string
}

type NullSpelunker 

type NullSpelunker struct {
	Spelunker
}

NullSpelunker implements the Spelunker interface but returns an `ErrNotImplemented` error for every method. The easiest way to think about NullSpelunker is that its a template for implementing the Spelunker interface for an actual working database.

func (*NullSpelunker) CountDescendants 

func (s *NullSpelunker) CountDescendants(ctx context.Context, id int64) (int64, error)

func (*NullSpelunker) GetConcordances 

func (s *NullSpelunker) GetConcordances(ctx context.Context) (*Faceting, error)

func (*NullSpelunker) GetDescendants 

func (s *NullSpelunker) GetDescendants(ctx context.Context, pg_opts pagination.Options, id int64, filters []Filter) (spr.StandardPlacesResults, pagination.Results, error)

func (*NullSpelunker) GetDescendantsFaceted 

func (s *NullSpelunker) GetDescendantsFaceted(ctx context.Context, id int64, filters []Filter, facets []*Facet) ([]*Faceting, error)

func (*NullSpelunker) GetFeatureForId 

func (s *NullSpelunker) GetFeatureForId(ctx context.Context, id int64, uri_args *uri.URIArgs) ([]byte, error)

func (*NullSpelunker) GetPlacetypes 

func (s *NullSpelunker) GetPlacetypes(ctx context.Context) (*Faceting, error)

func (*NullSpelunker) GetRecent 

func (s *NullSpelunker) GetRecent(ctx context.Context, pg_opts pagination.Options, d time.Duration, filters []Filter) (spr.StandardPlacesResults, pagination.Results, error)

func (*NullSpelunker) GetRecentFaceted 

func (s *NullSpelunker) GetRecentFaceted(ctx context.Context, d time.Duration, filters []Filter, facets []*Facet) ([]*Faceting, error)

func (*NullSpelunker) GetRecordForId 

func (s *NullSpelunker) GetRecordForId(ctx context.Context, id int64, uri_args *uri.URIArgs) ([]byte, error)

func (*NullSpelunker) GetSPRForId 

func (s *NullSpelunker) GetSPRForId(ctx context.Context, id int64, uri_args *uri.URIArgs) (spr.StandardPlacesResult, error)

func (*NullSpelunker) GetTags 

func (s *NullSpelunker) GetTags(ctx context.Context) (*Faceting, error)

func (*NullSpelunker) HasConcordance 

func (s *NullSpelunker) HasConcordance(ctx context.Context, pg_opts pagination.Options, namespace string, predicate string, value any, filters []Filter) (spr.StandardPlacesResults, pagination.Results, error)

func (*NullSpelunker) HasConcordanceFaceted 

func (s *NullSpelunker) HasConcordanceFaceted(ctx context.Context, namespace string, predicate string, value any, filters []Filter, facets []*Facet) ([]*Faceting, error)

func (*NullSpelunker) HasPlacetype 

func (s *NullSpelunker) HasPlacetype(ctx context.Context, pg_opts pagination.Options, pt *placetypes.WOFPlacetype, filters []Filter) (spr.StandardPlacesResults, pagination.Results, error)

func (*NullSpelunker) HasPlacetypeFaceted 

func (s *NullSpelunker) HasPlacetypeFaceted(ctx context.Context, pt *placetypes.WOFPlacetype, filters []Filter, facets []*Facet) ([]*Faceting, error)

func (*NullSpelunker) HasTag 

func (s *NullSpelunker) HasTag(ctx context.Context, pg_opts pagination.Options, tag string, filters []Filter) (spr.StandardPlacesResults, pagination.Results, error)

func (*NullSpelunker) HasTagFaceted 

func (s *NullSpelunker) HasTagFaceted(ctx context.Context, tag string, filters []Filter, facets []*Facet) ([]*Faceting, error)

func (*NullSpelunker) Search 

func (s *NullSpelunker) Search(ctx context.Context, pg_opts pagination.Options, q *SearchOptions, filters []Filter) (spr.StandardPlacesResults, pagination.Results, error)

func (*NullSpelunker) SearchFaceted 

func (s *NullSpelunker) SearchFaceted(ctx context.Context, q *SearchOptions, filters []Filter, facets []*Facet) ([]*Faceting, error)

func (*NullSpelunker) VisitingNullIsland 

func (s *NullSpelunker) VisitingNullIsland(ctx context.Context, pg_opts pagination.Options, filters []Filter) (spr.StandardPlacesResults, pagination.Results, error)

func (*NullSpelunker) VisitingNullIslandFaceted 

func (s *NullSpelunker) VisitingNullIslandFaceted(ctx context.Context, filters []Filter, facets []*Facet) ([]*Faceting, error)

type PlacetypeFilter 

type PlacetypeFilter struct {
	Filter
	// contains filtered or unexported fields
}

func (*PlacetypeFilter) Scheme 

func (f *PlacetypeFilter) Scheme() string

func (*PlacetypeFilter) Value 

func (f *PlacetypeFilter) Value() any

type SearchOptions 

type SearchOptions struct {
	Query string `json:"query"`
}

type Sort 

type Sort struct {
	Property string `json:"property"`
	Order    string `json:"order"`
}

type Spelunker 

type Spelunker interface {

	// Retrieve properties (or more specifically the "document") for...
	GetRecordForId(context.Context, int64, *uri.URIArgs) ([]byte, error)
	GetSPRForId(context.Context, int64, *uri.URIArgs) (spr.StandardPlacesResult, error)
	// Retrive GeoJSON Feature for...
	GetFeatureForId(context.Context, int64, *uri.URIArgs) ([]byte, error)

	// Retrieve all the Who's On First record that are a descendant of a specific Who's On First ID.
	GetDescendants(context.Context, pagination.Options, int64, []Filter) (spr.StandardPlacesResults, pagination.Results, error)
	GetDescendantsFaceted(context.Context, int64, []Filter, []*Facet) ([]*Faceting, error)
	// Return the total number of Who's On First records that are a descendant of a specific Who's On First ID.
	CountDescendants(context.Context, int64) (int64, error)

	// Retrieve all the Who's On First records that match a search criteria.
	Search(context.Context, pagination.Options, *SearchOptions, []Filter) (spr.StandardPlacesResults, pagination.Results, error)
	SearchFaceted(context.Context, *SearchOptions, []Filter, []*Facet) ([]*Faceting, error)

	// Retrieve all the Who's On First records that have been modified with a window of time.
	GetRecent(context.Context, pagination.Options, time.Duration, []Filter) (spr.StandardPlacesResults, pagination.Results, error)
	GetRecentFaceted(context.Context, time.Duration, []Filter, []*Facet) ([]*Faceting, error)

	GetPlacetypes(context.Context) (*Faceting, error)
	HasPlacetype(context.Context, pagination.Options, *placetypes.WOFPlacetype, []Filter) (spr.StandardPlacesResults, pagination.Results, error)
	HasPlacetypeFaceted(context.Context, *placetypes.WOFPlacetype, []Filter, []*Facet) ([]*Faceting, error)

	GetConcordances(context.Context) (*Faceting, error)
	HasConcordance(context.Context, pagination.Options, string, string, any, []Filter) (spr.StandardPlacesResults, pagination.Results, error)
	HasConcordanceFaceted(context.Context, string, string, any, []Filter, []*Facet) ([]*Faceting, error)

	GetTags(context.Context) (*Faceting, error)
	HasTag(context.Context, pagination.Options, string, []Filter) (spr.StandardPlacesResults, pagination.Results, error)
	HasTagFaceted(context.Context, string, []Filter, []*Facet) ([]*Faceting, error)

	VisitingNullIsland(context.Context, pagination.Options, []Filter) (spr.StandardPlacesResults, pagination.Results, error)
	VisitingNullIslandFaceted(context.Context, []Filter, []*Facet) ([]*Faceting, error)
}

Spelunker is an interface for writing data to multiple sources or targets.

func NewNullSpelunker 

func NewNullSpelunker(ctx context.Context, uri string) (Spelunker, error)

func NewSpelunker 

func NewSpelunker(ctx context.Context, uri string) (Spelunker, error)

NewSpelunker returns a new `Spelunker` instance configured by 'uri'. The value of 'uri' is parsed as a `url.URL` and its scheme is used as the key for a corresponding `SpelunkerInitializationFunc` function used to instantiate the new `Spelunker`. It is assumed that the scheme (and initialization function) have been registered by the `RegisterSpelunker` method.

type SpelunkerInitializationFunc 

type SpelunkerInitializationFunc func(ctx context.Context, uri string) (Spelunker, error)

SpelunkerInitializationFunc is a function defined by individual spelunker package and used to create an instance of that spelunker

type TagFilter 

type TagFilter struct {
	Filter
	// contains filtered or unexported fields
}

func (*TagFilter) Scheme 

func (f *TagFilter) Scheme() string

func (*TagFilter) Value 

func (f *TagFilter) Value() any

Source Files

View all Source files

Directories

Path Synopsis
app
cli
cmd
spelunker

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.