crawler

Documentation

Overview

Package crawler implements a STAC resource crawler.

Index

• Variables
• func Crawl(resource string, visitor Visitor, options ...*Options) error
• func Join(resource string, visitor Visitor, options ...*Options) error
• func LinkTypeAnyJSON(link Link) bool
• func LinkTypeApplicationJSON(link Link) bool
• func LinkTypeGeoJSON(link Link) bool
• func LinkTypeNone(link Link) bool
• type Asset
  • func (a Asset) Description() string
  • func (a Asset) Href() string
  • func (a Asset) Roles() []string
  • func (a Asset) Title() string
  • func (a Asset) Type() string
• type ErrorHandler
• type Link
• type LinkMatcher
• type Links
  • func (links Links) Rel(rel string, matchers ...LinkMatcher) Link
• type Options
• type RecursionType
• type Resource
  • func (r Resource) Assets() map[string]Asset
  • func (r Resource) ConformsTo() []string
  • func (r Resource) Extensions() []string
  • func (r Resource) Links() Links
  • func (r Resource) Type() ResourceType
  • func (r Resource) Version() string
• type ResourceType
• type Task
• type Visitor

Variables

var DefaultOptions = &Options{
	Context:      context.Background(),
	Recursion:    Children,
	Concurrency:  runtime.GOMAXPROCS(0),
	ErrorHandler: func(err error) error { return err },
}

DefaultOptions used when creating a new crawler.

Functions

func Crawl

func Crawl(resource string, visitor Visitor, options ...*Options) error

Crawl calls the visitor for each resolved resource.

The resource can be a file path or a URL. Any error returned by visitor will stop crawling and be returned by this function. Context cancellation will also stop crawling and the context error will be returned.

func Join

func Join(resource string, visitor Visitor, options ...*Options) error

Join adds a crawler to an existing crawl instead of initiating a new one.

This is useful when two crawlers share the same queue and the first crawler has been taken down or the queue is getting too large. Join will return immediately if the queue of tasks is empty.

func LinkTypeAnyJSON

func LinkTypeAnyJSON(link Link) bool

func LinkTypeApplicationJSON

func LinkTypeApplicationJSON(link Link) bool

func LinkTypeGeoJSON

func LinkTypeGeoJSON(link Link) bool

func LinkTypeNone

func LinkTypeNone(link Link) bool

Types

type Asset

type Asset map[string]interface{}

Asset provides metadata about data for an item.

func (Asset) Description

func (a Asset) Description() string

Description returns the asset's description.

func (Asset) Href

func (a Asset) Href() string

Href returns the asset's href.

func (Asset) Roles

func (a Asset) Roles() []string

Roles returns the asset's description.

func (Asset) Title

func (a Asset) Title() string

Title returns the asset's title.

func (Asset) Type

func (a Asset) Type() string

Type returns the asset's type.

type ErrorHandler

type ErrorHandler func(error) error

ErrorHandler is called with any errors during a crawl. If the function returns nil, the crawl will continue. If the function returns an error, the crawl will stop.

type Link

type Link map[string]string

Link represents a link to a resource.

type LinkMatcher

type LinkMatcher func(link Link) bool

type Links

type Links []Link

Links is a slice of links.

func (Links) Rel

func (links Links) Rel(rel string, matchers ...LinkMatcher) Link

type Options

type Options struct {
	// Optional context.  If provided, the crawler will stop when the context is done.
	Context context.Context

	// Limit to the number of resources to fetch and visit concurrently.
	Concurrency int

	// Strategy to use when crawling linked resources.  Use None to visit
	// a single resource.  Use Children to only visit linked item/child resources.
	Recursion RecursionType

	// Optional function to limit which resources to crawl.  If provided, the function
	// will be called with the URL or absolute path to a resource before it is crawled.
	// If the function returns false, the resource will not be read and the visitor will
	// not be called.
	Filter func(string) bool

	// Optional function to handle any errors during the crawl.  By default, any error
	// will stop the crawl.  To continue crawling on error, provide a function that
	// returns nil.
	ErrorHandler ErrorHandler

	// Optional queue to use for crawling tasks.  If not provided, an in-memory queue
	// will be used.  When running a crawl across multiple processes, it can be useful
	// to provide a queue that is shared across processes.
	Queue workgroup.Queue[*Task]
}

Options for creating a crawler.

type RecursionType

type RecursionType string

RecursionType informs the crawler how to treat linked resources. None will only call the visitor for the first resource. Children will call the visitor for all child catalogs, collections, and items. All will call the visitor for parent resources as well as child resources.

const (
	None     RecursionType = "none"
	Children RecursionType = "children"
)

type Resource

type Resource map[string]interface{}

Resource represents a STAC catalog, collection, or item.

func (Resource) Assets

func (r Resource) Assets() map[string]Asset

Returns the assets (if any).

func (Resource) ConformsTo

func (r Resource) ConformsTo() []string

Returns the STAC / OGC Features API conformance classes (if any).

func (Resource) Extensions

func (r Resource) Extensions() []string

Extensions returns the resource extension URLs.

func (Resource) Links

func (r Resource) Links() Links

Links returns the resource links.

func (Resource) Type

func (r Resource) Type() ResourceType

Type returns the specific resource type.

func (Resource) Version

func (r Resource) Version() string

Version returns the STAC version.

type ResourceType

type ResourceType string

ResourceType indicates the STAC resource type.

const (
	Item       ResourceType = "item"
	Catalog    ResourceType = "catalog"
	Collection ResourceType = "collection"
)

type Task

type Task struct {
	Url  string
	Type string
}

type Visitor

type Visitor func(string, Resource) error

Visitor is called for each resource during crawling.

The resource location (URL or file path) is passed as the first argument. Any returned error will stop crawling and be returned by Crawl.