crawler

Documentation

Overview

Package crawler implements a STAC resource crawler.

Index

• Variables
• type Crawler
  • func New(visitor Visitor, options ...*Options) *Crawler
  • func (c *Crawler) Crawl(ctx context.Context, resource string) error
• type Link
• type Options
• type RecursionType
• type Resource
  • func (r Resource) ConformsTo() []string
  • func (r Resource) Extensions() []string
  • func (r Resource) Links() []Link
  • func (r Resource) Type() ResourceType
  • func (r Resource) Version() string
• type ResourceType
• type Visitor

Variables

var DefaultOptions = &Options{
	Recursion:   Children,
	Concurrency: runtime.GOMAXPROCS(0),
}

DefaultOptions used when creating a new crawler.

Types

type Crawler

type Crawler struct {
	// contains filtered or unexported fields
}

Crawler crawls STAC resources.

func New

func New(visitor Visitor, options ...*Options) *Crawler

New creates a crawler with the provided options (or DefaultOptions if none are provided).

The visitor will be called for each resource resolved.

func (*Crawler) Crawl

func (c *Crawler) Crawl(ctx context.Context, resource string) error

Crawl calls the visitor for each resolved resource.

The resource can be a file path or a URL. Any error returned by visitor will stop crawling and be returned by this function. Context cancellation will also stop crawling and the context error will be returned.

type Link

type Link map[string]string

Link represents a link to a resource.

type Options

type Options struct {
	// Limit to the number of resources to fetch and visit concurrently.
	Concurrency int

	// Strategy to use when crawling linked resources.  Use None to visit
	// a single resource.  Use Children to only visit linked item/child resources.
	// Use All to visit parent and child resources.
	Recursion RecursionType
}

Options for creating a crawler.

type RecursionType

type RecursionType string

RecursionType informs the crawler how to treat linked resources. None will only call the visitor for the first resource. Children will call the visitor for all child catalogs, collections, and items. All will call the visitor for parent resources as well as child resources.

const (
	All      RecursionType = "all"
	None     RecursionType = "none"
	Children RecursionType = "children"
)

type Resource

type Resource map[string]interface{}

Resource represents a STAC catalog, collection, or item.

func (Resource) ConformsTo

func (r Resource) ConformsTo() []string

Returns the STAC / OGC Features API conformance classes (if any).

func (Resource) Extensions

func (r Resource) Extensions() []string

Extensions returns the resource extension URLs.

func (Resource) Links

func (r Resource) Links() []Link

Links returns the resource links.

func (Resource) Type

func (r Resource) Type() ResourceType

Type returns the specific resource type.

func (Resource) Version

func (r Resource) Version() string

Version returns the STAC version.

type ResourceType

type ResourceType string

ResourceType indicates the STAC resource type.

const (
	Item       ResourceType = "item"
	Catalog    ResourceType = "catalog"
	Collection ResourceType = "collection"
)

type Visitor

type Visitor func(string, Resource) error

Visitor is called for each resource during crawling.

The resource location (URL or file path) is passed as the first argument. Any returned error will stop crawling and be returned by Crawl.