crawler

Documentation

Overview

Package crawler implements a STAC resource crawler.

Index

• Constants
• Variables
• type Crawler
  • func New(visitor Visitor) *Crawler
  • func NewWithOptions(visitor Visitor, options *Options) *Crawler
  • func (c *Crawler) Crawl(ctx context.Context, resource string) error
• type Link
• type Options
• type RecursionType
• type Resource
  • func (r Resource) Extensions() []string
  • func (r Resource) Links() []Link
  • func (r Resource) Type() ResourceType
  • func (r Resource) Version() string
• type ResourceType
• type Visitor

Constants

const (
	Item       = ResourceType("item")
	Catalog    = ResourceType("catalog")
	Collection = ResourceType("collection")
)

Variables

var DefaultOptions = &Options{
	Recursion:   Children,
	Concurrency: runtime.GOMAXPROCS(0),
}

DefaultOptions used when creating a new crawler.

Types

type Crawler

type Crawler struct {
	// contains filtered or unexported fields
}

Crawler crawls STAC resources.

func New

func New(visitor Visitor) *Crawler

New creates a crawler with the default options.

The visitor will be called for each resource resolved.

func NewWithOptions

func NewWithOptions(visitor Visitor, options *Options) *Crawler

NewWithOptions creates a crawler with the given options.

func (*Crawler) Crawl

func (c *Crawler) Crawl(ctx context.Context, resource string) error

Crawl calls the visitor for each resolved resource.

The resource can be a file path or a URL.

type Link

type Link map[string]string

Link represents a link to a resource.

type Options

type Options struct {
	// Limit to the number of resources to fetch and visit concurrently.
	Concurrency int

	// Strategy to use when crawling linked resources.  Use None to visit
	// a single resource.  Use Children to only visit linked item/child resources.
	// Use All to visit parent and child resources.
	Recursion RecursionType
}

Options for creating a crawler.

type RecursionType

type RecursionType string

const (
	All      RecursionType = "all"
	None     RecursionType = "none"
	Children RecursionType = "children"
)

type Resource

type Resource map[string]interface{}

Resource represents a STAC catalog, collection, or item.

func (Resource) Extensions

func (r Resource) Extensions() []string

Extensions returns the resource extension URLs.

func (Resource) Links

func (r Resource) Links() []Link

Links returns the resource links.

func (Resource) Type

func (r Resource) Type() ResourceType

Type returns the specific resource type.

func (Resource) Version

func (r Resource) Version() string

Version returns the STAC version.

type ResourceType

type ResourceType string

type Visitor

type Visitor func(string, Resource) error