deno

Documentation

Overview

Copyright 2020-2021 William Perron. All rights reserved. MIT License.

Copyright 2020-2021 William Perron. All rights reserved. MIT License.

Copyright 2020-2021 William Perron. All rights reserved. MIT License.

Copyright 2020-2021 William Perron. All rights reserved. MIT License.

Index

• Constants
• func Enqueue(mods chan Module, q Queue) (chan Module, chan error)
• func Exists() bool
• type ChanQueue
  • func NewChanQueue(buf int) ChanQueue
  • func (q *ChanQueue) Get() (Module, error)
  • func (q *ChanQueue) Put(m Module) error
• type Client
  • func DefaultClient() Client
  • func NewInstrumentedClient() Client
• type DenoInfo
  • func ExecInfo(ctx context.Context, target url.URL) (DenoInfo, error)
• type FileEntry
• type Module
• type Queue
• type SQSQueue
  • func NewSQSQueue(c aws.Config, url string, buf int) *SQSQueue
  • func (s *SQSQueue) Approx() (int, error)
  • func (s *SQSQueue) Delete(m Module) error
  • func (s *SQSQueue) Get() (Module, error)
  • func (s *SQSQueue) Put(m Module) error
• type XQueuedCrawler
  • func NewXQueuedCrawler(q Queue) *XQueuedCrawler
  • func (x *XQueuedCrawler) Crawl(ctx context.Context) chan error
  • func (x *XQueuedCrawler) Done() <-chan bool
  • func (x *XQueuedCrawler) IterateModules(ctx context.Context) (chan Module, chan error)

Constants

const API_HOST = "api.deno.land"

const CDN_HOST = "cdn.deno.land"

const PREFIX_LENGTH = len("https://deno.land/x/")

Functions

func Enqueue

func Enqueue(mods chan Module, q Queue) (chan Module, chan error)

Enqueue serves as a passthrough for channels of deno.Module. It puts the incoming messages in a Queue and consumes it to send messages down the output channel as well. It serves as an intermediary steps where an implementation of a Queue that uses a persistent back end like SQS or Kafka can be used. This is necessary to be able to start and stop the crawler arbitrarily and pick up where it left off

func Exists

func Exists() bool

Exists checks whether the `deno` executable is in path

Types

type ChanQueue

type ChanQueue struct {
	// contains filtered or unexported fields
}

ChanQueue is an in-memory queue that uses channels under the hood. If the channel is unbuffered, Put and Get are blocking operations

func NewChanQueue

func NewChanQueue(buf int) ChanQueue

NewChanQueue returns a new ChanQueue instance

func (*ChanQueue) Get

func (q *ChanQueue) Get() (Module, error)

Get gets the next message from the underlying channel

func (*ChanQueue) Put

func (q *ChanQueue) Put(m Module) error

Put sends a message to the underlying channel

type Client

type Client interface {
	DoRequest(*http.Request) (*http.Response, error)
}

Client interface defines the basic functions of an HTTP crawler

func DefaultClient

func DefaultClient() Client

DefaultClient returns an instance of a crawler that uses the default http client

func NewInstrumentedClient

func NewInstrumentedClient() Client

NewInstrumentedClient returns an instance of a crawler that uses an http client intstrumented with Prometheus

type DenoInfo

type DenoInfo struct {
	TotalSize int                  `json:"totalSize"`
	Module    string               `json:"module"`
	Map       *string              `json:"map"`
	Compiled  *string              `json:"compiled"`
	DepCount  int                  `json:"depCount"`
	FileType  string               `json:"fileType"`
	Files     map[string]FileEntry `json:"files"`
}

DenoInfo is the in-memory representation of the output of `deno info --json`

func ExecInfo

func ExecInfo(ctx context.Context, target url.URL) (DenoInfo, error)

ExecInfo executes `deno info` as a subcommand and returns the DenoInfo struct that it outputs

type FileEntry

type FileEntry struct {
	Deps []string `json:"deps"`
	Size int      `json:"size"`
}

FileEntry in the Files map of DenoInfo

type Module

type Module struct {
	Name     string
	Versions map[string][]directoryListing
}

Module contains the name of the volume and a map of all its versions to all the files contained in the module

type Queue

type Queue interface {
	Put(Module) error
	Get() (Module, error)
	// contains filtered or unexported methods
}

Queue interface for putting and getting messages. The interface doesn make any guarantees about message ordering, this concern must be managed by the interface implementation.

type SQSQueue

type SQSQueue struct {
	// contains filtered or unexported fields
}

SQSQueue is a simple abstraction over the standard sqs.Client struct that implements the Queue interface

func NewSQSQueue

func NewSQSQueue(c aws.Config, url string, buf int) *SQSQueue

NewSQSQueue instantiates a new SQS Client with the given config

func (*SQSQueue) Approx

func (s *SQSQueue) Approx() (int, error)

Approx returns the approximate total number of messages in the queue, visible, delayed or not visible.

func (*SQSQueue) Delete

func (s *SQSQueue) Delete(m Module) error

Delete uses the message's receipt handle to delete the message from the queue

func (*SQSQueue) Get

func (s *SQSQueue) Get() (Module, error)

Get returns a single message either from the internal buffer queue or from the SQS queue

func (*SQSQueue) Put

func (s *SQSQueue) Put(m Module) error

Put sends a message to SQS and returns any error encountered by the aws client

type XQueuedCrawler

type XQueuedCrawler struct {
	Client

	Queue
	// contains filtered or unexported fields
}

XQueuedCrawler is a composite type composed of both a Queue and a Crawler

func NewXQueuedCrawler

func NewXQueuedCrawler(q Queue) *XQueuedCrawler

NewXQueuedCrawler returns an instance of a crawler for https://deno.land with a Queue

func (*XQueuedCrawler) Crawl

func (x *XQueuedCrawler) Crawl(ctx context.Context) chan error

Crawl asynchronously crawls https://deno.land and puts each Module in the queue to be processed later

func (*XQueuedCrawler) Done

func (x *XQueuedCrawler) Done() <-chan bool

Done returns the done channel of the crawler

func (*XQueuedCrawler) IterateModules

func (x *XQueuedCrawler) IterateModules(ctx context.Context) (chan Module, chan error)

IterateModules asynchronously consumes the queue and sends each Module to a channel