feeds

README

Feeds

Each of the feeds have their own implementation and support their own set of configuration options.

Configuration options

packages this configuration option is only available on certain feeds, check the README of the feed you're interested in for information on this.

poll_rate this allows for setting the frequency of polling for this specific feed. This is supported by all feeds. The value should be a string formatted for duration parser. Setting this value will enable the scheduled polling regardless of the value of timer in the root of the configuration.

Example

Poll Pypi every 5 minutes

feeds:
- type: pypi
  options:
    poll_rate: "5m"

Poll npm every 10 minutes and crates every hour

feeds:
- type: npm
  options:
    poll_rate: "10m"
- type: crates
  options:
    poll_rate: "1h"

Poll a subset of pypi every 10 minutes

feeds:
- type: pypi
  options:
    packages:
    - numpy
    - django
    poll_rate: "10m"

Documentation

Index

• Constants
• Variables
• func FindCutoff(cutoff time.Time, pkgs []*Package) time.Time
• type FeedOptions
• type LossyFeedAlerter
  • func NewLossyFeedAlerter(eventHandler *events.Handler) *LossyFeedAlerter
  • func (lfa *LossyFeedAlerter) ProcessPackages(feed string, packages []*Package)
• type Package
  • func ApplyCutoff(pkgs []*Package, cutoff time.Time) []*Package
  • func NewArtifact(created time.Time, name, version, artifactID, feed string) *Package
  • func NewPackage(created time.Time, name, version, feed string) *Package
• type PackagePollError
  • func (err PackagePollError) Error() string
• type ScheduledFeed
• type UnsupportedOptionError
  • func (err UnsupportedOptionError) Error() string

Constants

const (
	DefaultUserAgent = "package-feeds (github.com/ossf/package-feeds)"
)

Variables

var ErrNoPackagesPolled = errors.New("no packages were successfully polled")

Functions

func FindCutoff

func FindCutoff(cutoff time.Time, pkgs []*Package) time.Time

Types

type FeedOptions

type FeedOptions struct {
	// A collection of package names to poll instead of standard firehose behaviour.
	// Not supported by all feeds.
	Packages *[]string `yaml:"packages"`

	// Cron string for scheduling the polling for the feed.
	PollRate string `yaml:"poll_rate"`
}

General configuration options for feeds.

type LossyFeedAlerter

type LossyFeedAlerter struct {
	// contains filtered or unexported fields
}

func NewLossyFeedAlerter

func NewLossyFeedAlerter(eventHandler *events.Handler) *LossyFeedAlerter

Creates a LossyFeedAlerter, capable of tracking packages and identifying potential loss in feeds using RSS style APIs. This can only be used in feeds which produce an overlap of packages upon their requests to the API, if a timestamp is used to query the API then loss is unlikely due to requesting data since a previous query.

func (*LossyFeedAlerter) ProcessPackages

func (lfa *LossyFeedAlerter) ProcessPackages(feed string, packages []*Package)

Processes a new collection of packages and compares against the previously processed slice of packages, if an overlap is not detected this is a sign of potential loss of data and the configured event handler is notified via a LossyFeedEvent.

type Package

type Package struct {
	Name        string    `json:"name"`
	Version     string    `json:"version"`
	CreatedDate time.Time `json:"created_date"`
	Type        string    `json:"type"`
	ArtifactID  string    `json:"artifact_id"`
	SchemaVer   string    `json:"schema_ver"`
}

Marshalled json output validated against package.schema.json.

func ApplyCutoff

func ApplyCutoff(pkgs []*Package, cutoff time.Time) []*Package

func NewArtifact

func NewArtifact(created time.Time, name, version, artifactID, feed string) *Package

NewArtifact creates a Package object with the artifact ID field populated.

func NewPackage

func NewPackage(created time.Time, name, version, feed string) *Package

NewPackage creates a Package object without the artifact ID field populated.

type PackagePollError

type PackagePollError struct {
	Err  error
	Name string
}

func (PackagePollError) Error

func (err PackagePollError) Error() string

type ScheduledFeed

type ScheduledFeed interface {
	Latest(cutoff time.Time) ([]*Package, time.Time, []error)
	GetFeedOptions() FeedOptions
	GetName() string
}

type UnsupportedOptionError

type UnsupportedOptionError struct {
	Option string
	Feed   string
}

func (UnsupportedOptionError) Error

func (err UnsupportedOptionError) Error() string