thingfulx

package module
v0.8.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 19, 2019 License: MIT Imports: 13 Imported by: 2

README

thingfulx

Library providing an interface for Thingful fetcher and parser interfaces to implement to provide new upstream source parsing capability.

wercker status GoDoc Go Report Card

Documentation

Overview

Package thingfulx provides a compatibility layer for implementing fetchers for use with the Thingful indexing system. New upstream providers can be added by implementing the core Fetcher interface contained within this library. These libraries can then be easily imported and used by Thingful.

Index

Constants

View Source 
const (
	// ReproductionPerm is "making multiple copies of a dataset
	ReproductionPerm Permission = "cc:Reproduction"

	// DistributionPerm is "distribution, public display, and publicly performance"
	DistributionPerm Permission = "cc:Distribution"

	// DerivativeWorksPerm is "distribution of derivative works"
	DerivativeWorksPerm Permission = "cc:DerivativeWorks"

	// SharingPerm is "permits commercial derivatives, but only non-commercial
	// distribution"
	SharingPerm Permission = "cc:Sharing"

	// NoticeReq is "copyright and license notices be kept intact"
	NoticeReq Requirement = "cc:Notice"

	// AttributionReq is "credit required to give to copyright holder and/or author"
	AttributionReq Requirement = "cc:Attribution"

	// ShareAlikeReq is "derivative works be licensed under the same terms or
	// compatible terms as the original work"
	ShareAlikeReq Requirement = "cc:ShareAlike"

	// SourceCodeReq is "source code (the preferred form for making modifications)
	// must be provided when exercising some rights granted by the license"
	SourceCodeReq Requirement = "cc:SourceCode"

	// CopyleftReq is "derivative and combined works must be licensed under
	// specified terms, similar to those on the original work"
	CopyleftReq Requirement = "cc:Copyleft"

	// LesserCopyleftReq is "derivative works must be licensed under specified
	// terms, with at least the same conditions as the original work;
	// combinations with the work may be licensed under different terms"
	LesserCopyleftReq Requirement = "cc:LesserCopyleft"

	// CommercialUseProhib is "exercising rights for commercial purposes"
	CommercialUseProhib Prohibition = "cc:CommercialUse"

	// HighIncomeNationUseProhib is "use in a non-developing country"
	HighIncomeNationUseProhib = "cc:HighIncomeNationUse"

	// CC0V1URL is the URL identifier for CC0 1.0 Universal License
	CC0V1URL = "https://creativecommons.org/publicdomain/zero/1.0/"

	// CCByV3URL is the URL identifier for Creative Commons Attribution 3.0
	// Unported License
	CCByV3URL = "https://creativecommons.org/licenses/by/3.0/"

	// CCBySAV4URL is the URL identifier for Creative Commons
	// Attribution-ShareAlike 4.0 International
	CCBySAV4URL = "https://creativecommons.org/licenses/by-sa/4.0/"

	// CCByV4URL is the URL identifier for the Creative Commons Attribution 4.0
	// License
	CCByV4URL = "https://creativecommons.org/licenses/by/4.0/"

	// CCByNCV3URL is the URL identifier for Creative Commons
	// Atribution-NonCommercial 3.0 License
	CCByNCV3URL = "https://creativecommons.org/licenses/by-nc/3.0/"

	// CCByNDV3URL is the URL identifier for Creative Commons
	// Attribution-NoDerivs 3.0 License.
	CCByNDV3URL = "https://creativecommons.org/licenses/by-nd/3.0/"

	// CCByNCSAV3URL is the URL identifier for Creative Commons
	// Attribution-NonCommercial-Sharealike 3.0 License.
	CCByNCSAV3URL = "https://creativecommons.org/licenses/by-nc-sa/3.0/"

	// OGLV2URL is the URL identifier for the Open Government Licence version
	// 2.0 License.
	OGLV2URL = "http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/"

	// OGLV3URL is the URL identifier for the Open Government Licence version
	// 3.0 License.
	OGLV3URL = "http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/"

	// PDDLV1URL is the URL identifier for the Open Data Commons Public Domain
	// Dedication and License.
	PDDLV1URL = "https://opendatacommons.org/licenses/pddl/1.0/"

	// ODCByV1URL is the URL identifier for the Open Data Commons Attribution
	// License.
	ODCByV1URL = "https://opendatacommons.org/licenses/by/1.0/"

	// ODbLV1URL is a the URL identifier for the Open Data Commons Open Database
	// License.
	ODbLV1URL = "https://opendatacommons.org/licenses/odbl/1.0/"

	// IODLV2URL is the URL identifier for the Italian Open Data License
	IODLV2URL = "https://www.dati.gov.it/content/italian-open-data-license-v20"

	// SGODLV1 is the URL identifier for the Singapore Open Data License
	SGODLV1 = "https://data.gov.sg/open-data-licence"
)
View Source 
const (
	// ErrInvalidData is a generic error that can be returned if retrieved data
	// is invalid in any way
	ErrInvalidData = Error("thingfulx: invalid data")

	// ErrInvalidLocation is an error indicating that there is a problem with the
	// location of a fetched thing
	ErrInvalidLocation = Error("thingfulx: missing or invalid location data")

	// ErrInvalidTime is an error indicating that there is a problem with parsing
	// the time value from a retrieved thing
	ErrInvalidTime = Error("thingfulx: invalid time data")

	// ErrInvalidURL is an error used to flag an invalid url
	ErrInvalidURL = Error("thingfulx: invalid url")

	// ErrNotFound is an error that can be used for a 404 not found response
	ErrNotFound = Error("thingfulx: unexpected HTTP response code, got 404")

	// ErrRobotsForbidden is an error that is returned if access to a resource is
	// forbidden by the remote site's robots.txt
	ErrRobotsForbidden = Error("thingfulx: robots.txt forbids access to this resource")
)
View Source 
const (
	// Closed is the exported const representing a private endpoint requiring
	// authentication to use.
	Closed = Visibility("closed")

	// Shared is the exported const representing an endpoint that requires
	// authentication but is free to use once credentials are supplied.
	Shared = Visibility("shared")

	// Open is the exported const representing an open endpoint, i.e. requiring
	// no authentication to use.
	Open = Visibility("open")
)
View Source 
const (
	// ClientToken is the key we use when setting a token on a context when making
	// a request. Note it is the responsibility of the implementing indexer to
	// determine how to send this token value to the data infrastructure (i.e.
	// Authorization header, API Key in the query string or a header).
	ClientToken = ContextKey("client-token")
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Attribution 

type Attribution struct {
	// Name contains the name of an attribution that we can optionally attach to a
	// Channel should the license require it.
	Name string `json:"name"`

	// URL contains a link to an attribution that we can optionally attach to a
	// Channel should the license require it.
	URL string `json:"url"`
}

Attribution contains attribution information about the resource

type Channel 

type Channel struct {
	// URL is a unique identifier for the channel expressed as a fully qualified
	// URL that must be the URL at which data is available for the Channel.
	URL string `json:"url"`

	// Title is a human centered title for the channel that ideally also uniquely
	// identifies the Channel amongst other Channels published by a Device
	Title string `json:"title"`

	// Description is a human centered description of the channel.
	Description string `json:"description"`

	// Webpage is a link to a web resource containing information about the
	// channel or device that is publishing this data.
	Webpage string `json:"webpage"`

	// IndexedAt is a Time object indicating the timestamp when the channel was
	// indexed by Thingful
	IndexedAt time.Time `json:"indexedAt"`

	// Location is an embedded data structure containing the physical location of
	// the channel at time of indexing.
	Location *Location `json:"location"`

	// ProviderUID is the UID for the provider.
	ProviderUID string `json:"providerUid"`

	// Metadata is a data structure that holds a list of Metadata properties about
	// the Channel. This is for facts that do not fit neatly into the primary
	// attributes defined on the channel.
	Metadata []Metadata `json:"metadata"`

	// Visibility indicates what type of access is possible to the data provider
	// using the data spectrum classifications developed by the ODI, one of Open,
	// Shared or Closed.
	Visibility Visibility `json:"visibility"`

	// DataLicense contains information about the licensing requirements on the
	// data as set by the provider.
	DataLicense *DataLicense `json:"dataLicense"`

	// Attributions contains a list of attributions that we require to add to the
	// Channel
	Attributions []Attribution `json:"attributions"`

	// UpdateInterval indicates how often the upstream resource obtains fresh data
	// expressed as an integer number of seconds.
	UpdateInterval int32 `json:"updateInterval,omitempty"`

	// Type defines the type of data of this channel, eg; string, double,
	// date-time
	Type schema.DataType `json:"dataType"`

	// ContentType is used to encode the mime content type of the published data.
	ContentType string `json:"contentType,omitempty"`

	// QuantityKind defines measurement type of quantity kind of this channel, in
	// Ontology PoV, Channel is type QuantityKind, so type of channel can be
	// anything that classified as quantitykind for example m3-lite:Temperature,
	// m3-lite:WindSpeed
	QuantityKind string `json:"quantityKind"`

	// Unit defines unit of this channel, it is short for thingful:hasUnit, for
	// example m3-lite:PPM, m3-lite:Miles
	Unit string `json:"unit"`

	// DomainOfInterest defines domain of interest of this channel, it is short
	// for thingful:hasDomainOfInterest, for example m3-lite:Environment,
	// m3-lite:Weather
	DomainOfInterest []string `json:"domain"`

	// MeasuredBy defines type of sensor that measure this channel, it is short
	// for thingful:measuredBy, for example m3-lite:Seismometer,
	// m3-lite:LightSensor
	MeasuredBy string `json:"measuredBy"`

	// Observations contains a list of observations containing specific datapoints
	// for the Channel
	Observations []Observation `json:"observations"`
}

Channel represents an individual data channel being published by a Thing out there in the world somewhere.

type Client 

type Client interface {
	// DoHTTP is a thin wrapper around the `Do` method on `net/http.Client`. It
	// simply ensures we set the correct user agent header before making the
	// request. Note that unlike the wrapped standard library method we pass
	// context explicitly as the first parameter.
	DoHTTP(ctx context.Context, req *http.Request) (*http.Response, error)

	// Get is a helper function on the client interface for the most common usage
	// of the Client, making a simple Get request for a given URL, and returning a
	// slice of bytes containing the HTTP response. This uses client.DoHTTP
	// internally so that it ensures the correct user agent string is sent. Note
	// that unlike the wrapped standard library method we pass context explicitly
	// as the first parameter.
	GetHTTP(ctx context.Context, urlStr string) ([]byte, error)

	// PostHTTP defines a function for sending a POST request to a data
	// infrastructure in order to obtain data. Typically we send GET requests, but
	// some infrastructures may require POST requests. This function returns a
	// slice of bytes for any HTTP OK response, and an error for anything else. If
	// you require finer control of this you should use the DoHTTP method directly
	// which returns the http.Response for processing as required. Note that unlike
	// the wrapped standard library method we pass context explicitly as the first
	// parameter.
	PostHTTP(ctx context.Context, urlStr, contentType string, body io.Reader) ([]byte, error)
}

Client specifies an interface we will use for making outgoing requests within each Fetcher. Currently everything is HTTP, but that won't necessarily remain the case, so we add a layer of abstraction meaning we can add methods to our Client interface to support other protocols.

func NewClient 

func NewClient(userAgent string, timeout time.Duration) Client

NewClient is a constructor function that instantiates and returns a new client struct, ensuring it sets `timeout` properly on the returned http.Client

type Clock added in v0.6.0 

type Clock interface {
	// Now returns the current time as seen by the provider. This could be a fake
	// time for testing time providers.
	Now() time.Time
}

Clock is an interface that allows us to test time dependent code easily by providing an implementation of this interface that returns a 'canned' time rather than the real time.

func NewClock added in v0.6.0 

func NewClock() Clock

NewClock returns an instantiated DefaultClock instance.

func NewMockClock added in v0.6.0 

func NewMockClock() Clock

NewMockClock returns a new MockClock with the internal time set to the time at which the method was invoked.

func NewMockClockAt added in v0.6.1 

func NewMockClockAt(t time.Time) Clock

NewMockClockAt returns a new MockClock initialized to the passed in time variable.

func NewMockTimeProvider deprecated 

func NewMockTimeProvider(t time.Time) Clock

NewMockTimeProvider returns a mock time provider suitable for use in tests.

Deprecated: the TimeProvider type has been renamed to Clock, with matching mock instance constructors. New code should use the Clock versions.

type ContextKey 

type ContextKey string

ContextKey is a type alias used for context keys to avoid any issues wth clashing keys

type DataLicense 

type DataLicense struct {
	// Human readable name for this license
	Name string `json:"name"`

	// url to the short version of the license
	URL string `json:"id"`

	// url to the long or legal version of the license
	LegalCodeURL string `json:"legalCode,omitempty"`

	// array containing list of properties that this license permits
	Permits []Permission `json:"permits,omitempty"`

	// array containing list of properties that this license requires
	Requires []Requirement `json:"requires,omitempty"`

	// array containing list of properties that this license prohibits
	Prohibits []Prohibition `json:"prohibits,omitempty"`
}

DataLicense is struct used to describe data license

func GetDataLicense 

func GetDataLicense(licenseURL string) *DataLicense

GetDataLicense is a function that returns a copy of a data license instance to make it easier to apply a standard license to a Thing. It takes in a license URL as a parameter, and returns an instantiated DataLicense instance representing that license or nil if the license is not defined in Thingfulx.

type DefaultClock added in v0.6.0 

type DefaultClock struct{}

DefaultClock is an implementation of our TimeProvider interface that returns now directly from time.Now().

func (DefaultClock) Now added in v0.6.0 

func (c DefaultClock) Now() time.Time

Now returns the current time. In the default implementation this is just the value of time.Now()

type ErrBadData 

type ErrBadData struct {
	Msg string
}

ErrBadData is an error that can be returned for any bad data when parsing

func NewErrBadData 

func NewErrBadData(msg string) *ErrBadData

NewErrBadData is a constructor for creating ErrBadData instances

func (*ErrBadData) Error 

func (e *ErrBadData) Error() string

Error is the implementation of the default error interface. Returns the contained message

type ErrMissingConfig 

type ErrMissingConfig struct {
	Variable string
}

ErrMissingConfig is a simple struct for reporting a missing config variable. It contains a single Variable attribute which should contain the name of the missing variable.

func NewErrMissingConfig 

func NewErrMissingConfig(variable string) *ErrMissingConfig

NewErrMissingConfig is a constructor for creating ErrMissingConfig instances

func (*ErrMissingConfig) Error 

func (e *ErrMissingConfig) Error() string

Error is the implementation of the default error interface. Returns a simple string reporting the received status

type ErrUnexpectedResponse 

type ErrUnexpectedResponse struct {
	Status string
}

ErrUnexpectedResponse is a simple struct for reporting unexpected HTTP responses. It contains a single attribute which allows capturing the received HTTP status

func NewErrUnexpectedResponse 

func NewErrUnexpectedResponse(status string) *ErrUnexpectedResponse

NewErrUnexpectedResponse is a constructor for creating ErrUnexpectedResponse instances

func (*ErrUnexpectedResponse) Error 

func (e *ErrUnexpectedResponse) Error() string

Error is the implementation of the default error interface. Returns a simple string reporting the received status

type Error 

type Error string

Error is a type alias we use to create const possible error instances.

func (Error) Error 

func (e Error) Error() string

Error is the implementation of the error interface for our Error type.

type Indexer 

type Indexer interface {
	// Provider returns data for the current provider.
	Provider() *Provider

	// URLS returns the smallest set of URLs required to completely index the
	// upstream data provider. The returned values might be a single URL for hosts
	// that publish relatively few things that are all available via a single URL,
	// or it might be hundreds of thousands of values long for hosts that publish
	// data entirely on individual URLs.
	//
	// This function takes as parameters an instance of the Context interface for
	// request-scoped values or cancellation, a Client implementation which the
	// Indexer must use to make any outgoing requests, a delay Duration which
	// defines the minimum interval between consecutive requests to the
	// infrastructure. In addition the caller must pass in a channel by which the
	// function can return data to the caller. This is intended to allow the indexer
	// implementation to return chunks of data as it goes in order to support
	// infrastructures with hundreds of thousands of unique URLs to return.
	URLS(ctx context.Context, client Client, delay time.Duration, out chan<- URLData)

	// Fetch is our method that knows how to obtain raw data for an infrastructure,
	// and is therefore responsible for negotiating any authentication or protocol
	// weirdness when it comes to fetching data. It takes as parameters an instance
	// of the Context interface for request-scoped values and cancellation, the url
	// we want to get data from, a Client object that the indexer must use to
	// actually make the request, and a Clock to allow testing time related
	// functions. It returns the raw data from the infrastructure as a slice of
	// bytes for further parsing.
	Fetch(ctx context.Context, urlStr string, client Client, clock Clock) ([]byte, error)

	// Parse returns a slice of Channel objects extracted from that data source.
	// This function takes as parameters a slice of bytes representing the data
	// collected from the upstream data provider, the URL of the Channel being
	// indexed and a Clock instance used internally by the fetcher to record the
	// indexing time of the parser. This is to allow for easier testing. We
	// separate Parsing from Fetching as we have some systems that provide data to
	// us without having to be fetched from a remote HTTP source.
	Parse(rawData []byte, urlStr string, clock Clock) ([]*Channel, error)
}

Indexer is the main interface for things that know how to index and fetch resources from external data infrastructures and are thus responsible for handling any authentication requirements, parsing the returned data and generating normalized representations of IoT things to the caller. The interface attempts to be agnostic in terms of protocol so an implementing class is free to perform whatever steps required to get the data.

type IndexerBuilder 

type IndexerBuilder func() (Indexer, error)

IndexerBuilder is type definition for the constructor functions that new Indexer must implement.

type Location 

type Location struct {
	Lng       float32 `json:"long"`
	Lat       float32 `json:"lat"`
	Elevation float32 `json:"elevation,omitempty"`
}

Location is a data structure which holds an individual Thing or Observations geographic coordinates

type Metadata 

type Metadata struct {
	// Prop is an attribute used to hold the name of a property expressed as
	// either a namespaced string or a full URL.
	Prop string `json:"prop"`

	// Val is an attribute used to hold the value of the specified property.
	Val string `json:"val"`
}

Metadata is a data structure containing additional metadata about a Channel

type Observation 

type Observation struct {
	// RecordedAt is the timestamp at which the datapoint was recorded
	RecordedAt time.Time `json:"recordedAt"`

	// Location is the geographical location where the individual datapoint was
	// recorded. This is included separately from the main Location as a channel
	// may be mobile
	Location *Location `json:"location"`

	// Val contains the specific value of the datapoint expressed as a string.
	Val string `json:"value"`
}

Observation contains information about where and when the Observation was recorded.

type Permission added in v0.8.0 

type Permission string

Permission is a string type alias used to define a permission that may be granted by a license.

type Prohibition added in v0.8.0 

type Prohibition string

Prohibition is a string type alias used to define a prohibtion that may be dictated by a license.

type Provider 

type Provider struct {
	// UID contains a namespaced URI for the provider, e.g.
	// thingful:openweathermap
	UID string `json:"id"`

	// Name contains a human friendly name of the provider
	Name string `json:"name"`

	// Description contains a human description of the provider
	Description string `json:"description,omitempty"`

	// Webpage should contain a link to a high level public web site about the
	// provider.
	Webpage string `json:"webpage"`
}

Provider is a data structure containing information about the upstream data provider.

type Requirement added in v0.8.0 

type Requirement string

Requirement is a string type alias used to define a requirement that may be enforced by a license.

type TimeProvider 

type TimeProvider = Clock

TimeProvider is an alias for our Clock interface for backwards compatibility

type URLData added in v0.4.0 

type URLData struct {
	// URLS is a slice of url strings representing a chunk of candidate resource
	// URLs for the target infrastructure.
	URLS []string

	// Err is used to allow the process to signal back an error to the caller. We
	// package it with the data so we can signal an error at any point.
	Err error
}

URLData is a struct used to send back data from the URLS method. We now send back data asynchronously via a channel so we want to package our data we are actually interested in (i.e. the slice of strings), along with any error that might happen.

type Visibility 

type Visibility string

Visibility is a simple type alias for string used to represent an endpoint's visibility.

func (Visibility) String 

func (v Visibility) String() string

String is our implementation of Stringer for the Visibility type

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.