thingfulx

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 1, 2019 License: MIT Imports: 8 Imported by: 2

README

thingfulx

Library providing an interface for Thingful fetcher and parser interfaces to implement to provide new upstream source parsing capability.

wercker status GoDoc Go Report Card

Documentation

Overview

Package thingfulx provides a compatibility layer for implementing fetchers for use with the Thingful indexing system. New upstream providers can be added by implementing the core Fetcher interface contained within this library. These libraries can then be easily imported and used by Thingful.

Index

Constants

View Source 
const (
	// Home is the exported const representing the Thingful Home category
	Home = Category("http://purl.org/iot/vocab/thingful#Home")

	// Environment is the exported const representing the Thingful Environment
	// category
	Environment = Category("http://purl.org/iot/vocab/thingful#Environment")

	// Transport is the exported const representing the Thingful Transport
	// category
	Transport = Category("http://purl.org/iot/vocab/thingful#Transport")

	// Health is the exported const representing the Thingful Health category
	Health = Category("http://purl.org/iot/vocab/thingful#Health")

	// Energy is the exported const representing the Thingful Energy category
	Energy = Category("http://purl.org/iot/vocab/thingful#Energy")

	// Flora is the exported const representing the Thingful Flora and Fauna
	// category
	Flora = Category("http://purl.org/iot/vocab/thingful#FloraFauna")

	// Experiment is the exported const representing the Thingful Experiment
	// category
	Experiment = Category("http://purl.org/iot/vocab/thingful#Experiment")

	// Miscellaneous is the exported const representing the Thingful
	// Miscellaneous category
	Miscellaneous = Category("http://purl.org/iot/vocab/thingful#Miscellaneous")
)
View Source 
const (

	// Reproduction is "making multiple copies"
	Reproduction = "https://creativecommons.org/ns#Reproduction"

	// Distribution is "distribution, public display, and publicly performance"
	Distribution = "https://creativecommons.org/ns#Distribution"

	// DerivativeWorks is "distribution of derivative works"
	DerivativeWorks = "https://creativecommons.org/ns#DerivativeWorks"

	// Sharing is "permits commercial derivatives, but only non-commercial
	// distribution"
	Sharing = "https://creativecommons.org/ns#Sharing"

	// Notice is "copyright and license notices be kept intact"
	Notice = "https://creativecommons.org/ns#Notice"

	// Attribution is "credit be given to copyright holder and/or author"
	Attribution = "https://creativecommons.org/ns#Attribution"

	// ShareAlike is "derivative works be licensed under the same terms or
	// compatible terms as the original work"
	ShareAlike = "https://creativecommons.org/ns#ShareAlike"

	// SourceCode is "source code (the preferred form for making modifications)
	// must be provided when exercising some rights granted by the license"
	SourceCode = "https://creativecommons.org/ns#SourceCode"

	// Copyleft is "derivative and combined works must be licensed under
	// specified terms, similar to those on the original work"
	Copyleft = "https://creativecommons.org/ns#Copyleft"

	// LesserCopyleft is "derivative works must be licensed under specified
	// terms, with at least the same conditions as the original work;
	// combinations with the work may be licensed under different terms"
	LesserCopyleft = "https://creativecommons.org/ns#LesserCopyleft"

	// CommercialUse is "exercising rights for commercial purposes"
	CommercialUse = "https://creativecommons.org/ns#CommercialUse"

	// HighIncomeNationUse is "use in a non-developing country"
	HighIncomeNationUse = "https://creativecommons.org/ns#HighIncomeNationUse"

	// CC0V1URL is the URL identifier for CC0 1.0 Universal License
	CC0V1URL = "https://creativecommons.org/publicdomain/zero/1.0/"

	// CCByV3URL is the URL identifier for Creative Commons Attribution 3.0
	// Unported License
	CCByV3URL = "https://creativecommons.org/licenses/by/3.0/"

	// CCBySAV4URL is the URL identifier for Creative Commons
	// Attribution-ShareAlike 4.0 International
	CCBySAV4URL = "https://creativecommons.org/licenses/by-sa/4.0/"

	// CCByV4URL is the URL identifier for the Creative Commons Attribution 4.0
	// License
	CCByV4URL = "https://creativecommons.org/licenses/by/4.0/"

	// CCByNCV3URL is the URL identifier for Creative Commons
	// Atribution-NonCommercial 3.0 License
	CCByNCV3URL = "https://creativecommons.org/licenses/by-nc/3.0/"

	// CCByNDV3URL is the URL identifier for Creative Commons
	// Attribution-NoDerivs 3.0 License.
	CCByNDV3URL = "https://creativecommons.org/licenses/by-nd/3.0/"

	// CCByNCSAV3URL is the URL identifier for Creative Commons
	// Attribution-NonCommercial-Sharealike 3.0 License.
	CCByNCSAV3URL = "https://creativecommons.org/licenses/by-nc-sa/3.0/"

	// OGLV2URL is the URL identifier for the Open Government Licence version
	// 2.0 License.
	OGLV2URL = "http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/"

	// OGLV3URL is the URL identifier for the Open Government Licence version
	// 3.0 License.
	OGLV3URL = "http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/"

	// PDDLV1URL is the URL identifier for the Open Data Commons Public Domain
	// Dedication and License.
	PDDLV1URL = "https://opendatacommons.org/licenses/pddl/1.0/"

	// ODCByV1URL is the URL identifier for the Open Data Commons Attribution
	// License.
	ODCByV1URL = "https://opendatacommons.org/licenses/by/1.0/"

	// ODbLV1URL is a the URL identifier for the Open Data Commons Open Database
	// License.
	ODbLV1URL = "https://opendatacommons.org/licenses/odbl/1.0/"

	// IODLV2URL is the URL identifier for the Italian Open Data License
	IODLV2URL = "https://www.dati.gov.it/content/italian-open-data-license-v20"

	// SGODLV1 is the URL identifier for the Singapore Open Data License
	SGODLV1 = "http://www.nea.gov.sg/open-data-licence/"
)
View Source 
const (
	// ErrInvalidData is a generic error that can be returned if retrieved data
	// is invalid in any way
	ErrInvalidData = Error("thingfulx: invalid data")

	// ErrInvalidLocation is an error indicating that there is a problem with the
	// location of a fetched thing
	ErrInvalidLocation = Error("thingfulx: missing or invalid location data")

	// ErrInvalidTime is an error indicating that there is a problem with parsing
	// the time value from a retrieved thing
	ErrInvalidTime = Error("thingfulx: invalid time data")

	// ErrInvalidURL is an error used to flag an invalid url
	ErrInvalidURL = Error("thingfulx: invalid url")

	// ErrNotFound is an error that can be used for a 404 not found response
	ErrNotFound = Error("thingfulx: unexpected HTTP response code, got 404")
)
View Source 
const (
	// Closed is the exported const representing a private endpoint requiring
	// authentication to use.
	Closed = Visibility("closed")

	// Shared is the exported const representing an endpoint that requires
	// authentication but is free to use once credentials are supplied.
	Shared = Visibility("shared")

	// Open is the exported const representing an open endpoint, i.e. requiring
	// no authentication to use.
	Open = Visibility("open")
)
View Source 
const (
	// ClientToken is the key we use when setting a token on a context when making
	// a request. Note it is the responsibility of the implementing indexer to
	// determine how to send this token value to the data infrastructure (i.e.
	// Authorization header, API Key in the query string or a header).
	ClientToken = ContextKey("client-token")
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Category 

type Category string

Category is our type alias for thingful category instances

func Categorise 

func Categorise(str string) Category

Categorise is an extract of Thingful's original quick'n'nasty categorisation logic.

func (Category) String 

func (c Category) String() string

String is our implementation of Stringer for the Category type

type Channel 

type Channel struct {
	ID           string        `json:"id"`
	Metadata     []Metadata    `json:"metadata"`
	Observations []Observation `json:"observations"`

	// dataType defines the type of data of this channel, eg; string, double, date-time
	Type schema.DataType `json:"dataType"`

	// QuantityKind defines measurement type of quantity kind of this channel,
	// in Ontology PoV, Channel is type QuantityKind, so type of channel can be anything that classified as quantitykind
	// for example m3-lite:Temperature, m3-lite:WindSpeed
	QuantityKind string `json:"quantityKind"`

	// Unit defines unit of this channel, it is short for thingful:hasUnit,
	// for example m3-lite:PPM, m3-lite:Miles
	Unit string `json:"unit"`

	// DomainOfInterest defines domain of interest of this channel, it is short for thingful:hasDomainOfInterest,
	// for example m3-lite:Environment, m3-lite:Weather
	DomainOfInterest []string `json:"domain"`

	// MeasuredBy defines type of sensor that measure this channel, it is short for thingful:measuredBy,
	// for example m3-lite:Seismometer, m3-lite:LightSensor
	MeasuredBy string `json:"measuredBy"`
}

Channel represents an individual data channel being published by a Thing out there in the world somewhere.

type Client 

type Client interface {
	// DoHTTP is a thin wrapper around the `Do` method on `net/http.Client`. It
	// simply ensures we set the correct user agent header before making the
	// request. Note that unlike the wrapped standard library method we pass
	// context explicitly as the first parameter.
	DoHTTP(ctx context.Context, req *http.Request) (*http.Response, error)

	// Get is a helper function on the client interface for the most common usage
	// of the Client, making a simple Get request for a given URL, and returning a
	// slice of bytes containing the HTTP response. This uses client.DoHTTP
	// internally so that it ensures the correct user agent string is sent. Note
	// that unlike the wrapped standard library method we pass context explicitly
	// as the first parameter.
	GetHTTP(ctx context.Context, urlStr string) ([]byte, error)

	// PostHTTP defines a function for sending a POST request to a data
	// infrastructure in order to obtain data. Typically we send GET requests, but
	// some infrastructures may require POST requests. This function returns a
	// slice of bytes for any HTTP OK response, and an error for anything else. If
	// you require finer control of this you should use the DoHTTP method directly
	// which returns the http.Response for processing as required. Note that unlike
	// the wrapped standard library method we pass context explicitly as the first
	// parameter.
	PostHTTP(ctx context.Context, urlStr, contentType string, body io.Reader) ([]byte, error)
}

Client specifies an interface we will use for making outgoing requests within each Fetcher. Currently everything is HTTP, but that won't necessarily remain the case, so we add a layer of abstraction meaning we can add methods to our Client interface to support other protocols.

func NewClient 

func NewClient(userAgent string, timeout time.Duration) Client

NewClient is a constructor function that instantiates and returns a new client struct, ensuring it sets `timeout` properly on the returned http.Client

type ContextKey 

type ContextKey string

ContextKey is a type alias used for context keys to avoid any issues wth clashing keys

type DataLicense 

type DataLicense struct {
	// Human readable name for this license
	Name string `json:"name"`

	// url to the short version of the license
	URL string `json:"id"`

	// url to the long or legal version of the license
	LegalCodeURL string `json:"legalCode,omitempty"`

	// array containing list of properties that this license permits
	Permits []string `json:"permits,omitempty"`

	// array containing list of properties that this license requires
	Requires []string `json:"requires,omitempty"`

	// array containing list of properties that this license prohibits
	Prohibits []string `json:"prohibits,omitempty"`
}

DataLicense is struct used to describe data license

func GetDataLicense 

func GetDataLicense(licenseURL string) *DataLicense

GetDataLicense is a function that returns a copy of a data license instance to make it easier to apply a standard license to a Thing. It takes in a license URL as a parameter, and returns an instantiated DataLicense instance representing that license or nil if the license is not defined in Thingfulx.

type DefaultTimeProvider 

type DefaultTimeProvider struct{}

DefaultTimeProvider is an implementation of our TimeProvider interface that returns now directly from time.Now().

func (DefaultTimeProvider) Now 

func (t DefaultTimeProvider) Now() time.Time

Now returns the current time. In the default implementation this is just the value of time.Now()

type Endpoint 

type Endpoint struct {
	URL         string `json:"url"`
	ContentType string `json:"contentType"`
}

Endpoint is a data structure containing the actual URL we go to fetch data from for this thing, the page MIME type and information about the endpoint authentication. The URL value must be unique for every thing indexed. We can achieve this for things that don't make a real unique URL available by adding a synthetic identifier to the URL string, i.e. http://example.com?city=Indiana#section_id=1234

type ErrBadData 

type ErrBadData struct {
	Msg string
}

ErrBadData is an error that can be returned for any bad data when parsing

func NewErrBadData 

func NewErrBadData(msg string) *ErrBadData

NewErrBadData is a constructor for creating ErrBadData instances

func (*ErrBadData) Error 

func (e *ErrBadData) Error() string

Error is the implementation of the default error interface. Returns the contained message

type ErrMissingConfig 

type ErrMissingConfig struct {
	Variable string
}

ErrMissingConfig is a simple struct for reporting a missing config variable. It contains a single Variable attribute which should contain the name of the missing variable.

func NewErrMissingConfig 

func NewErrMissingConfig(variable string) *ErrMissingConfig

NewErrMissingConfig is a constructor for creating ErrMissingConfig instances

func (*ErrMissingConfig) Error 

func (e *ErrMissingConfig) Error() string

Error is the implementation of the default error interface. Returns a simple string reporting the received status

type ErrUnexpectedResponse 

type ErrUnexpectedResponse struct {
	Status string
}

ErrUnexpectedResponse is a simple struct for reporting unexpected HTTP responses. It contains a single attribute which allows capturing the received HTTP status

func NewErrUnexpectedResponse 

func NewErrUnexpectedResponse(status string) *ErrUnexpectedResponse

NewErrUnexpectedResponse is a constructor for creating ErrUnexpectedResponse instances

func (*ErrUnexpectedResponse) Error 

func (e *ErrUnexpectedResponse) Error() string

Error is the implementation of the default error interface. Returns a simple string reporting the received status

type Error 

type Error string

Error is a type alias we use to create const possible error instances.

func (Error) Error 

func (e Error) Error() string

Error is the implementation of the error interface for our Error type.

type Indexer 

type Indexer interface {
	// UID returns a unique identifier for the data infrastructure indexed by
	// this Indexer. Good practice for this to be the same as the package name.
	UID() string

	// URLS returns the smallest set of URLs required to completely index the
	// upstream data provider. The returned values might be a single URL for hosts
	// that publish relatively few things that are all available via a single URL,
	// or it might be hundreds of thousands of values long for hosts that publish
	// data entirely on individual URLs.
	//
	// This function takes as parameters an instance of the Context interface for
	// request-scoped values or cancellation, a Client implementation which the
	// Indexer must use to make any outgoing requests, a delay Duration which
	// defines the minimum interval between consecutive requests to the
	// infrastructure. In addition the caller must pass in a channel by which the
	// function can return data to the caller. This is intended to allow the indexer
	// implementation to return chunks of data as it goes in order to support
	// infrastructures with hundreds of thousands of unique URLs to return.
	URLS(ctx context.Context, client Client, delay time.Duration, out chan<- []URLData)

	// Fetch is our method that knows how to obtain raw data for an infrastructure,
	// and is therefore responsible for negotiating any authentication or protocol
	// weirdness when it comes to fetching data. It takes as parameters an instance
	// of the Context interface for request-scoped values and cancellation, the url
	// we want to get data from, a Client object that the indexer must use to
	// actually make the request. It returns the raw data from the infrastructure
	// as a slice of bytes for further parsing.
	Fetch(ctx context.Context, urlStr string, client Client) ([]byte, error)

	// Parse returns a slice of Thing objects extracted from that data source. This
	// function takes as parameters a slice of bytes representing the data
	// collected from the upstream data provider, the URL of the Thing being
	// indexed and a TimeProvider instance used internally by the fetcher to record
	// the indexing time of the parser. This is to allow for easier testing. We
	// separate Parsing from Fetching as we have some systems that provide data to
	// us without having to be fetched from a remote HTTP source.
	Parse(rawData []byte, urlStr string, timeProvider TimeProvider) ([]Thing, error)
}

Indexer is the main interface for things that know how to index and fetch resources from external data infrastructures and are thus responsible for handling any authentication requirements, parsing the returned data and generating normalized representations of IoT things to the caller. The interface attempts to be agnostic in terms of protocol so an implementing class is free to perform whatever steps required to get the data.

type IndexerBuilder 

type IndexerBuilder func() (Indexer, error)

IndexerBuilder is type definition for the constructor functions that new Indexer must implement.

type Location 

type Location struct {
	Lng       float64 `json:"long"`
	Lat       float64 `json:"lat"`
	Elevation float64 `json:"elevation,omitempty"`
}

Location is a data structure which holds an individual Thing or Observations geographic coordinates

type Metadata 

type Metadata struct {
	Prop string `json:"prop"`
	Val  string `json:"val"`
}

Metadata is a data structure containing additional metadata about the Thing. A key requirement for Metadata at the Thing level is that it must contain a 'thingful:Category' property whose value must be a Category.Name as well as a 'thingful:CategoryUID' whose value must me a Category.UID. These two metadata object are required for correctly displaying Things on the thingful.net map. The Metadata object must also contain information about the data visibility - see http://theodi.org/data-spectrum for more info.

type MockTimeProvider 

type MockTimeProvider struct {
	InternalTime time.Time
}

MockTimeProvider is an implementation of the TimeProvider interface that gives back a canned time for all calls to Now(). Intended for use in testing Fetchers.

func (MockTimeProvider) Now 

func (t MockTimeProvider) Now() time.Time

Now returns the internal time of the MockTimeProvider.

type Observation 

type Observation struct {
	RecordedAt time.Time `json:"recordedAt"`
	Location   *Location `json:"location"`
	Val        string    `json:"value"`
}

Observation contains information about where and when the Observation was recorded.

type Provider 

type Provider struct {
	ID          string `json:"id"`
	Name        string `json:"name"`
	Description string `json:"description,omitempty"`
	URL         string `json:"url"`
}

Provider is a data structure containing information about the upstream data provider.

type Thing 

type Thing struct {
	// Title is a short descriptive label for the thing
	Title string `json:"title"`

	// Description is a long form description about the thing
	Description string `json:"description"`

	// IndexedAt is a Time object indicating the timestamp when the thing was
	// indexed by Thingful
	IndexedAt time.Time `json:"indexedAt"`

	// Webpage is a URL to a human intelligible page about the thing. This
	// doesn't have to be a unique page for the thing as typically one might not
	// be available. In that case, this could be a link to a page describing the
	// type of thing, or just a link to the main homepage of the project.
	Webpage string `json:"webpage"`

	// DataURL is an embedded data structure which describes the actual data entry point.
	Endpoint *Endpoint `json:"endpoint"`

	// Metadata is a data structure containing additional metadata about the Thing.
	Metadata []Metadata `json:"metadata"`

	// Location is an embedded data structure which describes location data.
	Location *Location `json:"location"`

	// Provider is an embedded data structure which describes the data provider.
	Provider *Provider `json:"provider"`

	// Channels are a slice of Channel objects which capture the actual data of
	// the thing
	Channels []Channel `json:"channels"`

	// Visibility indicates generally what type of access is possible to the
	// Thing using the data spectrum classifications from the ODI: Open, Shared
	// or Closed
	Visibility Visibility `json:"visibility"`

	// ThingType indicates what type is this thing, it is short for thingful:isThingType
	// the most generic thing type is thingful:ConnectedDevice
	ThingType string `json:"thingType"`

	// Category indicates what type this thing belongs to, it is short for
	// thingful:hasCategory This is mainly used by thingful.net.  The value will
	// be those defined by m3-lite's domain of interest, plus some additionally
	// defined by thingful
	Category Category `json:"category"`

	// DataLicense indicates what of data license this thing has, it is short for
	// cc:license This could be used to decide what can we do with this thing's
	// data It takes DataLicense struct which can be manually defined or use
	// predefined ones
	DataLicense *DataLicense `json:"license"`

	// AttributionName indicates name ot the author of the data (if required)
	// it is short for cc:attributionName
	AttributionName string `json:"attributionName,omitempty"`

	// AttributionURL indicates name ot the author of the data (if required)
	// it is short for cc:attributionURL
	AttributionURL string `json:"attributionURL,omitempty"`

	// UpdateInterval indicates how often this resource should get updated (in
	// seconds)
	UpdateInterval int `json:"updateInterval,omitempty"`
}

Thing is the top level data structure representing an individual thing as indexed on some upstream data provider.

func (*Thing) MetaVal 

func (t *Thing) MetaVal(prop string) string

MetaVal returns the first value for the given property in the Thing's metadata. If the property is not found this returns the empty string. If you need to access multiple values with the same property key, you should use the map directly.

type TimeProvider 

type TimeProvider interface {
	// Now returns the current time as seen by the provider. This could be a fake
	// time for testing time providers.
	Now() time.Time
}

TimeProvider is an interface that allows us to test time dependent code easily by providing an implementation of this interface that returns a 'canned' time rather than the real time.

func NewMockTimeProvider 

func NewMockTimeProvider(t time.Time) TimeProvider

NewMockTimeProvider is a convenience helper that returns a new MockTimeProvider initialized to the specified time.

func NewTimeProvider 

func NewTimeProvider() TimeProvider

NewTimeProvider returns an instantiated DefaultTimeProvider instance.

type URLData added in v0.4.0 

type URLData struct {
	// URLS is a slice of url strings representing a chunk of candidate resource
	// URLs for the target infrastructure.
	URLS []string

	// Err is used to allow the process to signal back an error to the caller. We
	// package it with the data so we can signal an error at any point.
	Err error
}

URLData is a struct used to send back data from the URLS method. We now send back data asynchronously via a channel so we want to package our data we are actually interested in (i.e. the slice of strings), along with any error that might happen.

type Visibility 

type Visibility string

Visibility is a simple type alias for string used to represent an endpoint's visibility.

func (Visibility) String 

func (v Visibility) String() string

String is our implementation of Stringer for the Visibility type

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.