twins

README

Twins

Service twins is used for CRUD and update of digital twins. Twin is a semantic representation of a real world data system consisting of data producers and consumers. It stores the sequence of attribute based definitions of a system and refers to a time series of definition based states that store the system historical data.

Configuration

The service is configured using the environment variables presented in the following table. Note that any unset variables will be replaced with their default values.

Variable	Description	Default
MG_TWINS_LOG_LEVEL	Log level for twin service (debug, info, warn, error)	info
MG_TWINS_HTTP_PORT	Twins service HTTP port	9018
MG_TWINS_SERVER_CERT	Path to server certificate in PEM format
MG_TWINS_SERVER_KEY	Path to server key in PEM format
MG_JAEGER_URL	Jaeger server URL	http://jaeger:14268/api/traces
MG_TWINS_DB	Database name	magistrala
MG_TWINS_DB_HOST	Database host address	localhost
MG_TWINS_DB_PORT	Database host port	27017
MG_THINGS_STANDALONE_ID	User ID for standalone mode (no gRPC communication with users)
MG_THINGS_STANDALONE_TOKEN	User token for standalone mode that should be passed in auth header
MG_TWINS_CLIENT_TLS	Flag that indicates if TLS should be turned on	false
MG_TWINS_CA_CERTS	Path to trusted CAs in PEM format
MG_TWINS_CHANNEL_ID	Message broker notifications channel ID
MG_MESSAGE_BROKER_URL	Magistrala Message broker URL	nats://localhost:4222
MG_AUTH_GRPC_URL	Auth service gRPC URL	localhost:7001
MG_AUTH_GRPC_TIMEOUT	Auth service gRPC request timeout in seconds	1s
MG_TWINS_CACHE_URL	Cache database URL	redis://localhost:6379/0
MG_SEND_TELEMETRY	Send telemetry to magistrala call home server	true

Deployment

The service itself is distributed as Docker container. Check the twins service section in docker-compose file to see how service is deployed.

To start the service outside of the container, execute the following shell script:

# download the latest version of the service
go get github.com/andychao217/magistrala

cd $GOPATH/src/github.com/andychao217/magistrala

# compile the twins
make twins

# copy binary to bin
make install

# set the environment variables and run the service
MG_TWINS_LOG_LEVEL=[Twins log level] \
MG_TWINS_HTTP_PORT=[Service HTTP port] \
MG_TWINS_SERVER_CERT=[String path to server cert in pem format] \
MG_TWINS_SERVER_KEY=[String path to server key in pem format] \
MG_JAEGER_URL=[Jaeger server URL] \
MG_TWINS_DB=[Database name] \
MG_TWINS_DB_HOST=[Database host address] \
MG_TWINS_DB_PORT=[Database host port] \
MG_THINGS_STANDALONE_EMAIL=[User email for standalone mode (no gRPC communication with auth)] \
MG_THINGS_STANDALONE_TOKEN=[User token for standalone mode that should be passed in auth header] \
MG_TWINS_CLIENT_TLS=[Flag that indicates if TLS should be turned on] \
MG_TWINS_CA_CERTS=[Path to trusted CAs in PEM format] \
MG_TWINS_CHANNEL_ID=[Message broker notifications channel ID] \
MG_MESSAGE_BROKER_URL=[Magistrala Message broker URL] \
MG_AUTH_GRPC_URL=[Auth service gRPC URL] \
MG_AUTH_GRPC_TIMEOUT=[Auth service gRPC request timeout in seconds] \
MG_TWINS_CACHE_URL=[Cache database URL] \
$GOBIN/magistrala-twins

Usage

Starting twins service

The twins service publishes notifications on a Message broker subject of the format channels.<MG_TWINS_CHANNEL_ID>.messages.<twinID>.<crudOp>, where crudOp stands for the crud operation done on twin - create, update, delete or retrieve - or state - save state. In order to use twin service notifications, one must inform it - via environment variables - about the Magistrala channel used for notification publishing. You must use an already existing channel, since you cannot know in advance or set the channel ID (Magistrala does it automatically).

To set the environment variable, please go to .env file and set the following variable:

MG_TWINS_CHANNEL_ID=

with the corresponding values of the desired channel. If you are running magistrala natively, than do the same thing in the corresponding console environment.

For more information about service capabilities and its usage, please check out the API documentation.

Documentation

Overview

Package twins contains the domain concept definitions needed to support Magistrala twins service functionality. Twin is a digital representation of a real world data system consisting of data producers and consumers. It stores the sequence of attribute based definitions of a data system and refers to a time series of definition based states that store the system historical data.

Index

• Constants
• type Attribute
• type Definition
• type Metadata
• type Page
• type PageMetadata
• type Service
  • func New(publisher messaging.Publisher, auth magistrala.AuthServiceClient, ...) Service
• type State
• type StateRepository
• type StatesPage
• type Twin
• type TwinCache
• type TwinRepository

Constants

const (
	SubtopicWildcard = ">"
)

Types

type Attribute

type Attribute struct {
	Name         string `json:"name"`
	Channel      string `json:"channel"`
	Subtopic     string `json:"subtopic"`
	PersistState bool   `json:"persist_state"`
}

Attribute stores individual attribute data.

type Definition

type Definition struct {
	ID         int         `json:"id"`
	Created    time.Time   `json:"created"`
	Attributes []Attribute `json:"attributes"`
	Delta      int64       `json:"delta"`
}

Definition stores entity's attributes.

type Metadata

type Metadata map[string]interface{}

Metadata stores arbitrary twin data.

type Page

type Page struct {
	PageMetadata
	Twins []Twin
}

Page contains page related metadata as well as a list of twins that belong to this page.

type PageMetadata

type PageMetadata struct {
	Total  uint64
	Offset uint64
	Limit  uint64
}

PageMetadata contains page metadata that helps navigation.

type Service

type Service interface {
	// AddTwin adds new twin related to user identified by the provided key.
	AddTwin(ctx context.Context, token string, twin Twin, def Definition) (tw Twin, err error)

	// UpdateTwin updates twin identified by the provided Twin that
	// belongs to the user identified by the provided key.
	UpdateTwin(ctx context.Context, token string, twin Twin, def Definition) (err error)

	// ViewTwin retrieves data about twin with the provided
	// ID belonging to the user identified by the provided key.
	ViewTwin(ctx context.Context, token, twinID string) (tw Twin, err error)

	// RemoveTwin removes the twin identified with the provided ID, that
	// belongs to the user identified by the provided key.
	RemoveTwin(ctx context.Context, token, twinID string) (err error)

	// ListTwins retrieves data about subset of twins that belongs to the
	// user identified by the provided key.
	ListTwins(ctx context.Context, token string, offset uint64, limit uint64, name string, metadata Metadata) (Page, error)

	// ListStates retrieves data about subset of states that belongs to the
	// twin identified by the id.
	ListStates(ctx context.Context, token string, offset uint64, limit uint64, twinID string) (StatesPage, error)

	// SaveStates persists states into database
	SaveStates(ctx context.Context, msg *messaging.Message) error
}

Service specifies an API that must be fullfiled by the domain service implementation, and all of its decorators (e.g. logging & metrics).

func New

func New(publisher messaging.Publisher, auth magistrala.AuthServiceClient, twins TwinRepository, tcache TwinCache, sr StateRepository, idp magistrala.IDProvider, chann string, logger *slog.Logger) Service

New instantiates the twins service implementation.

type State

type State struct {
	TwinID     string
	ID         int64
	Definition int
	Created    time.Time
	Payload    map[string]interface{}
}

State stores actual snapshot of entity's values.

type StateRepository

type StateRepository interface {
	// Save persists the state
	Save(ctx context.Context, state State) error

	// Update updates the state
	Update(ctx context.Context, state State) error

	// Count returns the number of states related to state
	Count(ctx context.Context, twin Twin) (int64, error)

	// RetrieveAll retrieves the subset of states related to twin specified by id
	RetrieveAll(ctx context.Context, offset uint64, limit uint64, twinID string) (StatesPage, error)

	// RetrieveLast retrieves the last saved state
	RetrieveLast(ctx context.Context, twinID string) (State, error)
}

StateRepository specifies a state persistence API.

type StatesPage

type StatesPage struct {
	PageMetadata
	States []State
}

StatesPage contains page related metadata as well as a list of twins that belong to this page.

type Twin

type Twin struct {
	Owner       string
	ID          string
	Name        string
	Created     time.Time
	Updated     time.Time
	Revision    int
	Definitions []Definition
	Metadata    Metadata
}

Twin is a Magistrala data system representation. Each twin is owned by a single user, and is assigned with the unique identifier.

type TwinCache

type TwinCache interface {
	// Save stores twin ID as element of channel-subtopic keyed set and vice versa.
	Save(ctx context.Context, twin Twin) error

	// SaveIDs stores twin IDs as elements of channel-subtopic keyed set and vice versa.
	SaveIDs(ctx context.Context, channel, subtopic string, twinIDs []string) error

	// Update updates update twin id and channel-subtopic attributes mapping
	Update(ctx context.Context, twin Twin) error

	// ID returns twin IDs for given attribute.
	IDs(ctx context.Context, channel, subtopic string) ([]string, error)

	// Removes twin from cache based on twin id.
	Remove(ctx context.Context, twinID string) error
}

TwinCache contains twin caching interface.

type TwinRepository

type TwinRepository interface {
	// Save persists the twin
	Save(ctx context.Context, twin Twin) (string, error)

	// Update performs an update to the existing twin. A non-nil error is
	// returned to indicate operation failure.
	Update(ctx context.Context, twin Twin) error

	// RetrieveByID retrieves the twin having the provided identifier.
	RetrieveByID(ctx context.Context, twinID string) (Twin, error)

	// RetrieveByAttribute retrieves twin ids whose definition contains
	// the attribute with given channel and subtopic
	RetrieveByAttribute(ctx context.Context, channel, subtopic string) ([]string, error)

	// RetrieveAll retrieves the subset of twins owned by the specified user.
	RetrieveAll(ctx context.Context, owner string, offset, limit uint64, name string, metadata Metadata) (Page, error)

	// Remove removes the twin having the provided identifier.
	Remove(ctx context.Context, twinID string) error
}

TwinRepository specifies a twin persistence API.