hyper

package module
v0.3.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 9, 2023 License: AGPL-3.0 Imports: 10 Imported by: 0

README

Hyper

This repository contains the code of the Hyper mesh router & overlay network system, which provides a programmable network & communication layer for distributed systems. It provides a gRPC server & client to exchange messages between different actors (hyper), a service directory (sd) that provides a public registry of signed information about actors in the system, and a TLS passthrough proxy (proxy) that enables actors to directly receive data from each other and third-parties via TLS.

Please also check out the documentation for more detailed information. Please check the docs subfolder for instructions on how to build/view the documentation locally.

Getting Started

Please ensure your Golang version is recent enough (>=1.13) before you attempt to build the software.

To build the hyper, sd and proxy binaries, simply run

make

For testing and development you'll also need TLS certificates, which you can generate via

make certs

Please note that you need openssl on your system for this to work. This will generate all required certificates and put them in the settings/dev/certs and settings/dev/test folders. Please do not use these certificates in a production setting and do not check them into version control. Warning: Running this command again will delete an re-create all certificates from scratch.

Please see below for additional dependencies you might need to install for various purposes (e.g. to recompile protobuf code).

To build the example services (e.g. the "locations" services hyper-ls) simply run

make examples

Defining Settings

The hyper binary will look for settings in a list of colon (:) separated directories as defined by the HYPER_SETTINGS environment variable (or, if it is undefined in the settings subdirectory of the current directory). The development settings include an environment-based variable HYPER_OP that allows you to use different certificates for testing. You should define these variables before running the development server:

export HYPER_SETTINGS=`readlink -f settings/dev`
export HYPER_OP=hd-1 # run server as the 'hd-1' operator

You can also source these things from the local .dev-setup script, which includes everything you need to get started:

source .dev-setup # load all development environment variables

There are also role-specific development/test settings in the settings/dev/roles directory. Those can be used to set up multiple Hyper servers and test the communication between them. Please have a a look at the integration guidelines for more information about this.

Important: The settings parser includes support for variable replacement and many other things. But with great power comes great responsibility and attack surface, so make sure you only feed trusted YAML input to it, as it is not designed to handle untrusted or potentially malicious settings.

Running The Service Directory

All Hyper servers rely on the service directory (SD) to discover each other and learn about permissions, certificates and other important settings. For development, you can either use a JSON-based service directory, or run the service directory API like this:

SD_SETTINGS=settings/dev/roles/sd-1 sd run

To initialize the service directory you can upload the JSON-based directory:

# for development
make sd-setup
# for testing
make sd-test-setup

This should give you a fully functional API-based service directory with certificate and service information.

Running The Hyper Server

To run the development Hyper server simply run (from the main directory)

HYPER_SETTINGS=settings/dev/roles/hd-1 hyper server run

This will run the Hyper server for the role hd-1 (simulating a health department in the system). For this to work you need to ensure that your GOPATH is in your PATH. This will open the JSON RPC server and (depending on the settings) also a gRPC server.

Running The Proxy Servers

To run the public and private proxy servers simply run (from the main directory)

# private proxy server
PROXY_SETTINGS=settings/dev/roles/private-proxy-1 proxy run private
# public proxy server
PROXY_SETTINGS=settings/dev/roles/private-proxy-1 proxy run public

Testing

To run the tests

make test # run normal tests
make test-races # test for race conditions

Benchmarks

To run the benchmarks

make bench

Debugging

If you're stuck debugging a problem please have a look at the debugging guidelines, which contain a few pointers that might help you to pinpoint problems in the system.

You can generate and update copyright headers as follows

make copyright

This will add appropriate headers to all Golang files. You can edit the generation and affected file types directly in the script (in .scripts). You should run this before committing code. Please note that any additional comments that appear directly at the top of the file will be replaced by this.

License

Currently this code is licensed under Affero GPL 3.0.

Development Requirements

If you make modifications to the protocol buffers (.proto files) you need to recompile them using protoc. To install this on Debian/Ubuntu systems:

sudo apt install protobuf-compiler

To generate TLS certificates for testing and development you need to have openssl installed.

Deployment

You can easily deploy the server as a service using systemd or Docker. Specific documentation coming up soon.

Feedback

If you have any questions just contact us.

Participation

We are happy about your contribution to the project! In order to ensure compliance with the licensing conditions and the future development of the project, we require a signed contributor license agreement (CLA) for all contributions in accordance with the Harmony standard. Please sign the corresponding document for natural persons or for organizations and send it to us.

Supporting organizations

Documentation

Index

Constants

View Source 
const (
	PanicLogLevel = Level(log.PanicLevel)
	FatalLogLevel = Level(log.FatalLevel)
	ErrorLogLevel = Level(log.ErrorLevel)
	WarnLogLevel  = Level(log.WarnLevel)
	InfoLogLevel  = Level(log.InfoLevel)
	DebugLogLevel = Level(log.DebugLevel)
	TraceLogLevel = Level(log.TraceLevel)
)
View Source 
const (
	NullType = 0
)

Variables

View Source 
var DirectoryQueryForm = forms.Form{
	Fields: []forms.Field{
		{
			Name: "group",
			Validators: []forms.Validator{
				forms.IsOptional{},
				forms.IsString{},
			},
		},
		{
			Name: "operator",
			Validators: []forms.Validator{
				forms.IsOptional{},
				forms.IsString{},
			},
		},
		{
			Name: "channels",
			Validators: []forms.Validator{
				forms.IsOptional{},
				forms.IsStringList{},
			},
		},
	},
}
View Source 
var IDAddressRegexp = regexp.MustCompile(`(?i)^(.*)\.([^\(\.]+)\(([^\)]+)\)$`)
View Source 
var Log = Logger{}
View Source 
var MethodNameRegexp = regexp.MustCompile(`(?i)^(.*)\.(.*)$`)
View Source 
var NoEntryFound = fmt.Errorf("no directory entry found")
View Source 
var Version = "development"

this variable gets updated using the build process

Functions

func CanCall 

func CanCall(caller, callee *DirectoryEntry, method string) bool

func GetPeerGroups 

func GetPeerGroups(entry *DirectoryEntry) []string

get all groups that may call service methods of this entry

func SetLogFormat 

func SetLogFormat(format string, service string) error

Types

type Address 

type Address struct {
	Operator string `json:"operator"`
	Method   string `json:"method"`
	ID       string `json:"id"`
}

func GetAddress 

func GetAddress(id string) (*Address, error)

type BaseChannel 

type BaseChannel struct {
	// contains filtered or unexported fields
}

func (*BaseChannel) Directory 

func (b *BaseChannel) Directory() Directory

func (*BaseChannel) DirectoryEntry 

func (b *BaseChannel) DirectoryEntry(operator string, channel string) (*DirectoryEntry, error)

func (*BaseChannel) MessageBroker 

func (b *BaseChannel) MessageBroker() MessageBroker

func (*BaseChannel) OperatorEntry 

func (b *BaseChannel) OperatorEntry(name string) (*DirectoryEntry, error)

func (*BaseChannel) SetDirectory 

func (b *BaseChannel) SetDirectory(directory Directory) error

func (*BaseChannel) SetMessageBroker 

func (b *BaseChannel) SetMessageBroker(broker MessageBroker) error

type BaseDirectory 

type BaseDirectory struct {
	Name_ string
}

func (*BaseDirectory) Name 

func (b *BaseDirectory) Name() string

type BasicMessageBroker 

type BasicMessageBroker struct {
	// contains filtered or unexported fields
}

func MakeBasicMessageBroker 

func MakeBasicMessageBroker(directory Directory) (*BasicMessageBroker, error)

func (*BasicMessageBroker) AddChannel 

func (b *BasicMessageBroker) AddChannel(channel Channel) error

func (*BasicMessageBroker) Channels 

func (b *BasicMessageBroker) Channels() []Channel

func (*BasicMessageBroker) DeliverRequest 

func (b *BasicMessageBroker) DeliverRequest(request *Request, clientInfo *ClientInfo) (*Response, error)

type ChangeRecord 

type ChangeRecord struct {
	Name      string       `json:"name"`
	Section   string       `json:"section"`
	Data      interface{}  `json:"data"`
	CreatedAt HashableTime `json:"created_at"`
}

describes a change in a specific section of the service directory

type Channel 

type Channel interface {
	Type() string
	MessageBroker() MessageBroker
	SetMessageBroker(MessageBroker) error
	CanDeliverTo(*Address) bool
	DeliverRequest(*Request) (*Response, error)
	SetDirectory(Directory) error
	Directory() Directory
	Close() error
	Open() error
}

A channel can deliver and accept message

type ChannelDefinition 

type ChannelDefinition struct {
	Name              string            `json:"name"`
	Description       string            `json:"description"`
	Maker             ChannelMaker      `json:"-"`
	SettingsValidator SettingsValidator `json:"-"`
}

type ChannelDefinitions 

type ChannelDefinitions map[string]ChannelDefinition

type ChannelMaker 

type ChannelMaker func(settings interface{}) (Channel, error)

type ChannelSettings 

type ChannelSettings struct {
	Name     string      `json:"name"`
	Type     string      `json:"type"`
	Services []string    `json:"services"`
	Settings interface{} `json:"settings"`
}

type ClientInfo 

type ClientInfo struct {
	Name  string          `json:"name"`
	Entry *DirectoryEntry `json:"entry"`
}

func (*ClientInfo) AsStruct 

func (c *ClientInfo) AsStruct() (map[string]interface{}, error)

for inclusion in protobuf... A bit dirty as we use JSON here, but it works...

type CommandsDefinition 

type CommandsDefinition struct {
	Name        string        `json:"name"`
	Description string        `json:"description"`
	Maker       CommandsMaker `json:"-"`
}

type CommandsDefinitions 

type CommandsDefinitions []CommandsDefinition

type CommandsMaker 

type CommandsMaker func(settings *Settings) ([]cli.Command, error)

type ConnectionRequest 

type ConnectionRequest struct {
	Endpoint string      `json:"endpoint"`
	Channel  string      `json:"channel"`
	Token    []byte      `json:"token"`
	Client   *ClientInfo `json:"client"`
}

type DataEntry 

type DataEntry struct {
	Type uint8
	ID   []byte
	Data []byte
}

type Datastore 

type Datastore interface {
	// Write data to the store
	Write(*DataEntry) error
	// Read data from the store
	Read() ([]*DataEntry, error)
	Init() error
}

type DatastoreDefinition 

type DatastoreDefinition struct {
	Name              string            `json:"name"`
	Description       string            `json:"description"`
	Maker             DatastoreMaker    `json:"-"`
	SettingsValidator SettingsValidator `json:"-"`
}

type DatastoreDefinitions 

type DatastoreDefinitions map[string]DatastoreDefinition

type DatastoreMaker 

type DatastoreMaker func(settings interface{}) (Datastore, error)

type DatastoreSettings 

type DatastoreSettings struct {
	Type     string      `json:"type"`
	Settings interface{} `json:"settings"`
}

type Definitions 

type Definitions struct {
	DirectoryDefinitions
	DatastoreDefinitions
	CommandsDefinitions
	ChannelDefinitions
}

func MergeDefinitions 

func MergeDefinitions(a, b Definitions) Definitions

func (Definitions) Marshal 

func (d Definitions) Marshal() map[string]interface{}

func (Definitions) MarshalJSON 

func (d Definitions) MarshalJSON() ([]byte, error)

We perform JSON marshalling manually to gain more flexibility...

type Directory 

type Directory interface {
	Entries(*DirectoryQuery) ([]*DirectoryEntry, error)
	EntryFor(string) (*DirectoryEntry, error)
	OwnEntry() (*DirectoryEntry, error)
	Name() string
}

A directory can deliver and accept message

type DirectoryDefinition 

type DirectoryDefinition struct {
	Name              string            `json:"name"`
	Description       string            `json:"description"`
	Maker             DirectoryMaker    `json:"-"`
	SettingsValidator SettingsValidator `json:"-"`
}

type DirectoryDefinitions 

type DirectoryDefinitions map[string]DirectoryDefinition

type DirectoryEntries 

type DirectoryEntries []*DirectoryEntry

type DirectoryEntry 

type DirectoryEntry struct {
	Name         string                 `json:"name"`
	Groups       []string               `json:"groups"`
	Channels     []*OperatorChannel     `json:"channels"`
	Services     []*OperatorService     `json:"services"`
	Certificates []*OperatorCertificate `json:"certificates"`
	Settings     []*OperatorSettings    `json:"settings"`
	Preferences  []*OperatorPreferences `json:"preferences"`
	Records      []*SignedChangeRecord  `json:"records"`
	Properties   *OperatorProperties    `json:"properties"`
}

func FilterDirectoryEntriesByQuery 

func FilterDirectoryEntriesByQuery(entries []*DirectoryEntry, query *DirectoryQuery) []*DirectoryEntry

helper function that can be used by directory implementations that have a list of local directory entries

func GetPeers 

func GetPeers(baseEntry *DirectoryEntry, entries []*DirectoryEntry, incomingOnly bool) []*DirectoryEntry

Return all entries that are relevant for the given base entry (i.e. that can call a service on the base entrys' endpoint or can be called by the base entrys' endpoint, respectively)

func MakeDirectoryEntry 

func MakeDirectoryEntry() *DirectoryEntry

func (*DirectoryEntry) Channel 

func (d *DirectoryEntry) Channel(channelType string) *OperatorChannel

func (*DirectoryEntry) SettingsFor 

func (d *DirectoryEntry) SettingsFor(service, operator string) *OperatorSettings

type DirectoryMaker 

type DirectoryMaker func(name string, settings interface{}) (Directory, error)

type DirectoryQuery 

type DirectoryQuery struct {
	Group    string   `json:"group"`
	Operator string   `json:"operator"`
	Channels []string `json:"channels"`
}

type DirectorySettings 

type DirectorySettings struct {
	Type     string      `json:"type"`
	Settings interface{} `json:"settings"`
}

type Error 

type Error struct {
	Code    int                    `json:"code"`
	Message string                 `json:"message"`
	Data    map[string]interface{} `json:"data,omitempty"`
}

type HashableTime 

type HashableTime struct {
	time.Time
}

func (HashableTime) HashValue 

func (h HashableTime) HashValue() interface{}

type IRISFormatter 

type IRISFormatter struct {
	// contains filtered or unexported fields
}

func (*IRISFormatter) Format 

func (c *IRISFormatter) Format(entry *log.Entry) ([]byte, error)

type Level 

type Level log.Level

func ParseLevel 

func ParseLevel(level string) (Level, error)

type Logger 

type Logger struct {
}

func (*Logger) Debug 

func (l *Logger) Debug(args ...interface{})

func (*Logger) Debugf 

func (l *Logger) Debugf(format string, args ...interface{})

func (*Logger) Error 

func (l *Logger) Error(args ...interface{})

func (*Logger) Errorf 

func (l *Logger) Errorf(format string, args ...interface{})

func (*Logger) Fatal 

func (l *Logger) Fatal(args ...interface{})

func (*Logger) Fatalf 

func (l *Logger) Fatalf(format string, args ...interface{})

func (*Logger) Info 

func (l *Logger) Info(args ...interface{})

func (*Logger) Infof 

func (l *Logger) Infof(format string, args ...interface{})

func (*Logger) SetLevel 

func (l *Logger) SetLevel(level Level)

func (*Logger) Trace 

func (l *Logger) Trace(args ...interface{})

func (*Logger) Tracef 

func (l *Logger) Tracef(format string, args ...interface{})

func (*Logger) Warning 

func (l *Logger) Warning(args ...interface{})

func (*Logger) Warningf 

func (l *Logger) Warningf(format string, args ...interface{})

type MessageBroker 

type MessageBroker interface {
	AddChannel(Channel) error
	Channels() []Channel
	DeliverRequest(*Request, *ClientInfo) (*Response, error)
}

type MetricsSettings 

type MetricsSettings struct {
	BindAddress string `json:"bind_address"`
}

type OperatorCertificate 

type OperatorCertificate struct {
	Fingerprint string `json:"fingerprint"`
	KeyUsage    string `json:"key_usage"`
}

type OperatorChannel 

type OperatorChannel struct {
	Type     string                 `json:"type"`
	Settings map[string]interface{} `json:"settings"`
}

type OperatorPreferences 

type OperatorPreferences struct {
	Operator    string                 `json:"operator"`
	Service     string                 `json:"service"`
	Environment string                 `json:"environment"`
	Preferences map[string]interface{} `json:"preferences"`
}

preferences may be set by the corresponding operator itself

type OperatorProperties 

type OperatorProperties struct {
	DisplayName string `json:"displayName"`
}

type OperatorService 

type OperatorService struct {
	Name        string           `json:"name"`
	Permissions []*Permission    `json:"permissions"`
	Methods     []*ServiceMethod `json:"methods"`
}

func ServiceFor 

func ServiceFor(entry *DirectoryEntry, method string) *OperatorService

type OperatorSettings 

type OperatorSettings struct {
	Operator    string                 `json:"operator"`
	Service     string                 `json:"service"`
	Environment string                 `json:"environment"`
	Settings    map[string]interface{} `json:"settings"`
}

settings may only be set by a directory admin

type Permission 

type Permission struct {
	Group  string   `json:"group"`
	Rights []string `json:"rights"`
}

type ProxyChannel 

type ProxyChannel interface {
	HandleConnectionRequest(address *Address, request *Request) (*Response, error)
}

type Request 

type Request struct {
	Method string                 `json:"method"`
	Params map[string]interface{} `json:"params"`
	ID     string                 `json:"id"`
}

type Response 

type Response struct {
	Result map[string]interface{} `json:"result,omitempty"`
	Error  *Error                 `json:"error,omitempty"`
	ID     *string                `json:"id"`
}

func ChannelError 

func ChannelError(id *string, message string, data map[string]interface{}) *Response

func PermissionDenied 

func PermissionDenied(id *string, message string, data map[string]interface{}) *Response

type ServiceMethod 

type ServiceMethod struct {
	Name        string              `json:"name"`
	Permissions []*Permission       `json:"permissions"`
	Parameters  []*ServiceParameter `json:"parameters"`
}

type ServiceParameter 

type ServiceParameter struct {
	Name       string              `json:"name"`
	Validators []*ServiceValidator `json:"validators"`
}

type ServiceValidator 

type ServiceValidator struct {
	Type       string                 `json:"type"`
	Parameters map[string]interface{} `json:"parameters"`
}

type Settings 

type Settings struct {
	Signing     *SigningSettings   `json:"signing"`
	Definitions *Definitions       `json:"definitions"`
	Channels    []*ChannelSettings `json:"channels"`
	Directory   *DirectorySettings `json:"directory"`
	Metrics     *MetricsSettings   `json:"metrics"`
	Name        string             `json:"name"`
}

type SettingsValidator 

type SettingsValidator func(settings map[string]interface{}) (interface{}, error)

type Signature 

type Signature struct {
	R           string `json:"r"`
	S           string `json:"s"`
	Certificate string `json:"c"`
}

type SignedChangeRecord 

type SignedChangeRecord struct {
	ParentHash string        `json:"parent_hash"`
	Hash       string        `json:"hash"`
	Signature  *Signature    `json:"signature"`
	Record     *ChangeRecord `json:"record"`
}

type SignedData 

type SignedData struct {
	Signature *Signature  `json:"signature"`
	Data      interface{} `json:"data"`
}

type SigningSettings 

type SigningSettings struct {
	Name                           string   `json:"name"`
	CACertificateFile              string   `json:"ca_certificate_file"`
	CAIntermediateCertificateFiles []string `json:"ca_intermediate_certificate_files"`
	CertificateFile                string   `json:"certificate_file"`
	KeyFile                        string   `json:"key_file"`
}

type WritableDirectory 

type WritableDirectory interface {
	Directory
	// required for submitting change records
	Tip() (*SignedChangeRecord, error)
	Submit([]*SignedChangeRecord) error
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
helpers
hyper
proxy
sd
helpers
sd
helpers
fixtures

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.