xconnect

README

xconnect - declaring microservices connectivity

Xconnect is a structure in YAML that describes how a service can accept connections and what connections a service needs to operate. With this information, a static overview of the landscape of intercconnected services can be constructed.

specification in YAML

how does it work

Every application/service uses some kind of configuration to specifiy what other services is connects to. This can be a database, another service, a filesystem etc. By standardising a part of that configuration, a tool can extract connectivity information from that configuration. This is called the xconnect section.

xonnect section

The xconnect section of a configuration (now YAML only), consists of 3 parts:

• description of what a service can provide by connecting to it, the listen part.
• description of what a service needs to consume or produce to using a connection, the connect part
• metadata about the service such as version, opex (ownership) and custom tags.

Not all application configuration is related to connectivity ; the section for connectivity can be part of it (embedding). So instead of keeping connection related information separate from the rest of the configuration, with the inevitable effect of becoming out of date, the xconnect section should be integrated in the complete configuration. The actual format of the xconnect data is free-form YAML, meaning that users are free to add their own service metadata if desired.

Example

doc-field: doc-value

xconnect:
  extra-field: extra-value
  
  meta:
    # name for discovery
    name: account-service
    # tagged version of the implementation
    version: v1.2.3
    # team that owns the code and operates it
    opex: team accounts
    tags:
      - account
      - registration
      - search    
  listen:
    api:
      host: localhost
      protocol: grpc
      port: 9443
    web:
      host: localhost
      protocol: http
      secure: true
      port: 443
  connect:
    some-db:
      kind: db
      url: jdbc:postgresql://localhost:5432/postgres?reWriteBatchedInserts=true
    some-cache:
      host: #REDIS_IP
      port: 6379
      kind: db
    variant-publish:
      kind: gcp.pubsub
      resource: VariantToAssortment_Push_v1-topic          
    variant-pull:
      kind: gcp.pubsub:
      resource: subscription: Variant_v1-subscription
        test:
          topic: Variant_v1-topic

Use as Go package

import ( "github.com/emicklei/xconnect" )

doc, err := GetConfig("ENV_VAR", "config.yaml") // env or file

version := doc.XConnect.Meta.Version
// alternative: doc.FindString("xconnect/meta/version")

webPort := doc.Xconnect.Listen["web"].Port
// alternative: doc.FindInt("xconnect/listen/web/port")

variantPullTestTopic := doc.FindString("xconnect/connect/variant-pull/resource/test/topic")

Sprint Boot application configration

A Spring configuration needs it own root element in a YAML file. To use an xconnect section in this file, relevant property values should be referenced using the ${..} notation supported by Spring.

xconnect:
  meta: 
  listen:
  connect:
    some-db:
      url: jdbc:postgresql://localhost:5432/postgres?reWriteBatchedInserts=true

spring:
  datasource:
    # use a reference to the actual value, within this document
    url: ${xconnect.connect.some-db.url}

extract

To extract the xconnect section using the command line tool:

xconnect -input application.yml -target file://xconnect-from-application.yml

Kubernetes configration (ConfigMap)

A Kubernetes configuration as its defined structure in a YAML file. To use an xconnect section in this file, its content must be inside the data field.

apiVersion: v1
data:
    xconnect:
        meta: 
            ...
        listen:
            ...
        connect:
            ...
        
kind: ConfigMap
metadata:
    creationTimestamp: null
    name: some
    namespace: somewhere

extract

To extract the xconnect section using the command line tool:

xconnect -input configmap.yml -k8s -target file://xconnect-from-configmap.yml

view

xconnect -dot | dot -Tpng > graph.png && open graph.png

Getting the extra fields

See xconnect_test.go

Inspiration

• https://dzone.com/articles/cataloguing-microservices

© 2019+, ernestmicklei.com. MIT License. Contributions welcome.

Documentation

Index

• func FindBool(f finder, path string) (bool, error)
• func FindInt(f finder, path string) (int, error)
• func FindString(f finder, path string) (string, error)
• type ConnectEntry
  • func (e ConnectEntry) NetworkID() string
  • func (e ConnectEntry) ResourceID() string
• type ConnectionEnd
• type Document
  • func GetConfig(envKey string, filename string) (Document, error)
  • func LoadConfig(filename string) (Document, error)
  • func (d Document) FindBool(path string) (bool, error)
  • func (d Document) FindInt(path string) (int, error)
  • func (d Document) FindString(path string) (string, error)
  • func (d Document) MustBool(path string) bool
  • func (d Document) MustInt(path string) int
  • func (d Document) MustString(path string) string
• type K8SConfiguration
  • func (k K8SConfiguration) ExtractConfig() (x XConnect, err error)
• type ListenEntry
  • func (e ListenEntry) NetworkID() string
• type MetaProperties
• type XConnect

Functions

func FindBool

func FindBool(f finder, path string) (bool, error)

func FindInt

func FindInt(f finder, path string) (int, error)

FindInt returns a integer for a given slash path.

func FindString

func FindString(f finder, path string) (string, error)

FindString returns a string for a given slash path, e.g xconnect/connect/db/url .

Types

type ConnectEntry

type ConnectEntry struct {
	Protocol string `yaml:"protocol,omitempty" json:"scheme,omitempty"`
	Secure   *bool  `yaml:"secure,omitempty" json:"secure,omitempty"`
	Host     string `yaml:"host,omitempty" json:"host,omitempty"`
	Port     *int   `yaml:"port,omitempty" json:"port,omitempty"`
	URL      string `yaml:"url,omitempty" json:"url,omitempty"`
	Disabled bool   `yaml:"disabled,omitempty" json:"disabled,omitempty"`
	Kind     string `yaml:"kind,omitempty" json:"kind,omitempty"`
	// Resource is to identify the virtual listen part
	Resource    string                 `yaml:"resource,omitempty" json:"resource,omitempty"`
	ExtraFields map[string]interface{} `yaml:"-,inline"`
}

ConnectEntry is a list element in the xconnect.connect config.

func (ConnectEntry) NetworkID

func (e ConnectEntry) NetworkID() string

NetworkID returns URL or HOST:PORT

func (ConnectEntry) ResourceID

func (e ConnectEntry) ResourceID() string

ResourceID returns NetworkID() or KIND:RESOURCE

type ConnectionEnd

type ConnectionEnd interface {
	NetworkID() string
}

type Document

type Document struct {
	XConnect    XConnect               `yaml:"xconnect"`
	ExtraFields map[string]interface{} `yaml:"-,inline"`
}

Document is the root YAML element

func GetConfig

func GetConfig(envKey string, filename string) (Document, error)

GetConfig will first check the environment value at {envKey} to find the source of the confguration. If the environment value is not available (empty) then try reading the filename to get the configuration.

func LoadConfig

func LoadConfig(filename string) (Document, error)

LoadConfig returns the document containing the xconnect section.

func (Document) FindBool

func (d Document) FindBool(path string) (bool, error)

FindBool returns a bool for a given slash path.

func (Document) FindInt

func (d Document) FindInt(path string) (int, error)

FindInt returns a integer for a given slash path, e.g xconnect/listen/api/port .

func (Document) FindString

func (d Document) FindString(path string) (string, error)

FindString returns a string for a given slash path, e.g xconnect/connect/db/url .

func (Document) MustBool

func (d Document) MustBool(path string) bool

MustBool same as FindBool but panics if not found. E.g xconnect/listen/api/secure

func (Document) MustInt

func (d Document) MustInt(path string) int

MustInt same as FindInt but panics if not found. E.g xconnect/listen/api/port

func (Document) MustString

func (d Document) MustString(path string) string

MustString same as FindString but panics if not found. E.g xconnect/connect/db/url .

type K8SConfiguration

type K8SConfiguration struct {
	APIVersion string                 `yaml:"apiVersion"`
	Data       map[string]interface{} `yaml:"data"`
	Kind       string                 `yaml:"kind" `
	Metadata   struct {
		Name              string    `yaml:"name" `
		Namespace         string    `yaml:"namespace"`
		CreationTimestamp time.Time `yaml:"creationTimestamp"`
	} `yaml:"metadata"`
}

K8SConfiguration represents a Kubernetes configuration.

func (K8SConfiguration) ExtractConfig

func (k K8SConfiguration) ExtractConfig() (x XConnect, err error)

ExtractConfig expects a "xconnect" key in the data map and parses that part into a xconnect.Config.

type ListenEntry

type ListenEntry struct {
	Protocol string `yaml:"protocol,omitempty" json:"scheme,omitempty"`
	Host     string `yaml:"host,omitempty" json:"host,omitempty"`
	Port     *int   `yaml:"port,omitempty" json:"port,omitempty"`
	// for database connection strings
	URL         string                 `yaml:"url,omitempty" json:"url,omitempty"`
	Secure      *bool                  `yaml:"secure,omitempty" json:"secure,omitempty"`
	Disabled    bool                   `yaml:"disabled,omitempty" json:"disabled,omitempty"`
	ExtraFields map[string]interface{} `yaml:"-,inline"`
}

ListenEntry is a list element in the xconnect.accept config.

func (ListenEntry) NetworkID

func (e ListenEntry) NetworkID() string

type MetaProperties

type MetaProperties struct {
	Name    string `yaml:"name,omitempty" json:"name,omitempty"`
	Version string `yaml:"version,omitempty" json:"version,omitempty"`
	// Operational expenditure, or owner
	Opex        string                 `yaml:"opex,omitempty" json:"opex,omitempty"`
	Labels      []string               `yaml:"tags,omitempty" json:"tags,omitempty"`
	ExtraFields map[string]interface{} `yaml:"-,inline"`
	Kind        string                 `yaml:"kind,omitempty" json:"kind,omitempty"`
}

MetaProperties represents the meta element in the xconnect data section.

type XConnect

type XConnect struct {
	Meta        MetaProperties          `yaml:"meta" json:"meta"`
	Listen      map[string]ListenEntry  `yaml:"listen" json:"listen"`
	Connect     map[string]ConnectEntry `yaml:"connect" json:"connect"`
	ExtraFields map[string]interface{}  `yaml:"-,inline"`
}

XConnect represents the xconnect data section of a YAML document. See spec-xconnect.yaml.