cloudconfigclient

README

Go Config Server Client

Go library for Spring Config Server. Inspired by the Java library Cloud Config Client.

go get github.com/Piszmog/cloudconfigclient/v2

V2 Migration

See V2 Migration for details on how to migrate from V1 to V2

Description

Spring's Config Server provides way to externalize configurations of applications. Spring's Spring Cloud Config Client can be used to load the base configurations that an application requires to properly function.

This library provides clients the ability to load Configurations and Files from the Config Server.

Compatibility

This library is compatible with versions of Spring Config Server greater than or equal to 1.4.x.RELEASE. Prior versions of the Config Server do not provide the endpoint necessary to retrieve files for the Config Server's default branch.

Spring Cloud Config Server v3.x

Since Spring Cloud Services v3.0, the service name in VCAP_SERVICES has changed from p-config-server to be p.config-server.

To help mitigate migration difficulties, cloudconfigclient.New(cloudconfigclient.DefaultCFService()) will first search for the service p.config-server (v3.x). If the v3.x service could not be found, p-config-server (v2.x) will be search for.

See Spring Cloud Services Differences for more details.

Example Usage

Below is an example usage of the library to retrieve a file from the Config Server and to retrieve the application's configurations

• For local config client, there are two options (Option) the create a client
  1. Call LocalEnv(). Set the environment variable CONFIG_SERVER_URLS. It is a comma separated list of all the base URLs
  2. Call Local(baseUrls ...string). Provide the array of base URLs of Config Servers.
• If the config server is protected with basic auth, call Basic with the username and password.
• For running in Cloud Foundry, ensure a Config Server is bounded to the application. VCAP_SERVICES will be provided as an environment variables with the credentials to access the Config Server
• For connecting to a Config Server via OAuth2 and not deployed to Cloud Foundry, an OAuth2 Client can be created with OAuth2(baseURL string, clientId string, secret string, tokenURI string)

package main

import (
	"fmt"
	"github.com/Piszmog/cloudconfigclient/v2"
	"net/http"
)

type File struct {
	Example Example `json:"example"`
}

type Example struct {
	Field string `json:"field"`
}

func main() {
	// To create a Client for a locally running Spring Config Server
	configClient, err := cloudconfigclient.New(cloudconfigclient.LocalEnv(&http.Client{}))
	// Or
	configClient, err = cloudconfigclient.New(cloudconfigclient.Local(&http.Client{}, "http://localhost:8888"))
	// or to create a Client for a Spring Config Server using Basic Authentication
	configClient, err = cloudconfigclient.New(cloudconfigclient.Basic(&http.Client{}, "username", "password" "http://localhost:8888"))
	// or to create a Client for a Spring Config Server in Cloud Foundry
	configClient, err = cloudconfigclient.New(cloudconfigclient.DefaultCFService())
	// or to create a Client for a Spring Config Server with OAuth2
	configClient, err = cloudconfigclient.New(cloudconfigclient.OAuth2("config server uri", "client id", "client secret",
		"access token uri"))
	// or a combination of local, Cloud Foundry, and OAuth2
	configClient, err = cloudconfigclient.New(
		cloudconfigclient.Local(&http.Client{}, "http://localhost:8888"),
		cloudconfigclient.DefaultCFService(),
		cloudconfigclient.OAuth2("config server uri", "client id", "client secret", "access token uri"),
	)

	if err != nil {
		fmt.Println(err)
		return
	}
	var file File
	// Retrieves a 'temp1.json' from the Config Server's default branch in directory 'temp' and deserialize to File
	err = configClient.GetFile("temp", "temp1.json", &file)
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Printf("%+v\n", file)
	// Retrieves a 'temp2.txt' from the Config Server's default branch in directory 'temp' as a byte slice ([]byte)
	b, err := configClient.GetFileRaw("temp", "temp2.txt")
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(string(b))

	// Retrieves the configurations from the Config Server based on the application name and active profiles
	config, err := configClient.GetConfiguration("testApp", "dev")
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Printf("%+v", config)
	// if we want, we can convert the config to a struct
	var configStruct Config
	err = config.Unmarshal(&configStruct)
	if err != nil {
		fmt.Println(err)
	}
}

type Config struct {
	Field1 string `json:"example"`
	Field2 int    `json:"field2"`
}

VCAP_SERVICES

When an application is deployed to Cloud Foundry, services can be bounded to the application. When a service is bounded to an application, the application will have the necessary connection information provided in the environment variable VCAP_SERVICES.

Structure of the VCAP_SERVICES value

{
  "<service type :: e.g. p-config-server>": [
    {
      "name": "<the service name>",
      "instance_name": "<service name>",
      "binding_name": "<bounded name of the service>",
      "credentials": {
        "uri": "<URI of the service :: used to connect to the service>",
        "client_secret": "<OAuth2 client secret>",
        "client_id": "<OAuth2 client id>",
        "access_token_uri": "<OAuth2 token URI>"
      },
      ...
    }
  ]
}

CredHub Reference

Newer versions of PCF (>=2.6) may have services that use a CredHub Reference to store credential information.

When viewing the Environment Variables of an application via the UI, the credentials may appear as the following

{
  "credentials": {
    "credhub-ref": "/c/example-service-broker/example-service/faa677f5-25cd-4f1e-8921-14a9d5ab48b8/credentials"
  }
}

When the application starts up, the credhub-ref is replaced with the actual credential values that application will need to connect to the service.

Configurations

The Config Server allows the ability to retrieve configurations for an application. Only files that follow a strict naming convention will be loaded,

File Name
application.{yml/properties}
application-{profile}.{yml/properties}
{application name}.{yml/properties}
{application name}-{profile}.{yml/properties}

The loaded configurations are in the following JSON format,

{
  "name": "<name of application>",
  "profiles": "<profiles passed in request>",
  "label": "<GIT branch configurations loaded from>",
  "version": "<version>",
  "state": "<state>",
  "propertySources": [
    {
      "<propertySourceName>": {
        "name": "<property source name>",
        "source": {
          "<source path in .properties format>": "<value>"
        }
      }
    }
  ]
}

To use the library to retrieve configurations, create a Client and invoke the method GetConfiguration(applicationName string, profiles ...string). The return will be the struct representation of the configuration JSON - client.Configuration.

Resources

Spring's Config Server allows two ways to retrieve files from a backing repository.

URL Path
/<appName>/<profiles>/<directory>/<file>?useDefaultLabel=true
/<appName>/<profiles>/<branch>/<directory>/<file>

• When retrieving a file from the Config Server's default branch, the file must not exist at the root of the repository.
• If the directory is in the searchPath, it does not have to be specified (depending on SCCS version)

The functions available to retrieve resource files are GetFile(directory string, file string, interfaceType interface{}) and GetFileFromBranch(branch string, directory string, file string, interfaceType interface{}). To retrieve the data from the files, the functions available are GetFileRaw(directory string, file string) and GetFileFromBranchRaw(branch string, directory string, file string)

• The interfaceType is the object to deserialize the file to

Spring Cloud Config Server v3.x Changes

The following is only for certain versions of SCCS v3.x. If a file is not being found by the client, the following may be true.

SCCS v3.x slightly changed how files are retrieved. If the Config Server specified a directory in the searchPaths, the path should be excluded from the GetFile(..) invocation.

For example if common has been specified in the searchPaths and the file common/foo.txt needs to be retrieved, then the directory to provide to GetFile(..) should be "" (blank).

This differs with SCS v2.x where the directory in searchPaths did not impact the directory provided to GetFile(..) (e.g. to retrieve file common/foo.txt, directory would be "common").

Documentation

Index

• Constants
• Variables
• type Client
  • func New(options ...Option) (*Client, error)
  • func (c *Client) GetConfiguration(applicationName string, profiles ...string) (Source, error)
  • func (c *Client) GetConfigurationWithLabel(label string, applicationName string, profiles ...string) (Source, error)
  • func (c *Client) GetFile(directory string, file string, interfaceType interface{}) error
  • func (c *Client) GetFileFromBranch(branch string, directory string, file string, interfaceType interface{}) error
  • func (c *Client) GetFileFromBranchRaw(branch string, directory string, file string) ([]byte, error)
  • func (c *Client) GetFileRaw(directory string, file string) ([]byte, error)
• type Configuration
• type HTTPClient
  • func (h *HTTPClient) Get(paths []string, params map[string]string) (*http.Response, error)
  • func (h *HTTPClient) GetResource(paths []string, params map[string]string, dest interface{}) error
  • func (h *HTTPClient) GetResourceRaw(paths []string, params map[string]string) ([]byte, error)
• type Option
  • func Basic(client *http.Client, username, password string, urls ...string) Option
  • func CFService(service string) Option
  • func DefaultCFService() Option
  • func Local(client *http.Client, urls ...string) Option
  • func LocalEnv(client *http.Client) Option
  • func OAuth2(baseURL string, clientID string, secret string, tokenURI string) Option
• type PropertySource
• type PropertySourceHandler
• type Resource
• type Source
  • func (s *Source) GetPropertySource(fileName string) (PropertySource, error)
  • func (s *Source) HandlePropertySources(handler PropertySourceHandler)
  • func (s *Source) Unmarshal(v interface{}) error

Constants

const (
	// ConfigServerName the service name of the Config Server in PCF.
	ConfigServerName = "p-config-server"
	// EnvironmentLocalConfigServerUrls is an environment variable for setting base URLs for local Config Servers.
	EnvironmentLocalConfigServerUrls = "CONFIG_SERVER_URLS"
	// SpringCloudConfigServerName the service name of the Spring Cloud Config Server in PCF.
	SpringCloudConfigServerName = "p.config-server"
)

Variables

var ErrPropertySourceDoesNotExist = errors.New("property source does not exist")

ErrPropertySourceDoesNotExist is the error that is returned when there are no PropertySource that match the specified file name.

var ErrResourceNotFound = errors.New("failed to find resource")

ErrResourceNotFound is a special error that is used to propagate 404s.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client contains the clients of the Config Servers.

func New

func New(options ...Option) (*Client, error)

New creates a new Client based on the provided options. A Client can be configured to communicate with a local Config Server, an OAuth2 Server, and Config Servers in Cloud Foundry.

At least one option must be provided.

func (*Client) GetConfiguration

func (c *Client) GetConfiguration(applicationName string, profiles ...string) (Source, error)

GetConfiguration retrieves the configurations/property sources of an application based on the name of the application and the profiles of the application.

func (*Client) GetConfigurationWithLabel

func (c *Client) GetConfigurationWithLabel(label string, applicationName string, profiles ...string) (Source, error)

GetConfigurationWithLabel retrieves the configurations/property sources of an application based on the name of the application and the profiles of the application and the label.

func (*Client) GetFile

func (c *Client) GetFile(directory string, file string, interfaceType interface{}) error

GetFile retrieves the specified file from the provided directory from the Config Server's default branch.

The file will be deserialized into the specified interface type.

func (*Client) GetFileFromBranch

func (c *Client) GetFileFromBranch(branch string, directory string, file string, interfaceType interface{}) error

GetFileFromBranch retrieves the specified file from the provided branch in the provided directory.

The file will be deserialized into the specified interface type.

func (*Client) GetFileFromBranchRaw

func (c *Client) GetFileFromBranchRaw(branch string, directory string, file string) ([]byte, error)

GetFileFromBranchRaw retrieves the file from the specified branch as a byte slice.

func (*Client) GetFileRaw

func (c *Client) GetFileRaw(directory string, file string) ([]byte, error)

GetFileRaw retrieves the file from the default branch as a byte slice.

type Configuration

type Configuration interface {
	// GetConfiguration retrieves the configurations/property sources of an application based on the name of the application
	// and the profiles of the application.
	GetConfiguration(applicationName string, profiles ...string) (Source, error)
	// GetConfigurationWithLabel retrieves the configurations/property sources of an application based on the name of the application
	// and the profiles of the application and the label.
	GetConfigurationWithLabel(label string, applicationName string, profiles ...string) (Source, error)
}

Configuration interface for retrieving an application's configuration files from the Config Server.

type HTTPClient

type HTTPClient struct {
	*http.Client
	// BaseURL is the base URL for the Config Server.
	BaseURL string
	// Authorization is the authorization header value for the Config Server. If not provided, no authorization header is not explicitly set.
	// If the client is using OAuth2, the authorization header is set automatically.
	Authorization string
}

HTTPClient is a wrapper for http.Client.

func (*HTTPClient) Get

func (h *HTTPClient) Get(paths []string, params map[string]string) (*http.Response, error)

Get performs a http.MethodGet operation. Builds the URL based on the provided paths and params.

func (*HTTPClient) GetResource

func (h *HTTPClient) GetResource(paths []string, params map[string]string, dest interface{}) error

GetResource performs a http.MethodGet operation. Builds the URL based on the provided paths and params. Deserializes the response to the specified destination.

Capable of unmarshalling YAML, JSON, and XML. If file type is of another type, use GetResourceRaw instead.

func (*HTTPClient) GetResourceRaw

func (h *HTTPClient) GetResourceRaw(paths []string, params map[string]string) ([]byte, error)

GetResourceRaw performs a http.MethodGet operation. Builds the URL based on the provided paths and params. Returns the byte slice response.

type Option

type Option func(*[]*HTTPClient) error

Option creates a slice of httpClients per Config Server instance.

func Basic

func Basic(client *http.Client, username, password string, urls ...string) Option

Basic creates a clients for a Config Server based on the provided basic authentication information.

func CFService

func CFService(service string) Option

CFService creates a clients for each Config Servers the application is bounded to in Cloud Foundry. The environment variable 'VCAP_SERVICES' provides a JSON. The JSON should contain the entry matching the specified name. This entry and used to build an OAuth Client.

func DefaultCFService

func DefaultCFService() Option

DefaultCFService creates a clients for each Config Servers the application is bounded to in Cloud Foundry. The environment variable 'VCAP_SERVICES' provides a JSON that contains an entry with the key 'p-config-server' (v2.x) or 'p.config-server' (v3.x).

The service 'p.config-server' is search for first. If not found, 'p-config-server' is searched for.

func Local

func Local(client *http.Client, urls ...string) Option

Local creates a clients for a locally running Config Servers.

func LocalEnv

func LocalEnv(client *http.Client) Option

LocalEnv creates a clients for a locally running Config Servers. The URLs to the Config Servers are acquired from the environment variable 'CONFIG_SERVER_URLS'.

func OAuth2

func OAuth2(baseURL string, clientID string, secret string, tokenURI string) Option

OAuth2 creates a Client for a Config Server based on the provided OAuth2.0 information.

type PropertySource

type PropertySource struct {
	Source map[string]interface{} `json:"source"`
	Name   string                 `json:"name"`
}

PropertySource is the property source for the application.

A property source is either a YAML or a PROPERTIES file located in the repository that a Config Server is pointed at.

type PropertySourceHandler

type PropertySourceHandler func(propertySource PropertySource)

PropertySourceHandler handles the specific PropertySource.

type Resource

type Resource interface {
	// GetFile retrieves the specified file from the provided directory from the Config Server's default branch.
	//
	// The file will be deserialized into the specified interface type.
	GetFile(directory string, file string, interfaceType interface{}) error
	// GetFileFromBranch retrieves the specified file from the provided branch in the provided directory.
	//
	// The file will be deserialized into the specified interface type.
	GetFileFromBranch(branch string, directory string, file string, interfaceType interface{}) error
	// GetFileRaw retrieves the file from the default branch as a byte slice.
	GetFileRaw(directory string, file string) ([]byte, error)
	// GetFileFromBranchRaw retrieves the file from the specified branch as a byte slice.
	GetFileFromBranchRaw(branch string, directory string, file string) ([]byte, error)
}

Resource interface describes how to retrieve files from the Config Server.

type Source

type Source struct {
	Name            string           `json:"name"`
	Profiles        []string         `json:"profiles"`
	Label           string           `json:"label"`
	Version         string           `json:"version"`
	State           string           `json:"state"`
	PropertySources []PropertySource `json:"propertySources"`
}

Source is the application's source configurations. It con contain zero to n number of property sources.

func (*Source) GetPropertySource

func (s *Source) GetPropertySource(fileName string) (PropertySource, error)

GetPropertySource retrieves the PropertySource that has the specifies fileName. The fileName is the name of the file with extension - e.g. application-foo.yml.

Usually the Config Server will return the PropertySource.Name as an URL of sorts (e.g. ssh://base-url.com/path/to/repository/path/to/file.yml/properties). So in order to find the specific file with the desired configurations, the ending of the name needs to be matched against.

func (*Source) HandlePropertySources

func (s *Source) HandlePropertySources(handler PropertySourceHandler)

HandlePropertySources handles all PropertySource configurations that are files. This is a convenience method to handle boilerplate for-loop code and filtering of non-configuration files.

Config Server may return other configurations (e.g. credhub property sources) that contain no configurations (PropertySource.Source is empty).

func (*Source) Unmarshal

func (s *Source) Unmarshal(v interface{}) error

Unmarshal converts the Source.PropertySources to the specified type. The type must be a pointer to a struct.

The provided pointer struct must use JSON tags to map to the PropertySource.Source.

This function is not optimized (ugly) and is intended to only be used at startup.