gsb

README

Go Service Bus (GSB)

Vision

1. More impact, More quickly, Less code.

Principles

1. Keep it Simple
2. You aren't gonna need it
3. Design for Evolution
4. Least Astonishment
5. Avoid Premature Optimisation

Front of Mind

1. What are you doing
2. What can go wrong

Rules

1. Anything that fails during startup / initialisation should fail immediatley with descriptive STDOUT.
2. Anything that fails during execution should return an error and be processed by the system, to allow continuous operation.
3. Handlers must implement Handler interface
   1. Handlers can optionally implement an Init() method, with optional return error.
4. All user space code execution is initiated by receiving a message.

Guidelines

1. All external resources are initialised and validated on start up, then provided on execution as Application Resources.

Rationale

1. Try to raise the level of infrastructure.

2. The infrastructure,

   1. takes care of starting, committing and rolling back transactions.
   2. runs all code within the context of a message.
   3. provides automatic retry if the code errors
   4. moves failed messages to the error queue

Description

The backbone of GSB is a message queue.

All work is initiated by pulling messages from this message queue.

If all external dependencies, eg, queues, databases, etc are provided as Application Resources, gsb will manage all transactions, meaning the message then provides a transaction boundary.

In this model, a number of options present when errors occur while processing a message. One, they can be retried. For example, their could be a network glitch which has corrected itself by the time the message is retried. Two, for persistent errors, the message can be moved to error queue. For example, a database may be down. In this case, the database can be restarted, and the message sent back for processing.

Variables

1. NAME
2. RETRIES
3. ERROR_Q
4. AUDIT_Q
5. MQ
6. GSB_
7. GSBMON_
8. VERBOSE

Backbone Message Queue

MQ=type://[username:password@]host/queue_name

1. beanstalk
2. sqs
3. inmemq

Application Resources

GSB_[name]=://[username:password@]/

All external resources are initialised and validated on start up, then provided on execution.

1. DynamoDb
   1. dynamodb
2. FileSystem
   1. s3
   2. fs
3. KeyValue
   1. dir
   2. redis
4. Queue
   1. beanstalk
   2. sqs
   3. inmemq

AWS Specific

1. AWS_ENDPOINT
2. AWS_REGION

Monitors

GSBMON_[name]=://[username:password@]/

A monitor will keep checking the end point for incoming messages. If found, it pull the message, wrap it in an envelope, and post it to the backbone mq for processing. The name of the message will be the name portion indicated in the URI above.

1. dir
2. queue

COMMANDS

To enable the ability to configure the running system, gsb will process commands coming in via the backbone MQ,

1. GSB_VERBOSE_ON
2. GSB_VERBOSE_OFF
3. GSB_AUDIT_ON
4. GSB_AUDIT_OFF

References

1. 8 Fallacies of Distributed Computing
2. 12 Factor App
3. GSB

Documentation

Index

• func Errorf(error error) error
• func ResetMqInMemory()
• type Bus
• type Handler
• type Host
  • func NewHost() *Host
  • func (h *Host) AddHandler(ha Handler)
  • func (h *Host) AppVerboseLog(msg string)
  • func (h *Host) Run()
  • func (h *Host) Send(name string, payload string) error
  • func (h *Host) SendMessage(q appres.Queue, name string, payload string) error
  • func (h *Host) Tick()
• type MonitorData
  • func ParseIncomingMonitorMessage(payload string) (*MonitorData, error)
• type MqInMemory
  • func NewMqInMemory(url *url.URL) *MqInMemory
  • func NewMqInMemoryFromString(urlString string) *MqInMemory
  • func (mq *MqInMemory) Begin() error
  • func (mq *MqInMemory) Commit() error
  • func (mq *MqInMemory) Init() error
  • func (mq *MqInMemory) Move(fromS string, toS string)
  • func (mq *MqInMemory) Name() string
  • func (mq *MqInMemory) Retrieve() (*string, error)
  • func (mq *MqInMemory) Rollback() error
  • func (mq *MqInMemory) Send(payload string) error
• type SnsMessage
  • func ParseIncomingSnsMessage(payload string) (*SnsMessage, error)
• type TestBus
  • func (tb *TestBus) AppVerboseLog(_ string)
  • func (tb *TestBus) Send(name string, payload string) error
• type TestEnvelope

Functions

func Errorf

func Errorf(error error) error

Errorf formats an error for GSB

func ResetMqInMemory

func ResetMqInMemory()

ResetMqInMemory reset the singleton

Types

type Bus

type Bus interface {
	Send(string, string) error
	AppVerboseLog(string)
}

Bus is the Handler facing interface to GSB

type Handler

type Handler interface {
	Handle(payload string) error
}

Handler is to be implemented by user space projects

type Host

type Host struct {
	Name       string
	Mq         mq
	MqURL      *url.URL
	LocalSendQ appres.Queue
	AuditQ     appres.Queue
	ErrorQ     appres.Queue
	Handler    map[string]*handlerWrapper
	// contains filtered or unexported fields
}

Host is the root of GSB

func NewHost

func NewHost() *Host

NewHost creates, configures and returns a host

func (*Host) AddHandler

func (h *Host) AddHandler(ha Handler)

AddHandler allows user space code to Add Handlers to GSB

func (*Host) AppVerboseLog

func (h *Host) AppVerboseLog(msg string)

AppVerboseLog provides user space verbose logging

func (*Host) Run

func (h *Host) Run()

Run is called from user space after configuration

func (*Host) Send

func (h *Host) Send(name string, payload string) error

Send is the channel for user space code to send a message with GSB

func (*Host) SendMessage

func (h *Host) SendMessage(q appres.Queue, name string, payload string) error

SendMessage wraps a message in an envelope and puts it to the passed in queue

func (*Host) Tick

func (h *Host) Tick()

Tick initiatae a single pass

type MonitorData

type MonitorData struct {
	Name    string `json:"Name" validate:"required"`
	Payload string `json:"Payload" validate:"required"`
}

MonitorData is supplied to a monitoring Handler

func ParseIncomingMonitorMessage

func ParseIncomingMonitorMessage(payload string) (*MonitorData, error)

ParseIncomingMonitorMessage is a user space helper function

type MqInMemory

type MqInMemory struct {
	List []string
	// contains filtered or unexported fields
}

MqInMemory swap in queue for testing

func NewMqInMemory

func NewMqInMemory(url *url.URL) *MqInMemory

NewMqInMemory returns the singleton, creating if necessary

func NewMqInMemoryFromString

func NewMqInMemoryFromString(urlString string) *MqInMemory

NewMqInMemoryFromString adapter for NewMqInMemory

func (*MqInMemory) Begin

func (mq *MqInMemory) Begin() error

Begin Transaction

func (*MqInMemory) Commit

func (mq *MqInMemory) Commit() error

Commit Transaction

func (*MqInMemory) Init

func (mq *MqInMemory) Init() error

Init wrapper for InMemory Queue

func (*MqInMemory) Move

func (mq *MqInMemory) Move(fromS string, toS string)

Move message to a different queue

func (*MqInMemory) Name

func (mq *MqInMemory) Name() string

Name wrapper for InMemory Queue

func (*MqInMemory) Retrieve

func (mq *MqInMemory) Retrieve() (*string, error)

Retrieve wrapper for InMemory Queue

func (*MqInMemory) Rollback

func (mq *MqInMemory) Rollback() error

Rollback Transaction

func (*MqInMemory) Send

func (mq *MqInMemory) Send(payload string) error

Send wrapper for InMemory Queue

type SnsMessage

type SnsMessage struct {
	Message string
}

SnsMessage represents an incoming aws SNS message

func ParseIncomingSnsMessage

func ParseIncomingSnsMessage(payload string) (*SnsMessage, error)

ParseIncomingSnsMessage populates the SnsMessage struct from the payload

type TestBus

type TestBus struct {
	SendList []TestEnvelope
}

TestBus is available to be used for user space tests

func (*TestBus) AppVerboseLog

func (tb *TestBus) AppVerboseLog(_ string)

AppVerboseLog is a wrapper for verbose logging

func (*TestBus) Send

func (tb *TestBus) Send(name string, payload string) error

Send is the mock for Sending Messages

type TestEnvelope

type TestEnvelope struct {
	// contains filtered or unexported fields
}

TestEnvelope is available to be used for user space tests