golambda

README

golambda

A suite of Go utilities for AWS Lambda functions to ease adopting best practices.

Overview

Features

• Event decapsulation: Parse event data received when invoking. Also golambda make easy to write unit test of Lambda function
• Structured logging: golambda provides requisite minimum logging interface for Lambda function. It output log as structured JSON.
• Error handling: Error structure with arbitrary variables and stack trace feature.
• Get secret parameters: Secret values should be stored in AWS Secrets Manager and can be got easily.

NOTE: The suite is NOT focusing to Lambda function for API gateway, but partially can be leveraged for the function.

How to use

$ go get github.com/m-mizutani/golambda/v2

Source event decapsulation

Lambda function can have event source(s) such as SQS, SNS, etc. The main data is encapsulated in their data structure. golambda provides not only decapsulation feature for Lambda execution but also encapsulation feature for testing. Following event sources are supported for now.

• SQS body: DecapSQS
• SNS message: DecapSNS
• SNS message over SQS: DecapSNSoverSQS

Lambda implementation

package main

import (
	"strings"

	"github.com/m-mizutani/golambda/v2"
)

// MyEvent is exported for test
type MyEvent struct {
	Message string `json:"message"`
}

// Handler is exported for test
func Handler(ctx context.Context, event *golambda.Event) (string, error) {
	// Decapsulate body message(s) in SQS Event structure
	events, err := event.DecapSQS()
	if err != nil {
		return "", err
	}

	var response []string
	// Iterate body message(S)
	for _, ev := range events {
		var msg MyEvent
		// Unmarshal golambda.Event to MyEvent
		if err := ev.Bind(&msg); err != nil {
			return "", err
		}

		// Do something
		response = append(response, msg.Message)
	}

	return strings.Join(response, ":"), nil
}

func main() {
	golambda.Start(Handler)
}

Unit test

package main_test

import (
	"testing"

	"github.com/m-mizutani/golambda/v2"
	"github.com/stretchr/testify/require"

	main "github.com/m-mizutani/golambda/v2/example/decapEvent"
)

func TestHandler(t *testing.T) {
	messages := []main.MyEvent{
		{
			Message: "blue",
		},
		{
			Message: "orange",
		},
	}
	event, err := golambda.NewSQSEvent(messages)
	require.NoError(t, err)

	resp, err := main.Handler(event)
	require.NoError(t, err)
	require.Equal(t, "blue:orange", resp)
}

Structured logging

Lambda function output log data to CloudWatch Logs by default. CloudWatch Logs and Insights that is rich CloudWatch Logs viewer supports JSON format logs. Therefore JSON formatted log is better for Lambda function.

golambda provides Logger for JSON format logging. It has With() to add a pair of key and value to a log message. Logger has lambda request ID by default if you use the logger with golambda.Start(). Logger is provided as zlog.Logger. RenewLogger() allows you to reconfigure logging setting.

Output with temporary variable

v1 := "say hello"
golambda.Logger.With("var1", v1).Info("Hello, hello, hello")
/* Output:
{
	"level": "info",
	"lambda.requestID": "565389dc-c13f-4fc0-b113-xxxxxxxxxxxx",
	"time": "2020-12-13T02:44:30Z",
	"var1": "say hello",
	"message": "Hello, hello, hello"
}
*/

Log level

golambda.Logger (golambda.LambdaLogger type) provides following log level. Log level can be configured by environment variable LOG_LEVEL.

• TRACE
• DEBUG
• INFO
• WARN
• ERROR

Lambda function should return error to top level function when occurring unrecoverable error, should not exit suddenly. Therefore PANIC and FATAL is not provided according to the thought.

Error handling

NOTE: golambda.Error is obsoleted and use github.com/m-mizutani/goerr instead.

golambda.Error can have pairs of key and value to keep context of error. For example, golambda.Error can bring original string data when failed to unmarshal JSON. The string data can be extracted in caller function.

Also, golambda.Start supports general error handling:

1. Output error log with
   • Pairs of key and value in goerr.Error of goerr as error.values
   • Stack trace of error as error.stacktrace
2. Send error record to sentry.io if SENTRY_DSN is set as environment variable
   • Stack trace of golambda.Error is also available in sentry.io by compatibility with github.com/pkg/errors
   • Output event ID of sentry to log as error.sentryEventID
   • You can set SENTRY_ENV and SENTRY_RELEASE also optionally.

package main

import (
	"github.com/m-mizutani/golambda/v2"
)

// Handler is exported for test
func Handler(event golambda.Event) (interface{}, error) {
	trigger := "something wrong"
	return nil, goerr.New("oops").With("trigger", trigger)
}

func main() {
	golambda.Start(Handler)
}

Then, golambda output following log to CloudWatch.

{
    "level": "error",
    "lambda.requestID": "565389dc-c13f-4fc0-b113-f903909dbd45",
    "trigger": "something wrong",
    "stacktrace": [
        {
            "func": "main.Handler",
            "file": "xxx/your/project/src/main.go",
            "line": 27
        },
        {
            "func": "github.com/m-mizutani/golambda.Start.func1",
            "file": "xxx/github.com/m-mizutani/golambda/lambda.go",
            "line": 107
        }
    ],
    "time": "2020-12-13T02:42:48Z",
    "message": "oops"
}

Get secret parameters

In general, parameters of Lambda function are stored sa environment variable, such as LOG_LEVEL. However secret parameters such as credential, API key/token, etc should be stored in AWS Secrets Manager or Parameter Store to control access permission more explicitly in many cases.

golambda.GetSecretValues fetches values of AWS Secrets Manager and binds to a structure variable.

type mySecret struct {
    Token string `json:"token"`
}
var secret mySecret
if err := golambda.GetSecretValues(os.Getenv("SECRET_ARN"), &secret); err != nil {
    log.Fatal("Failed: ", err)
}

// Access to other service with secret.Token

License

See LICENSE.

Documentation

Index

• Variables
• func GetSecretValues(secretArn string, values interface{}) error
• func GetSecretValuesWithFactory(secretArn string, values interface{}, factory SecretsManagerFactory) error
• func HandleError(err error)
• func NewSecretsManagerMock() (*SecretsManagerMock, SecretsManagerFactory)
• func RenewLogger(options ...zlog.Option)
• func Start[T any](callback Callback[T])
• func WithLogger(key string, value interface{})
• type Callback
• type Event
  • func NewEvent(ev any) *Event
  • func NewSQSEvent(v any) (*Event, error)
  • func (x *Event) Bind(v any) error
  • func (x *Event) DecapSNS() ([]EventRecord, error)
  • func (x *Event) DecapSNSoverSQS() ([]EventRecord, error)
  • func (x *Event) DecapSQS() ([]EventRecord, error)
  • func (x *Event) NewSNSEvent(v any) error
  • func (x *Event) NewSNSonSQSEvent(v any) error
• type EventRecord
  • func (x EventRecord) Bind(ev any) error
  • func (x EventRecord) String() string
• type SecretsManagerClient
• type SecretsManagerFactory
• type SecretsManagerMock
  • func (x *SecretsManagerMock) GetSecretValue(input *secretsmanager.GetSecretValueInput) (*secretsmanager.GetSecretValueOutput, error)

Variables

var (
	ErrFailedDecodeEvent = goerr.New("failed to decode event")
	ErrFailedEncodeEvent = goerr.New("failed to encode event")

	ErrNoEventData = goerr.New("no event data")

	ErrInvalidARN           = goerr.New("invalid ARN")
	ErrFailedSecretsManager = goerr.New("failed SecretsManager operation")
	ErrFailedDecodeSecret   = goerr.New("failed to decode secret")
)

var Logger *zlog.Logger

Logger is common logging interface

Functions

func GetSecretValues

func GetSecretValues(secretArn string, values interface{}) error

GetSecretValues bind secret data of AWS Secrets Manager to values. values should be set as pointer of struct with json meta tag.

type mySecret struct {
    Token string `json:"token"`
}
var secret mySecret
if err := golambda.GetSecretValues(secretARN, &secret); err != nil {
    log.Fatal("Failed: ", err)
}

func GetSecretValuesWithFactory

func GetSecretValuesWithFactory(secretArn string, values interface{}, factory SecretsManagerFactory) error

GetSecretValuesWithFactory can call SecretsManager.GetSecretValue with your SecretsManagerClient by factory. It uses newDefaultSecretsManager if factory is nil

func HandleError

func HandleError(err error)

HandleError emits the error to Sentry and outputs the error to logs

func NewSecretsManagerMock

func NewSecretsManagerMock() (*SecretsManagerMock, SecretsManagerFactory)

NewSecretsManagerMock returns both of mock and factory method of the mock for testing. Developer can set secrets value as JSON to SecretsManagerMock.Secrets with key (secretes ARN). Also the mock stores Region that is extracted from secretArn and Input of secretsmanager.GetSecretValue when invoking GetSecretValuesWithFactory.

func RenewLogger

func RenewLogger(options ...zlog.Option)

func Start

func Start[T any](callback Callback[T])

Start sets up Arguments and logging tools, then invoke Callback with Arguments. When exiting, it also does error handling if Callback returns error

func WithLogger

func WithLogger(key string, value interface{})

Types

type Callback

type Callback[T any] func(ctx context.Context, event *Event) (T, error)

Callback is callback function type of golambda.Start().

Trigger event data (SQS, SNS, etc) is included in Event.

Callback has 2 returned value. 1st value (interface{}) will be passed to Lambda. The 1st value is allowed nil if you do not want to return any value to Lambda. 2nd value (error) also will be passed to Lambda, however golambda.Start() does error handling: 1) Extract stack trace of error if err is golambda.Error 2) Send error record to sentry.io if SENTRY_DSN is set as environment variable 3) Output error log

type Event

type Event struct {
	// contains filtered or unexported fields
}

Event provides lambda original event converting utilities

func NewEvent

func NewEvent(ev any) *Event

NewEvent provides a new event with original data.

func NewSQSEvent

func NewSQSEvent(v any) (*Event, error)

NewSQSEvent sets v as SQSEvent body. This function overwrite Origin for testing. NewSQSEvent allows both of one record and multiple record as slice or array e.g.)

ev.NewSQSEvent("red") -> one SQSMessage in SQSEvent
ev.NewSQSEvent([]string{"blue", "orange"}) -> two SQSMessage in SQSEvent

func (*Event) Bind

func (x *Event) Bind(v any) error

Bind does json.Marshal original event and json.Unmarshal to v. If failed, return error with ErrInvalidEventData.

func (*Event) DecapSNS

func (x *Event) DecapSNS() ([]EventRecord, error)

DecapSNS decapsulate wrapped body data in SNSEvent. If no SQS records, it returns ErrNoEventData.

func (*Event) DecapSNSoverSQS

func (x *Event) DecapSNSoverSQS() ([]EventRecord, error)

DecapSNSoverSQS decapsulate wrapped body data to in SNSEntity over SQSEvent. If no SQS records, it returns ErrNoEventData.

func (*Event) DecapSQS

func (x *Event) DecapSQS() ([]EventRecord, error)

DecapSQS decapsulate wrapped body data in SQSEvent. If no SQS records, it returns ErrNoEventData.

func (*Event) NewSNSEvent

func (x *Event) NewSNSEvent(v any) error

NewSNSEvent sets v as SNS entity. This function overwrite Origin for testing. NewSNSEvent allows both of one record and multiple record as slice or array e.g.) ev.NewSNSEvent("red") -> one SNS entity in SNSEvent ev.NewSNSEvent([]string{"blue", "orange"}) -> two SNS entity in SNSEvent

func (*Event) NewSNSonSQSEvent

func (x *Event) NewSNSonSQSEvent(v any) error

NewSNSonSQSEvent sets v as SNS entity over SQS. This function overwrite Origin and should be used for testing. NewSNSonSQSEvent allows both of one record and multiple record as slice or array

e.g.)

ev.NewSNSonSQSEvent("red") // -> one SQS message on one SQS event
ev.NewSNSonSQSEvent([]string{"blue", "orange"}) // -> two SQS message on one SQS event

type EventRecord

type EventRecord []byte

EventRecord is decapsulate event data (e.g. Body of SQS event)

func (EventRecord) Bind

func (x EventRecord) Bind(ev any) error

Bind unmarshal event record to object. If failed, return error with ErrInvalidEventData.

func (EventRecord) String

func (x EventRecord) String() string

String returns raw string data

type SecretsManagerClient

type SecretsManagerClient interface {
	GetSecretValue(*secretsmanager.GetSecretValueInput) (*secretsmanager.GetSecretValueOutput, error)
}

SecretsManagerClient is wrapper of secretsmanager.SecretsManager

type SecretsManagerFactory

type SecretsManagerFactory func(region string) (SecretsManagerClient, error)

SecretsManagerFactory is factory function type to replace SecretsManagerClient

type SecretsManagerMock

type SecretsManagerMock struct {
	Secrets map[string]string
	Region  string
	Input   []*secretsmanager.GetSecretValueInput
}

SecretsManagerMock is mock of SecretsManagerClient for testing.

func (*SecretsManagerMock) GetSecretValue

func (x *SecretsManagerMock) GetSecretValue(input *secretsmanager.GetSecretValueInput) (*secretsmanager.GetSecretValueOutput, error)

GetSecretValue is mock method for SecretsManagerMock. It checks if the secretId (ARN) exists in SecretsManagerMock.Secrets as key. It returns a string value if existing or ResourceNotFoundException error if not existing.