faultinject

README

• Fault Injection
  • Getting Started
  • Features
  • Example
  • Description

Fault Injection

Fault injection is a chaos engineering utility that allows you to test error scenarios with real components by adding fault points.

The package has strictly tested.

Getting Started

To allow your code to be injected with faults, you must follow the context-checking idiom from the stdlib. This way, your code does a common practice with context verification, and you gain the ability to inject fault as well with the existing code base.

if err := ctx.Err(); err != nil {
    return err
}

// or

if ctx.Err() != nil {
    return ctx.Err()
}

// or

select {
case <-ctx.Done():
    return ctx.Err()
}

If you feel adventurous and want to do chaos engineering, then you can use Context#Value to look for injected errors for a given FaultName key. E.g., in a round-trip middleware, you inject a timeout error and test retries logic with it.

type NamedFault struct{}

// optionally you can hard code an error that makes the most sense in your code as a return value
if ctx.Value(NamedFault{}) != nil {
	return ErrMyDomainErr
}

// optionally you can also accept an injected error value
if err, ok := ctx.Value(NamedFault{}).(error); ok {
	return err
}

To utilise these existing conventions, the testcase/faultinject package provides you with the following tools:

• faultinject.Inject
  • Allows you to inject consumable faults into a context. Consumable faults are removed upon retrieval, thus allowing testing retry mechanism.
• CallerFault
  • Allows you to define what package/function/receiver should trigger an error in Context#Err.

Features

• You can add fault points to specific points
  • This simplifies your expectations in your integration tests; you can trigger these fault points instead of analyzing the underlying error handling of a given component when you don't want the test of the implementation details of a given dependent component
• The ability to simulate temporary errors
  • This allows testing retry logic with ease without the need to build and maintain intelligent mocks that try to mimic real components
• Global Enable switch to allow fault injection on demand
  • By default, fault injection ignores all calls, doesn't check, doesn't inject unless Fault Injection is explicitly allowed

Example

The Fault injection package doesn't depend on the testing package and should be safe to use in production code.

Description

This approach enables you to test out small error cases or event the cascading effects in a microservice setup. One of the instant benefits of fault injection is that your clients can test with your actual errors and don't need to maintain their mocks/stubs arrangements manually. If fuel injection is exposed on your API, then It also enables your clients to write integration tests against error scenarios with your system's API. Last but not least, it allows you to remove forced indirections from your codebase, where you have to use a header interface for the sake of testing error handling in a component.

One often mentioned argument about fault injection is the need to add something to the production codebase for testing, but in practice, if you have many header interfaces in your codebase, then you are already actively altering your production codebase for testing purposes, In the end, you need to judge if header interface-based indirections or fault injection makes more sense for your use-cases, as this is not a silver bullet.

By this time, I believe you might feel reticent to put fault injection into your non-test code. Engineering controlled chaos into your application is not a standard testing strategy. It has its pros and cons. For example, you can simplify your code through using less header interface based indirection to test error cases in your code. It allows you to trigger faults without the need to understand the internal logic of that concrete implementation's error cases. It also allows you to specify expectations about fault injection in your Role Interface's interface testing suite. You can do simulation of temporary outages and test retry mechanisms.

But on the grand scale, the real value with fault injection is the ability to test error cases at the system level in a micro-service setup, where errors can have unexpected cascading effects. I loved to see how easy to find bugs with fault injection when I was working with a mobile team in one of my previous job. Our biggest issue was that after you released a mobile client, it was a pain point to make our users upgrade to the latest client version when we identified rainy cases during production use.

Limitations / Known Issues

When context.cancelCtx is wrapping the faultinject.injectContext, the Err() method is not propagated from context.cancelCtx down to the parent context, instead it swallow the error checking until a Done closing is triggered.

To work around this limitation, call the context Value() method with any value to ensure fault injection checking.

// manual fault injection checking,
// the key for the value doesn't matter.
_ = ctx.Value(42)
return ctx.Err()

Or alternatively you can use the faultinject.Check method for a specific fault tag.

type FaultName struct{}

if err := faultinject.Check(ctx, FaultName{}); err != nil {
    return // err
}

Documentation

Overview

Example

package main

import (
	"context"
	"fmt"

	"go.llib.dev/testcase/faultinject"
)

type FaultTag struct{}

func main() {
	defer faultinject.Enable()()
	ctx := context.Background()
	fmt.Println(ctx.Err()) // no error as expected

	// arrange one fault injection for FaultTag
	ctx = faultinject.Inject(ctx, FaultTag{}, fmt.Errorf("example error to inject"))

	if err, ok := ctx.Value(FaultTag{}).(error); ok {
		fmt.Println(err) // prints the injected error
	}
	if err, ok := ctx.Value(FaultTag{}).(error); ok {
		fmt.Println(err) // code not reached as injectedFault is already consumed
	}
}

Example (ChaosEngineeringWithExplicitFaultPoints)

package main

import (
	"context"
	"errors"
	"fmt"

	"go.llib.dev/testcase/faultinject"
)

type FaultTag struct{}

func main() {
	defer faultinject.Enable()()
	ctx := context.Background()
	fmt.Println(MyFuncWithChaosEngineeringFaultPoints(ctx)) // no error

	ctx = faultinject.Inject(ctx, FaultTag{}, errors.New("boom")) // arrange fault injection for FaultTag
	fmt.Println(MyFuncWithChaosEngineeringFaultPoints(ctx))       // "boom" is returned
}

func MyFuncWithChaosEngineeringFaultPoints(ctx context.Context) error {
	if err, ok := ctx.Value(FaultTag{}).(error); ok {
		return err
	}

	if err := ctx.Err(); err != nil {
		return err
	}

	return nil
}

Example (FaultInjectWithFixErrorReplyFromTheFaultPoint)

package main

import (
	"context"
	"errors"
	"fmt"

	"go.llib.dev/testcase/faultinject"
)

type FaultTag struct{}

func main() {
	defer faultinject.Enable()()
	ctx := context.Background()
	ctx = faultinject.Inject(ctx, FaultTag{}, errors.New("ignored"))
	fmt.Println(MyFuncWithFixErrorReplyFromTheFaultPoin(ctx)) // error is returned
}

func MyFuncWithFixErrorReplyFromTheFaultPoin(ctx context.Context) error {
	if _, ok := ctx.Value(FaultTag{}).(error); ok {
		return errors.New("my error value")
	}

	return nil
}

Index

• Constants
• Variables
• func After(returnErr *error, ctx context.Context, faults ...any)
• func Check(ctx context.Context, faults ...any) error
• func Enable() (Disable func())
• func EnableForTest(tb testingTB)
• func Enabled() bool
• func Finish(returnErr *error, ctx context.Context, faults ...any)
• func Inject(ctx context.Context, fault any, err error) context.Context
• type CallerFault

Examples

• Package
• Package (ChaosEngineeringWithExplicitFaultPoints)
• Package (FaultInjectWithFixErrorReplyFromTheFaultPoint)
• After
• CallerFault
• Check
• Finish
• Inject (ByTargetingCaller)

Constants

const DefaultErr errT = "fault injected"

Variables

var WaitForContextDoneTimeout = time.Second / 2

Functions

func After

func After(returnErr *error, ctx context.Context, faults ...any)

After is function that can be called from a deferred context, and will inject fault after the function finished its execution. The error pointer should point to the function's named return error variable. If the function encountered an actual error, fault injection is skipped. It is safe to use from production code.

Example

package main

import (
	"context"
	"fmt"

	"go.llib.dev/testcase/faultinject"
)

func main() {
	type fault struct{}
	ctx := faultinject.Inject(context.Background(), fault{}, fmt.Errorf("boom"))

	_ = func(ctx context.Context) (returnErr error) {
		defer faultinject.After(&returnErr, ctx, fault{})

		return nil
	}(ctx)
}

func Check

func Check(ctx context.Context, faults ...any) error

Check is a fault-injection helper method which check if there is an injected fault(s) in the given context. It checks for errors injected as context value, or ensures to trigger a CallerFault. It is safe to use from production code.

Example

package main

import (
	"context"

	"go.llib.dev/testcase/faultinject"
)

func main() {
	type FaultName struct{}
	ctx := context.Background()

	if err := faultinject.Check(ctx, FaultName{}); err != nil {
		return // err
	}
}

func Enable

func Enable() (Disable func())

func EnableForTest

func EnableForTest(tb testingTB)

func Enabled

func Enabled() bool

func Finish

func Finish(returnErr *error, ctx context.Context, faults ...any)

Finish is an alias for After

DEPRECATED: use After instead

Example

package main

import (
	"context"

	"go.llib.dev/testcase/faultinject"
)

func main() {
	type FaultName struct{}
	ctx := context.Background()

	_ = func(ctx context.Context) (rErr error) {
		defer faultinject.After(&rErr, ctx, FaultName{})

		return nil
	}(ctx)
}

func Inject

func Inject(ctx context.Context, fault any, err error) context.Context

Inject will arrange context to trigger fault injection for the provided fault.

Example (ByTargetingCaller)

package main

import (
	"context"
	"errors"
	"fmt"

	"go.llib.dev/testcase/faultinject"
)

func main() {
	ctx := faultinject.Inject(
		context.Background(),
		faultinject.CallerFault{
			Package:  "", // empty will match everything
			Receiver: "", //
			Function: "", //
		},
		errors.New("boom"),
	)

	fmt.Println(ctx.Err()) // "boom"
}

Types

type CallerFault

type CallerFault struct {
	Package  string
	Receiver string
	Function string
}

CallerFault allows you to inject Fault by Caller stack position.

Example

package main

import (
	"context"
	"fmt"

	"go.llib.dev/testcase/faultinject"
	"go.llib.dev/testcase/random"
)

func main() {
	defer faultinject.Enable()()
	ctx := context.Background()
	fmt.Println(MyFuncWithStandardContextErrCheck(ctx)) // no error

	fault := faultinject.CallerFault{ // inject Fault that targets a specific context Err check
		Function: "MyFuncWithStandardContextErrCheck", // selector to tell where to inject
	}
	err := random.New(random.CryptoSeed{}).Error()

	ctx = faultinject.Inject(ctx, fault, err)           // some random error)
	fmt.Println(MyFuncWithStandardContextErrCheck(ctx)) // Fault.Error injected and returned
}

func MyFuncWithStandardContextErrCheck(ctx context.Context) error {
	if err := ctx.Err(); err != nil {
		return err
	}

	return nil
}