havoc

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 26, 2024 License: MIT Imports: 17 Imported by: 2

README

Havoc

The havoc package is a Go library designed to facilitate chaos testing within Kubernetes environments using Chaos Mesh. It offers a structured way to define, execute, and manage chaos experiments as code, directly integrated into Go applications or testing suites. This package simplifies the creation and control of Chaos Mesh experiments, including network chaos, pod failures, and stress testing on Kubernetes clusters.

Features

  • Chaos Object Management: Easily create, update, pause, resume, and delete chaos experiments using Go structures and methods.
  • Lifecycle Hooks: Utilize chaos listeners to hook into lifecycle events of chaos experiments, such as creation, start, pause, resume, and finish.
  • Support for Various Chaos Experiments: Create and manage different types of chaos experiments like NetworkChaos, IOChaos, StressChaos, PodChaos, and HTTPChaos.
  • Chaos Experiment Status Monitoring: Monitor and react to the status of chaos experiments programmatically.

Installation

To use havoc in your project, ensure you have a Go environment setup. Then, install the package using go get:

go get -u github.com/goplugin/plugin-testing-framework/havoc

Ensure your Kubernetes cluster is accessible and that you have Chaos Mesh installed and configured.

Monitoring and Observability in Chaos Experiments

havoc enhances chaos experiment observability through structured logging and Grafana annotations, facilitated by implementing the ChaosListener interface. This approach allows for detailed monitoring, debugging, and visual representation of chaos experiments' impact.

Structured Logging with ChaosLogger

ChaosLogger leverages the zerolog library to provide structured, queryable logging of chaos events. It automatically logs key lifecycle events such as creation, start, pause, and termination of chaos experiments, including detailed contextual information.

Instantiate ChaosLogger and register it as a listener to your chaos experiments:

logger := havoc.NewChaosLogger()
chaos.AddListener(logger)

Default package logger

havoc/logger.go contains default Logger instance for the package.

Visual Monitoring with Grafana Annotations

SingleLineGrafanaAnnotator is a ChaosListener that annotates Grafana dashboards with chaos experiment events. This visual representation helps correlate chaos events with their effects on system metrics and logs.

Initialize SingleLineGrafanaAnnotator with your Grafana instance details and register it alongside ChaosLogger:

annotator := havoc.NewSingleLineGrafanaAnnotator(
    "http://grafana-instance.com",
    "grafana-access-token",
    "dashboard-uid",
)
chaos.AddListener(annotator)

Creating a Chaos Experiment

To create a chaos experiment, define the chaos object options, initialize a chaos experiment with NewChaos, and then call Create to start the experiment.

Here is an example of creating and starting a PodChaos experiment:

package main

import (
    "context"
    "github.com/goplugin/plugin-testing-framework/havoc"
    "github.com/chaos-mesh/chaos-mesh/api/v1alpha1"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "time"
)

func main() {
    // Initialize dependencies
    client, err := havoc.NewChaosMeshClient()
    if err != nil {
        panic(err)
    }
    logger := havoc.NewChaosLogger()
    annotator := havoc.NewSingleLineGrafanaAnnotator(
        "http://grafana-instance.com",
        "grafana-access-token",
        "dashboard-uid",
    )

    // Define chaos experiment
    podChaos := &v1alpha1.PodChaos{ /* PodChaos spec */ }
    chaos, err := havoc.NewChaos(havoc.ChaosOpts{
        Object:      podChaos,
        Description: "Pod failure example",
        DelayCreate: 5 * time.Second,
        Client:      client,
    })
    if err != nil {
        panic(err)
    }

    // Register listeners
    chaos.AddListener(logger)
    chaos.AddListener(annotator)

    // Start chaos experiment
    chaos.Create(context.Background())

    // Manage chaos lifecycle...
}

Test Example

func TestChaosDON(t *testing.T) {
	testDuration := time.Minute * 60

    // Load test config
	cfg := &config.MercuryQAEnvChaos{}

	// Define chaos experiments and their schedule

	k8sClient, err := havoc.NewChaosMeshClient()
	require.NoError(t, err)

	// Test 3.2: Disable 2 nodes simultaneously

	podFailureChaos4, err := k8s_chaos.MercuryPodChaosSchedule(k8s_chaos.MercuryScheduledPodChaosOpts{
		Name:        "schedule-don-ocr-node-failure-4",
		Description: "Disable 2 nodes (clc-ocr-mercury-arb-testnet-qa-nodes-3 and clc-ocr-mercury-arb-testnet-qa-nodes-4)",
		DelayCreate: time.Minute * 0,
		Duration:    time.Minute * 20,
		Namespace:   cfg.ChaosNodeNamespace,
		PodSelector: v1alpha1.PodSelector{
			Mode: v1alpha1.AllMode,
			Selector: v1alpha1.PodSelectorSpec{
				GenericSelectorSpec: v1alpha1.GenericSelectorSpec{
					Namespaces: []string{cfg.ChaosNodeNamespace},
					ExpressionSelectors: v1alpha1.LabelSelectorRequirements{
						{
							Key:      "app.kubernetes.io/instance",
							Operator: "In",
							Values: []string{
								"clc-ocr-mercury-arb-testnet-qa-nodes-3",
								"clc-ocr-mercury-arb-testnet-qa-nodes-4",
							},
						},
					},
				},
			},
		},
		Client: k8sClient,
	})
	require.NoError(t, err)

	// Test 3.3: Disable 3 nodes simultaneously

	podFailureChaos5, err := k8s_chaos.MercuryPodChaosSchedule(k8s_chaos.MercuryScheduledPodChaosOpts{
		Name:        "schedule-don-ocr-node-failure-5",
		Description: "Disable 3 nodes (clc-ocr-mercury-arb-testnet-qa-nodes-3, clc-ocr-mercury-arb-testnet-qa-nodes-4 and clc-ocr-mercury-arb-testnet-qa-nodes-5)",
		DelayCreate: time.Minute * 40,
		Duration:    time.Minute * 20,
		Namespace:   cfg.ChaosNodeNamespace,
		PodSelector: v1alpha1.PodSelector{
			Mode: v1alpha1.AllMode,
			Selector: v1alpha1.PodSelectorSpec{
				GenericSelectorSpec: v1alpha1.GenericSelectorSpec{
					Namespaces: []string{cfg.ChaosNodeNamespace},
					ExpressionSelectors: v1alpha1.LabelSelectorRequirements{
						{
							Key:      "app.kubernetes.io/instance",
							Operator: "In",
							Values: []string{
								"clc-ocr-mercury-arb-testnet-qa-nodes-3",
								"clc-ocr-mercury-arb-testnet-qa-nodes-4",
								"clc-ocr-mercury-arb-testnet-qa-nodes-5",
							},
						},
					},
				},
			},
		},
		Client: k8sClient,
	})
	require.NoError(t, err)

	chaosList := []havoc.ChaosEntity{
		podFailureChaos4,
		podFailureChaos5,
	}

	for _, chaos := range chaosList {
		chaos.AddListener(havoc.NewChaosLogger())
		chaos.AddListener(havoc.NewSingleLineGrafanaAnnotator(cfg.GrafanaURL, cfg.GrafanaToken, cfg.GrafanaDashboardUID))

		// Fail the test if the chaos object already exists
		exists, err := havoc.ChaosObjectExists(chaos.GetObject(), k8sClient)
		require.NoError(t, err)
		require.False(t, exists, "chaos object already exists: %s. Delete it before starting the test", chaos.GetChaosName())

		chaos.Create(context.Background())
	}

	t.Cleanup(func() {
		for _, chaos := range chaosList {
			// Delete chaos object if it still exists
			chaos.Delete(context.Background())
		}
	})

	// Simulate user activity/load for the duration of the chaos experiments
	runUserLoad(t, cfg, testDuration)
}

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var Logger zerolog.Logger

Default logger

Functions

func ChaosObjectExists 

func ChaosObjectExists(object client.Object, c client.Client) (bool, error)

func CreateLogger 

func CreateLogger(config LoggerConfig) zerolog.Logger

Create initializes a zerolog.Logger based on the specified configuration.

func NewChaosMeshClient 

func NewChaosMeshClient() (client.Client, error)

NewChaosMeshClient initializes and returns a new Kubernetes client configured for Chaos Mesh

func Ptr 

func Ptr[T any](value T) *T

func WaitForAllChaosRunning 

func WaitForAllChaosRunning(chaosObjects []*Chaos, timeoutDuration time.Duration) error

WaitForAllChaosRunning waits for all chaos experiments to be running

Types

type Chaos 

type Chaos struct {
	Object      client.Object
	Description string
	DelayCreate time.Duration // Delay before creating the chaos object
	Status      ChaosStatus
	Client      client.Client
	// contains filtered or unexported fields
}

func NewChaos 

func NewChaos(opts ChaosOpts) (*Chaos, error)

func (*Chaos) AddListener 

func (c *Chaos) AddListener(listener ChaosListener)

func (*Chaos) Create 

func (c *Chaos) Create(ctx context.Context)

Create initiates a delayed creation of a chaos object, respecting context cancellation and deletion requests. It uses a timer based on `DelayCreate` and calls `create` method upon expiration unless preempted by deletion.

func (*Chaos) Delete 

func (c *Chaos) Delete(ctx context.Context) error

func (*Chaos) GetChaosDescription 

func (c *Chaos) GetChaosDescription() string

func (*Chaos) GetChaosDuration 

func (c *Chaos) GetChaosDuration() (time.Duration, error)

func (*Chaos) GetChaosEvents 

func (c *Chaos) GetChaosEvents() (*corev1.EventList, error)

func (*Chaos) GetChaosKind 

func (c *Chaos) GetChaosKind() string

func (*Chaos) GetChaosName 

func (c *Chaos) GetChaosName() string

func (*Chaos) GetChaosSpec 

func (c *Chaos) GetChaosSpec() interface{}

func (*Chaos) GetChaosStatus 

func (c *Chaos) GetChaosStatus() (*v1alpha1.ChaosStatus, error)

func (*Chaos) GetChaosTypeStr 

func (c *Chaos) GetChaosTypeStr() string

func (*Chaos) GetEndTime 

func (c *Chaos) GetEndTime() time.Time

GetEndTime returns the time when the chaos experiment ended

func (*Chaos) GetExpectedEndTime 

func (c *Chaos) GetExpectedEndTime() (time.Time, error)

GetExpectedEndTime returns the time when the chaos experiment is expected to end

func (*Chaos) GetExperimentStatus 

func (c *Chaos) GetExperimentStatus() (v1alpha1.ExperimentStatus, error)

func (*Chaos) GetObject 

func (c *Chaos) GetObject() client.Object

func (*Chaos) GetStartTime 

func (c *Chaos) GetStartTime() time.Time

GetStartTime returns the time when the chaos experiment started

func (*Chaos) Pause 

func (c *Chaos) Pause(ctx context.Context) error

func (*Chaos) Resume 

func (c *Chaos) Resume(ctx context.Context) error

func (*Chaos) Update 

func (c *Chaos) Update(ctx context.Context) error

type ChaosEntity 

type ChaosEntity interface {
	// Create initializes and submits the chaos object to Kubernetes.
	Create(ctx context.Context)
	// Delete removes the chaos object from Kubernetes.
	Delete(ctx context.Context) error
	// Registers a listener to receive updates about the chaos object's lifecycle.
	AddListener(listener ChaosListener)

	GetObject() client.Object
	GetChaosName() string
	GetChaosDescription() string
	GetChaosDuration() (time.Duration, error)
	GetChaosSpec() interface{}
	GetStartTime() time.Time
	GetEndTime() time.Time
	GetExpectedEndTime() (time.Time, error)
}

ChaosEntity is an interface that defines common behaviors for chaos management entities.

type ChaosEventDetails 

type ChaosEventDetails struct {
	Event string
	Chaos *Chaos
	Error error
}

type ChaosListener 

type ChaosListener interface {
	OnChaosCreated(chaos Chaos)
	OnChaosCreationFailed(chaos Chaos, reason error)
	OnChaosStarted(chaos Chaos)
	OnChaosPaused(chaos Chaos)
	OnChaosEnded(chaos Chaos)         // When the chaos is finished or deleted
	OnChaosStatusUnknown(chaos Chaos) // When the chaos status is unknown
	OnScheduleCreated(chaos Schedule)
	OnScheduleDeleted(chaos Schedule) // When the chaos is finished or deleted
}

type ChaosLogger 

type ChaosLogger struct {
	// contains filtered or unexported fields
}

func NewChaosLogger 

func NewChaosLogger(logger zerolog.Logger) *ChaosLogger

func (ChaosLogger) OnChaosCreated 

func (l ChaosLogger) OnChaosCreated(chaos Chaos)

func (ChaosLogger) OnChaosCreationFailed 

func (l ChaosLogger) OnChaosCreationFailed(chaos Chaos, reason error)

func (ChaosLogger) OnChaosDeleted 

func (l ChaosLogger) OnChaosDeleted(chaos Chaos)

func (ChaosLogger) OnChaosEnded 

func (l ChaosLogger) OnChaosEnded(chaos Chaos)

func (ChaosLogger) OnChaosPaused 

func (l ChaosLogger) OnChaosPaused(chaos Chaos)

func (ChaosLogger) OnChaosStarted 

func (l ChaosLogger) OnChaosStarted(chaos Chaos)

func (ChaosLogger) OnChaosStatusUnknown 

func (l ChaosLogger) OnChaosStatusUnknown(chaos Chaos)

func (ChaosLogger) OnScheduleCreated 

func (l ChaosLogger) OnScheduleCreated(schedule Schedule)

func (ChaosLogger) OnScheduleDeleted 

func (l ChaosLogger) OnScheduleDeleted(schedule Schedule)

type ChaosOpts 

type ChaosOpts struct {
	Object      client.Object
	Description string
	DelayCreate time.Duration
	Client      client.Client
	Listeners   []ChaosListener
	Logger      *zerolog.Logger
}

type ChaosStatus 

type ChaosStatus string

ChaosStatus represents the status of a chaos experiment. 

const (
	StatusCreated        ChaosStatus = "created"
	StatusCreationFailed ChaosStatus = "creation_failed"
	StatusRunning        ChaosStatus = "running"
	StatusPaused         ChaosStatus = "paused"
	StatusFinished       ChaosStatus = "finished"
	StatusDeleted        ChaosStatus = "deleted"
	StatusUnknown        ChaosStatus = "unknown" // For any state that doesn't match the above
)

These constants define possible states of a chaos experiment.

type LoggerConfig 

type LoggerConfig struct {
	LogOutput string // "json-console" for JSON output, empty or "console" for human-friendly console output
	LogLevel  string // Log level (e.g., "info", "debug", "error")
	LogType   string // Custom log type identifier
}

type NetworkChaosOpts 

type NetworkChaosOpts struct {
	Name        string
	Description string
	DelayCreate time.Duration
	Delay       *v1alpha1.DelaySpec
	Loss        *v1alpha1.LossSpec
	NodeCount   int
	Duration    time.Duration
	Selector    v1alpha1.PodSelectorSpec
	K8sClient   client.Client
}

func (*NetworkChaosOpts) Validate 

func (o *NetworkChaosOpts) Validate() error

type PodChaosOpts 

type PodChaosOpts struct {
	Name        string
	Description string
	DelayCreate time.Duration
	NodeCount   int
	Duration    time.Duration
	Spec        v1alpha1.PodChaosSpec
	K8sClient   client.Client
}

type RangeGrafanaAnnotator 

type RangeGrafanaAnnotator struct {
	// contains filtered or unexported fields
}

func NewRangeGrafanaAnnotator 

func NewRangeGrafanaAnnotator(grafanaURL, grafanaToken, dashboardUID string, logger zerolog.Logger) *RangeGrafanaAnnotator

func (RangeGrafanaAnnotator) OnChaosCreated 

func (l RangeGrafanaAnnotator) OnChaosCreated(chaos Chaos)

func (RangeGrafanaAnnotator) OnChaosEnded 

func (l RangeGrafanaAnnotator) OnChaosEnded(chaos Chaos)

func (RangeGrafanaAnnotator) OnChaosPaused 

func (l RangeGrafanaAnnotator) OnChaosPaused(chaos Chaos)

func (RangeGrafanaAnnotator) OnChaosStarted 

func (l RangeGrafanaAnnotator) OnChaosStarted(chaos Chaos)

func (RangeGrafanaAnnotator) OnChaosStatusUnknown 

func (l RangeGrafanaAnnotator) OnChaosStatusUnknown(chaos Chaos)

func (RangeGrafanaAnnotator) OnScheduleCreated 

func (l RangeGrafanaAnnotator) OnScheduleCreated(chaos Schedule)

func (RangeGrafanaAnnotator) OnScheduleDeleted 

func (l RangeGrafanaAnnotator) OnScheduleDeleted(chaos Schedule)

type Schedule 

type Schedule struct {
	Object      *v1alpha1.Schedule
	Description string
	DelayCreate time.Duration // Delay before creating the chaos object
	Duration    time.Duration // Duration for which the chaos object should exist
	Status      ChaosStatus
	Client      client.Client
	// contains filtered or unexported fields
}

func NewSchedule 

func NewSchedule(opts ScheduleOpts) (*Schedule, error)

func (*Schedule) AddListener 

func (s *Schedule) AddListener(listener ChaosListener)

func (*Schedule) Create 

func (s *Schedule) Create(ctx context.Context)

Create initiates a delayed creation of a chaos object, respecting context cancellation and deletion requests. It uses a timer based on `DelayCreate` and calls `create` method upon expiration unless preempted by deletion.

func (*Schedule) Delete 

func (s *Schedule) Delete(ctx context.Context) error

func (*Schedule) GetChaosDescription 

func (s *Schedule) GetChaosDescription() string

func (*Schedule) GetChaosDuration 

func (s *Schedule) GetChaosDuration() (time.Duration, error)

func (*Schedule) GetChaosName 

func (s *Schedule) GetChaosName() string

func (*Schedule) GetChaosSpec 

func (s *Schedule) GetChaosSpec() interface{}

func (*Schedule) GetEndTime 

func (s *Schedule) GetEndTime() time.Time

func (*Schedule) GetExpectedEndTime 

func (s *Schedule) GetExpectedEndTime() (time.Time, error)

func (*Schedule) GetObject 

func (s *Schedule) GetObject() client.Object

func (*Schedule) GetStartTime 

func (s *Schedule) GetStartTime() time.Time

type ScheduleOpts 

type ScheduleOpts struct {
	Object      *v1alpha1.Schedule
	Description string
	DelayCreate time.Duration
	Duration    time.Duration
	Client      client.Client
	Listeners   []ChaosListener
	Logger      *zerolog.Logger
}

type ScheduleStatus 

type ScheduleStatus string
const (
	ScheduleStatusCreated ScheduleStatus = "created"
	ScheduleStatusDeleted ScheduleStatus = "deleted"
	ScheduleStatusUnknown ScheduleStatus = "unknown" // For any state that doesn't match the above
)

type SimplifiedEvent 

type SimplifiedEvent struct {
	LastTimestamp string
	Type          string
	Message       string
}

type SingleLineGrafanaAnnotator 

type SingleLineGrafanaAnnotator struct {
	// contains filtered or unexported fields
}

func NewSingleLineGrafanaAnnotator 

func NewSingleLineGrafanaAnnotator(grafanaURL, grafanaToken, dashboardUID string, logger zerolog.Logger) *SingleLineGrafanaAnnotator

func (SingleLineGrafanaAnnotator) OnChaosCreated 

func (l SingleLineGrafanaAnnotator) OnChaosCreated(chaos Chaos)

func (SingleLineGrafanaAnnotator) OnChaosCreationFailed 

func (l SingleLineGrafanaAnnotator) OnChaosCreationFailed(chaos Chaos, reason error)

func (SingleLineGrafanaAnnotator) OnChaosEnded 

func (l SingleLineGrafanaAnnotator) OnChaosEnded(chaos Chaos)

func (SingleLineGrafanaAnnotator) OnChaosPaused 

func (l SingleLineGrafanaAnnotator) OnChaosPaused(chaos Chaos)

func (SingleLineGrafanaAnnotator) OnChaosStarted 

func (l SingleLineGrafanaAnnotator) OnChaosStarted(chaos Chaos)

func (SingleLineGrafanaAnnotator) OnChaosStatusUnknown 

func (l SingleLineGrafanaAnnotator) OnChaosStatusUnknown(chaos Chaos)

func (SingleLineGrafanaAnnotator) OnScheduleCreated 

func (l SingleLineGrafanaAnnotator) OnScheduleCreated(s Schedule)

func (SingleLineGrafanaAnnotator) OnScheduleDeleted 

func (l SingleLineGrafanaAnnotator) OnScheduleDeleted(s Schedule)

type StressChaosOpts 

type StressChaosOpts struct {
	Name        string
	Description string
	DelayCreate time.Duration
	NodeCount   int
	Stressors   *v1alpha1.Stressors
	Duration    time.Duration
	Selector    v1alpha1.PodSelectorSpec
	K8sClient   client.Client
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.