scenario

package
v1.5.0-alpha4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 19, 2024 License: Apache-2.0 Imports: 48 Imported by: 0

Documentation

Overview

Package scenario provides a high-level testing framework for running Bacalhau jobs in different configurations and making assertions against the results.

The unit of measure is the `Scenario` which describes a Bacalhau network, a job to be submitted to it, and a set of checks to exercise that the job was executed as expected.

Scenarios can be used in standalone way (see `pkg/test/executor/test_runner.go`) or using the provided `ScenarioRunner` can be used.

As well as executing jobs against real executors, a Scenario can instead use the NoopExecutor to implement a mocked out job. This makes is easier to test network features without needing to invent a real job.

Example (Noop) 
package main

import (
	"context"
	"strings"
	"testing"

	"github.com/stretchr/testify/suite"

	"github.com/bacalhau-project/bacalhau/pkg/downloader"

	"github.com/bacalhau-project/bacalhau/pkg/models"

	"github.com/bacalhau-project/bacalhau/pkg/executor"
	"github.com/bacalhau-project/bacalhau/pkg/executor/noop"
	"github.com/bacalhau-project/bacalhau/pkg/system"
)

func noopScenario(t testing.TB) Scenario {
	return Scenario{
		Stack: &StackConfig{
			ExecutorConfig: noop.ExecutorConfig{
				ExternalHooks: noop.ExecutorConfigExternalHooks{
					JobHandler: func(ctx context.Context, jobID string, resultsDir string) (*models.RunCommandResult, error) {
						return executor.WriteJobResults(resultsDir, strings.NewReader("hello, world!\n"), nil, 0, nil, executor.OutputLimits{
							MaxStdoutFileLength:   system.MaxStdoutFileLength,
							MaxStdoutReturnLength: system.MaxStdoutReturnLength,
							MaxStderrFileLength:   system.MaxStderrFileLength,
							MaxStderrReturnLength: system.MaxStderrReturnLength,
						}), nil
					},
				},
			},
		},
		Job: &models.Job{
			Name:  t.Name(),
			Type:  models.JobTypeBatch,
			Count: 1,
			Tasks: []*models.Task{
				{
					Name: t.Name(),
					Engine: &models.SpecConfig{
						Type:   models.EngineNoop,
						Params: make(map[string]interface{}),
					},
				},
			},
		},
		ResultsChecker: FileEquals(downloader.DownloadFilenameStdout, "hello, world!\n"),
		JobCheckers:    WaitUntilSuccessful(1),
	}
}

type NoopTest struct {
	ScenarioRunner
}

func main() {
	// In a real example, use the testing.T passed to the TestXxx method.
	suite.Run(&testing.T{}, new(NoopTest))
}

func (suite *NoopTest) TestRunNoop() {
	suite.RunScenario(noopScenario(suite.T()))
}

Output:

Index

Examples

Constants

View Source 
const AllowedListedLocalPathsSuffix = string(os.PathSeparator) + "*"

Variables

This section is empty.

Functions

func CreateSourcePath added in v1.4.0 

func CreateSourcePath(rootSourceDir string) (string, error)

CreateSourcePath creates a file/dir path in the provided directory with a random name.

func GetAllScenarios 

func GetAllScenarios(t testing.TB) map[string]Scenario

func GetCompletedExecutionStates added in v1.5.0 

func GetCompletedExecutionStates(jobState *JobState) []*models.Execution

func GetFilteredExecutionStates added in v1.5.0 

func GetFilteredExecutionStates(jobState *JobState, filterState models.ExecutionStateType) []*models.Execution

func InlineData 

func InlineData(data []byte) *models.InputSource

InlineData will store the file directly inline in the storage spec. Unlike the other storage set-ups, this function loads the file immediately. This makes it possible to store things deeper into the Spec object without the test system needing to know how to prepare them.

Types

type CheckResults 

type CheckResults func(resultsDir string) error

A CheckResults is a function that will examine job output that has been written to storage and assert something about it. If the condition it is checking is false, it returns an error, else it returns nil.

func FileContains 

func FileContains(
	outputFilePath string,
	expectedStrings []string,
	expectedLines int,
) CheckResults

FileContains returns a CheckResults that asserts that the expected string is in the output file and that the file itself is of the correct size. If expectedLine is set to -1 then a line-check is not performed.

func FileEquals 

func FileEquals(
	outputFilePath string,
	expectedString string,
) CheckResults

FileEquals returns a CheckResults that asserts that the expected string is exactly equal to the full contents of the output file.

func ManyChecks 

func ManyChecks(checks ...CheckResults) CheckResults

ManyCheckes returns a CheckResults that runs the passed checkers and returns an error if any of them fail.

type CheckSubmitResponse 

type CheckSubmitResponse func(response *apimodels.PutJobResponse, err error) error

A CheckSubmitResponse is a function that will examine and validate submitJob response. Useful when validating that a job should be rejected.

func SubmitJobErrorContains 

func SubmitJobErrorContains(msg string) CheckSubmitResponse

func SubmitJobFail 

func SubmitJobFail() CheckSubmitResponse

SubmitJobFail returns a CheckSubmitResponse that asserts an error was returned when submitting a job.

func SubmitJobSuccess 

func SubmitJobSuccess() CheckSubmitResponse

SubmitJobSuccess returns a CheckSubmitResponse that asserts no error was returned when submitting a job.

type JobState added in v1.5.0 

type JobState struct {
	ID         string
	Executions []*models.Execution
	State      models.State[models.JobStateType]
}

func (*JobState) GroupExecutionsByState added in v1.5.0 

func (s *JobState) GroupExecutionsByState() map[models.ExecutionStateType][]*models.Execution

GroupExecutionsByState groups the executions by state

type JobStateLoader added in v1.5.0 

type JobStateLoader interface {
	GetJob(ctx context.Context, id string) (*JobState, error)
}

type Scenario 

type Scenario struct {
	// An optional set of configuration options that define the network of nodes
	// that the job will be run against. When unspecified, the Stack will
	// consist of one node with requester and compute nodes set up according to
	// their default configuration, and without a Noop executor.
	Stack *StackConfig

	// Setup routines which define data available to the job.
	// If nil, no storage will be set up.
	Inputs SetupStorage

	// Output volumes that must be available to the job. If nil, no output
	// volumes will be attached to the job.
	Outputs []*models.ResultPath

	// The job specification
	Job *models.Job

	// A function that will assert submitJob response is as expected.
	// if nil, will use SubmitJobSuccess by default.
	SubmitChecker CheckSubmitResponse

	// A function that will decide whether or not the job was successful. If
	// nil, no check will be performed on job outputs.
	ResultsChecker CheckResults

	// A set of checkers that will decide when the job has completed, and maybe
	// whether it was successful or not. If empty, the job will not be waited
	// for once it has been submitted.
	JobCheckers []StateChecks
}

A Scenario represents a repeatable test case of submitting a job against a Bacalhau network.

The Scenario defines:

  • the topology and configuration of network that is required
  • the job that will be submitted
  • the conditions for the job to be considered successful or not

Most of the fields in a Scenario are optional and sensible defaults will be used if they are not present. All that is really required is the Spec which details what job to run.

func AwkFile 

func AwkFile(t testing.TB) Scenario

func CatFileToStdout 

func CatFileToStdout(t testing.TB) Scenario

func CatFileToVolume 

func CatFileToVolume(t testing.TB) Scenario

func GrepFile 

func GrepFile(t testing.TB) Scenario

func SedFile 

func SedFile(t testing.TB) Scenario

func WasmCsvTransform 

func WasmCsvTransform(t testing.TB) Scenario
func WasmDynamicLink(t testing.TB) Scenario

func WasmEnvVars 

func WasmEnvVars(t testing.TB) Scenario

func WasmExitCode added in v0.3.24 

func WasmExitCode(t testing.TB) Scenario

func WasmHelloWorld 

func WasmHelloWorld(t testing.TB) Scenario

func WasmLogTest added in v0.3.26 

func WasmLogTest(t testing.TB) Scenario

type ScenarioRunner 

type ScenarioRunner struct {
	suite.Suite
	Ctx    context.Context
	Config types.Bacalhau
	Repo   *repo.FsRepo
}

The ScenarioRunner is an object that can run a Scenario.

It will spin up an appropriate Devstack for the Scenario, submit and wait for the job to complete, and then make assertions against the results of the job.

ScenarioRunner implements a number of testify/suite interfaces making it appropriate as the basis for a test suite. If a test suite composes itself from the ScenarioRunner then default set up and tear down methods that instrument and configure the test will be used. Test suites should not define their own set up or tear down routines.

func (*ScenarioRunner) RunScenario 

func (s *ScenarioRunner) RunScenario(scenario Scenario) string

RunScenario runs the Scenario.

Spin up a devstack, execute the job, check the results, and tear down the devstack.

func (*ScenarioRunner) SetupTest 

func (s *ScenarioRunner) SetupTest()

type ScenarioTestSuite 

type ScenarioTestSuite interface {
	suite.SetupTestSuite
	suite.TearDownTestSuite
	suite.TestingSuite
}

type SetupStorage 

type SetupStorage func(
	ctx context.Context,
) ([]*models.InputSource, error)

A SetupStorage is a function that return a model.StorageSpec representing some data that has been prepared for use by a job. It is the responsibility of the function to ensure that the data has been set up correctly.

func ManyStores 

func ManyStores(stores ...SetupStorage) SetupStorage

ManyStores runs all of the passed setups and returns the model.StorageSpecs associated with all of them. If any of them fail, the error from the first to fail will be returned.

func StoredFile 

func StoredFile(
	rootSourceDir string,
	filePath string,
	mountPath string,
) SetupStorage

StoredFile will copy the passed file or directory into the provided mount path on the local filesystem and return the path to the file or directory in a model.StorageSpec.

func StoredText 

func StoredText(
	rootSourceDir string,
	fileContents string,
	mountPath string,
) SetupStorage

StoredText will store the passed string as a file on the local filesystem and return the path to the file in a *models.InputSource.

func URLDownload 

func URLDownload(
	server *httptest.Server,
	urlPath string,
	mountPath string,
) SetupStorage

URLDownload will return a model.StorageSpec referencing a file on the passed HTTP test server.

type StackConfig 

type StackConfig struct {
	*devstack.DevStackOptions
	node.ComputeConfig
	node.RequesterConfig
	noop.ExecutorConfig
}

All the information that is needed to uniquely define a devstack.

type StateChecks added in v1.5.0 

type StateChecks func(s *JobState) (bool, error)

func WaitExecutionsThrowErrors added in v1.5.0 

func WaitExecutionsThrowErrors(errorStates []models.ExecutionStateType) StateChecks

error if there are any errors in any of the states

func WaitForExecutionStates added in v1.5.0 

func WaitForExecutionStates(requiredStateCounts map[models.ExecutionStateType]int) StateChecks

wait for the given number of different states to occur

func WaitForSuccessfulCompletion added in v1.5.0 

func WaitForSuccessfulCompletion() StateChecks

func WaitForTerminalStates added in v1.5.0 

func WaitForTerminalStates() StateChecks

WaitForTerminalStates it is possible that a job is in a terminal state, but some executions are still running, such as when one node publishes the result before others. for that reason, we consider a job to be in a terminal state when: - all executions are in a terminal state - the job is in a terminal state to account for possible retries TODO validate this is comment is still valid.

func WaitForUnsuccessfulCompletion added in v1.5.0 

func WaitForUnsuccessfulCompletion() StateChecks

func WaitUntilSuccessful 

func WaitUntilSuccessful(nodes int) []StateChecks

WaitUntilSuccessful returns a set of job.CheckStatesFunctions that will wait until the job they are checking reaches the Completed state on the passed number of nodes. The checks will fail if any job errors.

type StateResolver added in v1.5.0 

type StateResolver struct {
	// contains filtered or unexported fields
}

func NewStateResolverFromAPI added in v1.5.0 

func NewStateResolverFromAPI(api client.API) *StateResolver

func NewStateResolverFromStore added in v1.5.0 

func NewStateResolverFromStore(s jobstore.Store) *StateResolver

func (*StateResolver) JobState added in v1.5.0 

func (s *StateResolver) JobState(ctx context.Context, id string) (*JobState, error)

func (*StateResolver) Wait added in v1.5.0 

func (s *StateResolver) Wait(ctx context.Context, id string, until ...StateChecks) error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.