testutil

package
v0.0.0-...-90deddd Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 18, 2023 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Overview

Copyright 2018 GRAIL, Inc. All rights reserved. Use of this source code is governed by the Apache 2.0 license that can be found in the LICENSE file.

Package testutil contains various utilities for testing Reflow functionality.

Index

Constants

This section is empty.

Variables

View Source
var Resources = reflow.Resources{"mem": 500 << 20, "cpu": 1, "disk": 10}

Resources is a convenient set of resources to use for testing.

View Source
var Transferer reflow.Transferer = &transferer{}

Transferer is a simple transferer for testing; it does not apply any limits or other policies.

Functions

func EvalAsync

func EvalAsync(ctx context.Context, e *flow.Eval) <-chan EvalResult

EvalAsync evaluates Flow f on Eval e asynchronously. It also ensures that any background tasks are completed before reporting completion. EvalAsync expects that e's toplevel flow returns a Fileset.

func EvalFlowAsync

func EvalFlowAsync(ctx context.Context, e *flow.Eval) <-chan EvalFlowResult

EvalFlowAsync evaluates a flow and returns the result as a value. This function is similar to EvalAsync. The only difference being that EvalAsync expects a fileset value whereas EvalFlowAsync doesn't restrict the value to any specific type. TODO(prasadgopal): move all tests to use EvalFlowAsync and let callers assert the type.

func Exists

func Exists(e *flow.Eval, keys ...digest.Digest) bool

Exists tells whether a value has been cached for the provided keys in the FilesetV2 format. Exists checks whether all the objects are present in the Eval's repository.

func File

func File(contents string) reflow.File

File returns a file object representing the given contents.

func Files

func Files(files ...string) reflow.Fileset

Files returns a value comprising the given files with contents derived from their names.

func List

func List(values ...reflow.Fileset) reflow.Fileset

List constructs a list value.

func NewNopTaskDB

func NewNopTaskDB(repo reflow.Repository) taskdb.TaskDB

NewNopTaskDB returns a nop taskdb (for tests).

func RunReflowTests

func RunReflowTests(t *testing.T, testFiles []string)

RunReflowTests executes .rf tests from disk

func SkipIfNoCreds

func SkipIfNoCreds(t *testing.T)

SkipIfNoCreds allows a test to be skipped if no credentials are found. Caution: Renaming/removing this will prevent execution of tests (with credentials) which call this.

func Value

func Value(e *flow.Eval, key digest.Digest) reflow.Fileset

Value returns the fileset stored for the provided key in the cache configured in Eval e.

func WriteCache

func WriteCache(e *flow.Eval, key digest.Digest, files ...string)

WriteCache writes the provided files into the eval's repository and registers a Fileset cache assoc.

func WriteCacheFileset

func WriteCacheFileset(e *flow.Eval, key digest.Digest, fs reflow.Fileset)

WriteCacheFileset writes the provided fileset into the eval's repository and registers a Fileset cache assoc.

func WriteFile

func WriteFile(r reflow.Repository, content string) reflow.File

WriteFile writes the provided contents into repository r and returns the corresponding Reflow file.

func WriteFiles

func WriteFiles(r reflow.Repository, files ...string) reflow.Fileset

WriteFiles writes the provided files into the repository r and returns a Fileset as in Files.

Types

type Cache

type Cache struct {
	// contains filtered or unexported fields
}

Cache is a Reflow cache that stores values (but not objects) in memory. It implements assoc.Assoc.

func (*Cache) Delete

func (c *Cache) Delete(ctx context.Context, id digest.Digest) error

Delete removes the key id from the cache.

func (*Cache) Exists

func (c *Cache) Exists(f *flow.Flow) bool

Exists tells whether a value has been stored for flow f.

func (*Cache) ExistsAll

func (c *Cache) ExistsAll(f *flow.Flow) bool

ExistsAll tells whether a value has been stored for flow f, for all of its cache keys.

func (*Cache) Init

func (c *Cache) Init()

Init initializes (or re-initializes) a Cache.

func (*Cache) Lookup

func (c *Cache) Lookup(ctx context.Context, id digest.Digest) (reflow.Fileset, error)

Lookup returns the value stored at id, or else an errors.NotExist.

func (*Cache) NeedTransfer

func (c *Cache) NeedTransfer(ctx context.Context, dst reflow.Repository, v reflow.Fileset) ([]reflow.File, error)

NeedTransfer returns the file objects in v that are missing from repository dst.

func (*Cache) Repository

func (c *Cache) Repository() reflow.Repository

Repository is not implemented.

func (*Cache) Transfer

func (c *Cache) Transfer(ctx context.Context, dst reflow.Repository, v reflow.Fileset) error

Transfer is not implemented in Cache.

func (*Cache) Value

func (c *Cache) Value(f *flow.Flow) reflow.Fileset

Value returns the value stored for flow f.

func (*Cache) Write

func (c *Cache) Write(ctx context.Context, id digest.Digest, v reflow.Fileset, repo reflow.Repository) error

Write stores the value v at key id.

type EvalFlowResult

type EvalFlowResult struct {
	// Val is the value produced by the flow
	Val values.T
	// Err is the error produced by the flow, if any.
	Err error
}

EvalFlowResult is the result of asynchronously evaluating a flow in EvalFlowAsync.

type EvalResult

type EvalResult struct {
	Val reflow.Fileset
	Err error
}

EvalResult stores the result of an asynchronous evaluation.

type Exec

type Exec struct {
	// contains filtered or unexported fields
}

Exec is the Exec type used by testExecutor. They permit the caller to rendezvous on results.

func (*Exec) Config

func (e *Exec) Config() reflow.ExecConfig

Config returns the exec's ExecConfig.

func (*Exec) Error

func (e *Exec) Error(err error)

Error rendezvous the error err as the result of this testExec.

func (*Exec) ID

func (e *Exec) ID() digest.Digest

ID returns the exec's ID

func (*Exec) Inspect

func (e *Exec) Inspect(ctx context.Context, repo *url.URL) (resp reflow.InspectResponse, err error)

Inspect rendezvous the result of this exec and returns the inspection output.

func (*Exec) Logs

func (e *Exec) Logs(ctx context.Context, stdout bool, stderr bool, follow bool) (io.ReadCloser, error)

Logs returns a ReadCloser for the exec.

func (*Exec) Ok

func (e *Exec) Ok(res reflow.Result)

Ok rendezvous the value v as a successful result by this testExec.

func (*Exec) OkInspect

func (e *Exec) OkInspect(res reflow.Result, inspect reflow.ExecInspect)

func (*Exec) Promote

func (e *Exec) Promote(ctx context.Context) error

Promote is a no-op for the test exec.

func (*Exec) RemoteLogs

func (e *Exec) RemoteLogs(ctx context.Context, stdout bool) (l reflow.RemoteLogs, err error)

func (*Exec) Result

func (e *Exec) Result(ctx context.Context) (reflow.Result, error)

Value rendezvous the result (value or error) of this exec.

func (*Exec) Shell

func (e *Exec) Shell(ctx context.Context) (io.ReadWriteCloser, error)

Shell is not implemented

func (*Exec) URI

func (e *Exec) URI() string

URI is not implemented

func (*Exec) Wait

func (e *Exec) Wait(ctx context.Context) error

Wait rendezvous this exec.

type ExecResult

type ExecResult struct {
	Result  reflow.Result
	Inspect reflow.ExecInspect
}

ExecResult stores the result of a completed exec.

type Executor

type Executor struct {
	reflow.Executor
	Have reflow.Resources

	Repo reflow.Repository
	// contains filtered or unexported fields
}

Executor implements Executor for testing purposes. It allows the caller to await creation of Execs, to introspect execs in the executor, and to set exec results.

func (*Executor) Equiv

func (e *Executor) Equiv(flows ...*flow.Flow) bool

Equiv tells whether this executor contains precisely a set of flows.

func (*Executor) Error

func (e *Executor) Error(ctx context.Context, f *flow.Flow, err error)

Error defines an erroneous result for the flow.

func (*Executor) Exec

func (e *Executor) Exec(ctx context.Context, f *flow.Flow) *Exec

Exec rendezvous the Exec for the provided flow.

func (*Executor) Execs

func (e *Executor) Execs(ctx context.Context) ([]reflow.Exec, error)

Execs enumerates the execs managed by this executor.

func (*Executor) Get

func (e *Executor) Get(ctx context.Context, id digest.Digest) (reflow.Exec, error)

Get retrieves an exec.

func (*Executor) Init

func (e *Executor) Init()

Init initializes the test executor.

func (*Executor) Ok

func (e *Executor) Ok(ctx context.Context, f *flow.Flow, res interface{})

Ok defines a successful result for a Flow.

func (*Executor) Pending

func (e *Executor) Pending(f *flow.Flow) bool

Pending returns whether the flow f has a pending execution.

func (*Executor) Put

func (e *Executor) Put(ctx context.Context, id digest.Digest, config reflow.ExecConfig) (reflow.Exec, error)

Put defines a new exec (idempotently).

func (*Executor) Remove

func (e *Executor) Remove(ctx context.Context, id digest.Digest) error

Remove removes the exec with id if it exists or returns an error.

func (*Executor) Repository

func (e *Executor) Repository() reflow.Repository

Repository returns this executor's repository.

func (*Executor) Resolve

func (*Executor) Resources

func (e *Executor) Resources() reflow.Resources

Resources returns this executor's total resources.

func (*Executor) Wait

func (e *Executor) Wait(ctx context.Context, f *flow.Flow)

Wait blocks until a Flow is defined in the executor.

func (*Executor) WaitAny

func (e *Executor) WaitAny(ctx context.Context, flows ...*flow.Flow) *flow.Flow

WaitAny returns the first of flows to be defined.

type ExpectGetFilerRepository

type ExpectGetFilerRepository struct {
	*ExpectRepository
}

ExpectGetFilerRepository is an ExpectRepository which also implements GetFile.

func (*ExpectGetFilerRepository) GetFile

GetFile implements the repository's GetFile call.

type ExpectRepository

type ExpectRepository struct {
	*testing.T          // the testing instance to use
	RepoURL    *url.URL // the repository URL
	// contains filtered or unexported fields
}

ExpectRepository is a Repository implementation used for testing; it takes a script of expected calls and replies. Violations are reported to the testing.T instance.

func NewExpectRepository

func NewExpectRepository(t *testing.T, rawurl string) *ExpectRepository

NewExpectRepository creates a new testing repository.

func (*ExpectRepository) Collect

Collect is not supported for the expect repository.

func (*ExpectRepository) CollectWithThreshold

func (r *ExpectRepository) CollectWithThreshold(ctx context.Context, live liveset.Liveset, dead liveset.Liveset, threshold time.Time, dryRun bool) error

CollectWithThreshold removes from this repository any objects not in the Liveset and whose creation times are not more recent than the threshold time.

func (*ExpectRepository) Complete

func (r *ExpectRepository) Complete() error

Complete should be called when testing is finished; it verifies that the entire script has been exhausted.

func (*ExpectRepository) Expect

func (r *ExpectRepository) Expect(call RepositoryCall)

Expect adds a call to the repository's script.

func (*ExpectRepository) Get

Get implements the repository's Get call.

func (*ExpectRepository) Put

Put implements the repository's Put call.

func (*ExpectRepository) ReadFrom

func (r *ExpectRepository) ReadFrom(_ context.Context, id digest.Digest, u *url.URL) error

ReadFrom implements the repository's ReadFrom call.

func (*ExpectRepository) Stat

Stat always returns a NotExist error.

func (*ExpectRepository) URL

func (r *ExpectRepository) URL() *url.URL

URL returns the repository's URL.

func (*ExpectRepository) WriteTo

func (r *ExpectRepository) WriteTo(_ context.Context, id digest.Digest, u *url.URL) error

WriteTo implements the repository's WriteTo call.

type FileWithContents

type FileWithContents struct {
	reflow.File
	// contains filtered or unexported fields
}

type Fuzz

type Fuzz struct{ *rand.Rand }

Fuzz provides a simple deterministic fuzzer for Reflow data types.

func NewFuzz

func NewFuzz(r *rand.Rand) *Fuzz

NewFuzz returns a new fuzzer based on the provided random number generator. If r is nil, NewFuzz creates one with a fixed seed.

func (*Fuzz) Digest

func (f *Fuzz) Digest() digest.Digest

Digest returns a random Reflow digest.

func (*Fuzz) File

func (f *Fuzz) File(refok, aok bool) reflow.File

File returns a random file. If refok is true, then the returned file may be a reference file. If aok is true, then the returned file will contain assertions.

func (*Fuzz) Fileset

func (f *Fuzz) Fileset(refok, aok bool) reflow.Fileset

Fileset returns a random fileset. If refok is true, then the returned fileset may contain reference files. If aok is true, then the returned fileset will contain assertions.

func (*Fuzz) FilesetDeep

func (f *Fuzz) FilesetDeep(n, maxdepth int, refok, aok bool) reflow.Fileset

FilesetDeep returns a random fileset of the given depth and number of files.

func (*Fuzz) String

func (f *Fuzz) String(sep string) string

String returns a random string comprising gene names separated by the provided separator.

func (*Fuzz) StringMinLen

func (f *Fuzz) StringMinLen(minlen int, sep string) string

StringMinLen returns a random string comprising gene names of at least minlen length separated by the provided separator.

type InmemoryAssoc

type InmemoryAssoc struct {
	// contains filtered or unexported fields
}

func NewInmemoryAssoc

func NewInmemoryAssoc() *InmemoryAssoc

NewInmemoryAssoc returns a new assoc.Assoc that stores its mapping in memory.

func (*InmemoryAssoc) BatchGet

func (a *InmemoryAssoc) BatchGet(ctx context.Context, batch assoc.Batch) error

func (*InmemoryAssoc) CollectWithThreshold

CollectWithThreshold removes from this assoc any objects whose keys are not in the liveset and which have not been accessed more recently than the liveset's threshold time.

func (*InmemoryAssoc) Copy

func (a *InmemoryAssoc) Copy() *InmemoryAssoc

func (*InmemoryAssoc) Count

func (a *InmemoryAssoc) Count(ctx context.Context) (int64, error)

Count returns an estimate of the number of associations in this mapping.

func (*InmemoryAssoc) Delete

func (a *InmemoryAssoc) Delete(ctx context.Context, k digest.Digest) error

Delete deletes the key k unconditionally from the provided assoc.

func (*InmemoryAssoc) Get

func (*InmemoryAssoc) RawAssocs

func (a *InmemoryAssoc) RawAssocs() map[assocKey]digest.Digest

func (*InmemoryAssoc) Scan

func (a *InmemoryAssoc) Scan(ctx context.Context, kinds []assoc.Kind, handler assoc.MappingHandler) error

Scan calls the handler function for every association in the mapping. Note that the handler function may be called asynchronously from multiple threads.

func (*InmemoryAssoc) SetScanLabelsGenerator

func (a *InmemoryAssoc) SetScanLabelsGenerator(gen func() []string)

func (*InmemoryAssoc) SetScanTimeGenerator

func (a *InmemoryAssoc) SetScanTimeGenerator(gen func() time.Time)

func (*InmemoryAssoc) Store

func (a *InmemoryAssoc) Store(ctx context.Context, kind assoc.Kind, k, v digest.Digest) error

type InmemoryRepository

type InmemoryRepository struct {
	// contains filtered or unexported fields
}

InmemoryRepository is an in-memory repository used for testing.

func GetInMemoryRepository

func GetInMemoryRepository(repo *url.URL) *InmemoryRepository

func NewInmemoryRepository

func NewInmemoryRepository(name string) *InmemoryRepository

NewInmemoryRepository returns a repository that stores objects in memory. NewInmemoryRepository, if called with a non-empty name, will create a named repository, and other callers which use the same name will get the same underlying instance. So independent tests which do not want to share the repository should use unique names (or no name at all, in which case, a random instance is returned each time).

func (*InmemoryRepository) Collect

func (r *InmemoryRepository) Collect(_ context.Context, live liveset.Liveset) error

Collect removes any object not in the liveset.

func (*InmemoryRepository) CollectWithThreshold

func (r *InmemoryRepository) CollectWithThreshold(ctx context.Context, live liveset.Liveset, dead liveset.Liveset, threshold time.Time, dryRun bool) error

CollectWithThreshold removes from this repository any objects not in the Liveset and whose creation times are not more recent than the threshold time.

func (*InmemoryRepository) Copy

func (*InmemoryRepository) Delete

func (r *InmemoryRepository) Delete(_ context.Context, id digest.Digest)

Delete removes the key id from this repository.

func (*InmemoryRepository) Get

Get returns the blob named by id.

func (*InmemoryRepository) Put

Put installs the blob rd and returns its digest.

func (*InmemoryRepository) RawFiles

func (r *InmemoryRepository) RawFiles() map[digest.Digest][]byte

func (*InmemoryRepository) ReadFrom

func (r *InmemoryRepository) ReadFrom(_ context.Context, id digest.Digest, u *url.URL) error

ReadFrom is not supported.

func (*InmemoryRepository) SetLocation

func (r *InmemoryRepository) SetLocation(_ context.Context, id digest.Digest, loc string) error

SetLocation returns metadata for the blob named by id.

func (*InmemoryRepository) Stat

Stat returns metadata for the blob named by id.

func (*InmemoryRepository) URL

func (r *InmemoryRepository) URL() *url.URL

URL returns a nil URL.

func (*InmemoryRepository) Vacuum

func (r *InmemoryRepository) Vacuum(ctx context.Context, repo *InmemoryRepository)

Vacuum moves all objects from the given repository to this one.

func (*InmemoryRepository) WriteTo

func (r *InmemoryRepository) WriteTo(_ context.Context, id digest.Digest, u *url.URL) error

WriteTo is not supported.

type RepositoryCall

type RepositoryCall struct {
	Kind         RepositoryCallKind
	ArgID        digest.Digest
	ArgURL       url.URL
	ArgBytes     []byte
	ArgReadError error

	ReplyID         digest.Digest
	ReplyFile       reflow.File
	ReplyErr        error
	ReplyReadCloser io.ReadCloser
	ReplyN          int64
}

RepositoryCall describes a single call to a Repository: its expected arguments, and a reply. Repository calls are used with an ExpectRepository.

type RepositoryCallKind

type RepositoryCallKind int

RepositoryCallKind indicates the type of repository call.

const (
	RepositoryGet RepositoryCallKind = iota
	RepositoryPut
	RepositoryStat
	RepositoryWriteTo
	RepositoryReadFrom
	RepositoryGetFile
)

The concrete types of repository calls.

func (RepositoryCallKind) String

func (i RepositoryCallKind) String() string

type WaitCache

type WaitCache struct {
	// contains filtered or unexported fields
}

WaitCache implements a reflow.Cache used for testing. WaitCaches rendezvous callers, acting as a concurrency control mechanism for tests.

func (*WaitCache) Delete

func (c *WaitCache) Delete(ctx context.Context, id digest.Digest) error

Delete is not implemented in WaitCache, and will panic.

func (*WaitCache) Hit

func (c *WaitCache) Hit(f *flow.Flow, v reflow.Fileset)

Hit sets the value of flow f to v. Hit returns when the value has been consumed by the code under test.

func (*WaitCache) Init

func (c *WaitCache) Init()

Init (re-) initializes a WaitCache.

func (*WaitCache) Lookup

func (c *WaitCache) Lookup(ctx context.Context, id digest.Digest) (reflow.Fileset, error)

Lookup implements cache lookups. Lookup returns when the value for id has been set (through Hit or Miss) or when the context is done.

func (*WaitCache) Miss

func (c *WaitCache) Miss(f *flow.Flow)

Miss sets the value of flow f to a cache miss. Miss returns when it has been consumed by the code under test.

func (*WaitCache) NeedTransfer

func (c *WaitCache) NeedTransfer(ctx context.Context, dst reflow.Repository, v reflow.Fileset) ([]reflow.File, error)

NeedTransfer returns the file objects in v that are missing from repository dst.

func (*WaitCache) Repository

func (*WaitCache) Repository() reflow.Repository

Repository is not implemented.

func (*WaitCache) Transfer

func (c *WaitCache) Transfer(ctx context.Context, dst reflow.Repository, v reflow.Fileset) error

Transfer always returns (immediate) success.

func (*WaitCache) Write

Write always returns (immediate) success.

type WaitRepository

type WaitRepository struct {
	// contains filtered or unexported fields
}

WaitRepository is a repository implementation for testing that lets the caller rendeszvous calls so that concurrency can be controlled.

func NewWaitRepository

func NewWaitRepository(rawurl string) *WaitRepository

NewWaitRepository creates a new WaitRespository with the provided URL. If the URL is invalid, NewWaitRepository will panic.

func (*WaitRepository) Call

func (w *WaitRepository) Call(kind RepositoryCallKind, digest digest.Digest) chan RepositoryCall

Call rendeszvous with another caller. The returned channel is used to reply to the call.

func (*WaitRepository) Collect

Collect is not supported by WaitRepository.

func (*WaitRepository) CollectWithThreshold

func (r *WaitRepository) CollectWithThreshold(ctx context.Context, live liveset.Liveset, dead liveset.Liveset, threshold time.Time, dryRun bool) error

CollectWithThreshold removes from this repository any objects not in the Liveset and whose creation times are not more recent than the threshold time.

func (*WaitRepository) Get

Get waits for another caller to rendeszvous and then returns the caller's reply.

func (*WaitRepository) Put

Put reads the contents of r, waits for another caller to rendeszvous and then returns the caller's reply.

func (*WaitRepository) ReadFrom

func (w *WaitRepository) ReadFrom(_ context.Context, id digest.Digest, u *url.URL) error

ReadFrom is not supported by WaitRepository.

func (*WaitRepository) Stat

Stat waits for another caller to rendeszvous and then returns the caller's reply.

func (*WaitRepository) URL

func (w *WaitRepository) URL() *url.URL

URL returns the repository's URL.

func (*WaitRepository) WriteTo

func (w *WaitRepository) WriteTo(_ context.Context, id digest.Digest, u *url.URL) error

WriteTo is not supported by WaitRepository.

type WaitTransferer

type WaitTransferer struct {
	// contains filtered or unexported fields
}

WaitTransferer is a transfer manager for testing. It lets the test code rendezvous with the caller. In this way, Transferer provides both a "mock" transfer manager implementation and also permits concurrency control for the tester.

func (*WaitTransferer) Err

func (t *WaitTransferer) Err(err error, dst, src reflow.Repository, files ...reflow.File)

Err rendezvous the call named by the destination repository, source repository and fileset with failure.

func (*WaitTransferer) Init

func (t *WaitTransferer) Init()

Init initializes a WaitTransferer.

func (*WaitTransferer) NeedTransfer

func (t *WaitTransferer) NeedTransfer(ctx context.Context, dst reflow.Repository, files ...reflow.File) ([]reflow.File, error)

NeedTransfer returns all the missing objects in the destination repository. The test transferer assumes they are all missing.

func (*WaitTransferer) Ok

func (t *WaitTransferer) Ok(dst, src reflow.Repository, files ...reflow.File)

Ok rendezvous the call named by the destination repository, source repository and fileset with succcess.

func (*WaitTransferer) Transfer

func (t *WaitTransferer) Transfer(ctx context.Context, dst, src reflow.Repository, files ...reflow.File) error

Transfer waits for the tester to rendezvous, returning its result.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL