testplanet

package
v1.68.0-rc Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 23, 2022 License: AGPL-3.0 Imports: 106 Imported by: 6

Documentation

Overview

Package testplanet implements full network wiring for testing.

testplanet provides access to most of the internals of satellites, storagenodes and uplinks.

Database

It does require setting two variables for the databases: 

STORJ_TEST_POSTGRES=postgres://storj:storj-pass@test-postgres/teststorj?sslmode=disable
STORJ_TEST_COCKROACH=cockroach://root@localhost:26257/master?sslmode=disable

When you wish to entirely omit either of them from the test output, it's possible to use: 

STORJ_TEST_POSTGRES=omit
STORJ_TEST_COCKROACH=omit

Host

It's possible to change the listing host with: 

STORJ_TEST_HOST=127.0.0.2;127.0.0.3

Debugging

For debugging, it's possible to set STORJ_TEST_MONKIT to get a trace per test. 

STORJ_TEST_MONKIT=svg
STORJ_TEST_MONKIT=json

By default, it saves the output the same folder as the test. However, if you wish to specify a separate folder, you can specify an absolute directory: 

STORJ_TEST_MONKIT=svg,dir=/home/user/debug/trace

Note, due to how go tests work, it's not possible to specify a relative directory.

Index

Constants

This section is empty.

Variables

View Source 
var Combine = func(elements ...func(log *zap.Logger, index int, config *satellite.Config)) func(log *zap.Logger, index int, config *satellite.Config) {
	return func(log *zap.Logger, index int, config *satellite.Config) {
		for _, f := range elements {
			f(log, index, config)
		}
	}
}

Combine combines satellite reconfigure functions.

View Source 
var DisablePeerCAWhitelist = Reconfigure{
	Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
		config.Server.UsePeerCAWhitelist = false
	},
	StorageNode: func(index int, config *storagenode.Config) {
		config.Server.UsePeerCAWhitelist = false
	},
}

DisablePeerCAWhitelist returns a `Reconfigure` that sets `UsePeerCAWhitelist` for all node types that use kademlia.

View Source 
var DisableQUIC = Reconfigure{
	Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
		config.Server.DisableQUIC = true
	},
	StorageNode: func(index int, config *storagenode.Config) {
		config.Server.DisableQUIC = true
	},
}

DisableQUIC prevents both satellite and storagenode being able to accept new quic connections.

View Source 
var DisableTCP = Reconfigure{
	Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
		config.Server.DisableTCPTLS = true
	},
	StorageNode: func(index int, config *storagenode.Config) {
		config.Server.DisableTCPTLS = true
	},
}

DisableTCP prevents both satellite and storagenode being able to accept new tcp connections.

View Source 
var MaxMetadataSize = func(maxMetadataSize memory.Size) func(log *zap.Logger, index int, config *satellite.Config) {
	return func(log *zap.Logger, index int, config *satellite.Config) {
		config.Metainfo.MaxMetadataSize = maxMetadataSize
	}
}

MaxMetadataSize returns function to change satellite max metadata size value.

View Source 
var MaxObjectKeyLength = func(maxObjectKeyLength int) func(log *zap.Logger, index int, config *satellite.Config) {
	return func(log *zap.Logger, index int, config *satellite.Config) {
		config.Metainfo.MaxEncryptedObjectKeyLength = maxObjectKeyLength
	}
}

MaxObjectKeyLength returns function to change satellite max object key length value.

View Source 
var MaxSegmentSize = func(maxSegmentSize memory.Size) func(log *zap.Logger, index int, config *satellite.Config) {
	return func(log *zap.Logger, index int, config *satellite.Config) {
		config.Metainfo.MaxSegmentSize = maxSegmentSize
	}
}

MaxSegmentSize returns function to change satellite max segment size value.

View Source 
var ReconfigureRS = func(minThreshold, repairThreshold, successThreshold, totalThreshold int) func(log *zap.Logger, index int, config *satellite.Config) {
	return func(log *zap.Logger, index int, config *satellite.Config) {
		config.Metainfo.RS.Min = minThreshold
		config.Metainfo.RS.Repair = repairThreshold
		config.Metainfo.RS.Success = successThreshold
		config.Metainfo.RS.Total = totalThreshold
	}
}

ReconfigureRS returns function to change satellite redundancy scheme values.

View Source 
var RepairExcludedCountryCodes = func(repairExcludedCountryCodes []string) func(log *zap.Logger, index int, config *satellite.Config) {
	return func(log *zap.Logger, index int, config *satellite.Config) {
		config.Overlay.RepairExcludedCountryCodes = repairExcludedCountryCodes
	}
}

RepairExcludedCountryCodes returns function to change satellite repair excluded country codes.

View Source 
var ShortenOnlineWindow = Reconfigure{
	Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
		config.Overlay.Node.OnlineWindow = 1 * time.Second
	},
}

ShortenOnlineWindow returns a `Reconfigure` that sets the NodeSelection OnlineWindow to 1 second, meaning a connection failure leads to marking the nodes as offline.

View Source 
var UploadExcludedCountryCodes = func(uploadExcludedCountryCodes []string) func(log *zap.Logger, index int, config *satellite.Config) {
	return func(log *zap.Logger, index int, config *satellite.Config) {
		config.Overlay.Node.UploadExcludedCountryCodes = uploadExcludedCountryCodes
	}
}

UploadExcludedCountryCodes returns function to change satellite upload excluded country codes.

Functions

func Bench added in v1.65.1 

func Bench(b *testing.B, config Config, bench func(b *testing.B, ctx *testcontext.Context, planet *Planet))

Bench makes benchmark with testplanet as easy as running unit tests with Run method.

func NewLogger added in v1.48.1 

func NewLogger(t *testing.T) *zap.Logger

NewLogger creates a zaptest logger with nice defaults for tests.

func Run 

func Run(t *testing.T, config Config, test func(t *testing.T, ctx *testcontext.Context, planet *Planet))

Run runs testplanet in multiple configurations.

Types

type Config 

type Config struct {
	SatelliteCount   int
	StorageNodeCount int
	UplinkCount      int
	MultinodeCount   int

	IdentityVersion *storj.IDVersion
	Reconfigure     Reconfigure

	Name        string
	Host        string
	NonParallel bool
	Timeout     time.Duration
}

Config describes planet configuration.

type DatabaseConfig added in v1.1.1 

type DatabaseConfig struct {
	SatelliteDB string
}

DatabaseConfig defines connection strings for database.

type Multinode added in v1.41.1 

type Multinode struct {
	Name   string
	Config multinode.Config
	*multinode.Peer
}

Multinode contains all the processes needed to run a full multinode setup.

func (*Multinode) Addr added in v1.41.1 

func (system *Multinode) Addr() string

Addr returns the public address.

func (*Multinode) ConsoleURL added in v1.42.2 

func (system *Multinode) ConsoleURL() string

ConsoleURL returns the console URL.

func (*Multinode) ID added in v1.41.1 

func (system *Multinode) ID() storj.NodeID

ID returns multinode id.

func (*Multinode) Label added in v1.41.1 

func (system *Multinode) Label() string

Label returns name for debugger.

func (*Multinode) NodeURL added in v1.41.1 

func (system *Multinode) NodeURL() storj.NodeURL

NodeURL returns the storj.NodeURL.

func (*Multinode) URL added in v1.41.1 

func (system *Multinode) URL() string

URL returns the NodeURL as a string.

type Peer 

type Peer interface {
	Label() string

	ID() storj.NodeID
	Addr() string
	URL() string
	NodeURL() storj.NodeURL

	Run(context.Context) error
	Close() error
}

Peer represents one of StorageNode or Satellite.

type Planet 

type Planet struct {
	VersionControl *versioncontrol.Peer
	Satellites     []*Satellite
	StorageNodes   []*StorageNode
	Multinodes     []*Multinode
	Uplinks        []*Uplink
	// contains filtered or unexported fields
}

Planet is a full storj system setup.

func NewCustom 

func NewCustom(ctx *testcontext.Context, log *zap.Logger, config Config, satelliteDatabases satellitedbtest.SatelliteDatabases) (*Planet, error)

NewCustom creates a new full system with the specified configuration.

func (*Planet) FindNode added in v0.35.2 

func (planet *Planet) FindNode(nodeID storj.NodeID) *StorageNode

FindNode is a helper to retrieve a storage node record by its node ID.

func (*Planet) Identities 

func (planet *Planet) Identities() *testidentity.Identities

Identities returns the identity provider for this planet.

func (*Planet) Log added in v1.44.1 

func (planet *Planet) Log() *zap.Logger

Log returns the root logger.

func (*Planet) NewIdentity 

func (planet *Planet) NewIdentity() (*identity.FullIdentity, error)

NewIdentity creates a new identity for a node.

func (*Planet) NewListenAddress added in v1.52.1 

func (planet *Planet) NewListenAddress() string

NewListenAddress returns an address for listening.

func (*Planet) NewListener 

func (planet *Planet) NewListener() (net.Listener, error)

NewListener creates a new listener.

func (*Planet) NewVersionConfig 

func (planet *Planet) NewVersionConfig() checker.Config

NewVersionConfig returns the Version Config for this planet with tuned metrics.

func (*Planet) NewVersionInfo 

func (planet *Planet) NewVersionInfo() version.Info

NewVersionInfo returns the Version Info for this planet with tuned metrics.

func (*Planet) Shutdown 

func (planet *Planet) Shutdown() error

Shutdown shuts down all the nodes and deletes temporary directories.

func (*Planet) Size 

func (planet *Planet) Size() int

Size returns number of nodes in the network.

func (*Planet) Start 

func (planet *Planet) Start(ctx context.Context)

Start starts all the nodes.

func (*Planet) StopNodeAndUpdate added in v1.5.2 

func (planet *Planet) StopNodeAndUpdate(ctx context.Context, node *StorageNode) (err error)

StopNodeAndUpdate stops storage node and updates satellite overlay.

func (*Planet) StopPeer 

func (planet *Planet) StopPeer(peer Peer) error

StopPeer stops a single peer in the planet.

func (*Planet) WaitForStorageNodeDeleters added in v1.4.1 

func (planet *Planet) WaitForStorageNodeDeleters(ctx context.Context)

WaitForStorageNodeDeleters calls the Wait method on each storagenode's PieceDeleter.

func (*Planet) WaitForStorageNodeEndpoints added in v0.35.2 

func (planet *Planet) WaitForStorageNodeEndpoints(ctx context.Context) (err error)

WaitForStorageNodeEndpoints waits for storage node endpoints to finish their work. The call will return an error if they have not been completed after 1 minute.

func (*Planet) WriteWhitelist 

func (planet *Planet) WriteWhitelist(version storj.IDVersion) (string, error)

WriteWhitelist writes the pregenerated signer's CA cert to a "CA whitelist", PEM-encoded.

type Project added in v1.3.2 

type Project struct {
	ID    uuid.UUID
	Owner ProjectOwner

	Satellite Peer
	APIKey    string

	RawAPIKey *macaroon.APIKey
	// contains filtered or unexported fields
}

Project contains all necessary information about a user.

func (*Project) DialMetainfo added in v1.3.2 

func (project *Project) DialMetainfo(ctx context.Context) (_ *metaclient.Client, err error)

DialMetainfo dials the satellite with the appropriate api key.

type ProjectOwner added in v1.3.2 

type ProjectOwner struct {
	ID    uuid.UUID
	Email string
}

ProjectOwner contains information about the project owner.

type Reconfigure 

type Reconfigure struct {
	SatelliteDB         func(log *zap.Logger, index int, db satellite.DB) (satellite.DB, error)
	SatelliteMetabaseDB func(log *zap.Logger, index int, db *metabase.DB) (*metabase.DB, error)
	Satellite           func(log *zap.Logger, index int, config *satellite.Config)

	StorageNodeDB func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error)
	StorageNode   func(index int, config *storagenode.Config)
	UniqueIPCount int

	VersionControl func(config *versioncontrol.Config)

	Identities func(log *zap.Logger, version storj.IDVersion) *testidentity.Identities

	MultinodeDB func(index int, db multinode.DB, log *zap.Logger) (multinode.DB, error)
	Multinode   func(index int, config *multinode.Config)
}

Reconfigure allows to change node configurations.

type Satellite added in v1.1.1 

type Satellite struct {
	Name   string
	Config satellite.Config

	Core     *satellite.Core
	API      *satellite.API
	Repairer *satellite.Repairer
	Admin    *satellite.Admin
	GC       *satellite.GarbageCollection
	GCBF     *satellite.GarbageCollectionBF

	Log      *zap.Logger
	Identity *identity.FullIdentity
	DB       satellite.DB

	Dialer rpc.Dialer

	Server *server.Server

	Version *versionchecker.Service

	Contact struct {
		Service  *contact.Service
		Endpoint *contact.Endpoint
	}

	Overlay struct {
		DB                overlay.DB
		Service           *overlay.Service
		OfflineNodeEmails *offlinenodes.Chore
		DQStrayNodes      *straynodes.Chore
	}

	NodeEvents struct {
		DB       nodeevents.DB
		Notifier nodeevents.Notifier
		Chore    *nodeevents.Chore
	}

	Metainfo struct {
		// TODO remove when uplink will be adjusted to use Metabase.DB
		Metabase *metabase.DB
		Endpoint *metainfo.Endpoint
		// TODO remove when uplink will be adjusted to use Metabase.SegmentLoop
		SegmentLoop *segmentloop.Service
	}

	Metabase struct {
		DB          *metabase.DB
		SegmentLoop *segmentloop.Service
	}

	Inspector struct {
		Endpoint *inspector.Endpoint
	}

	Orders struct {
		DB       orders.DB
		Endpoint *orders.Endpoint
		Service  *orders.Service
		Chore    *orders.Chore
	}

	Repair struct {
		Checker  *checker.Checker
		Repairer *repairer.Service
	}

	Audit struct {
		VerifyQueue audit.VerifyQueue
		Worker      *audit.Worker
		Chore       *audit.Chore
		Verifier    *audit.Verifier
		Reporter    audit.Reporter
	}

	Reputation struct {
		Service *reputation.Service
	}

	GarbageCollection struct {
		Sender       *sender.Service
		BloomFilters *bloomfilter.Service
	}

	ExpiredDeletion struct {
		Chore *expireddeletion.Chore
	}

	ZombieDeletion struct {
		Chore *zombiedeletion.Chore
	}

	Accounting struct {
		Tally            *tally.Service
		NodeTally        *nodetally.Service
		Rollup           *rollup.Service
		ProjectUsage     *accounting.Service
		ProjectBWCleanup *projectbwcleanup.Chore
		RollupArchive    *rolluparchive.Chore
	}

	LiveAccounting struct {
		Cache accounting.Cache
	}

	ProjectLimits struct {
		Cache *accounting.ProjectLimitCache
	}

	Mail struct {
		Service *mailservice.Service
	}

	Console struct {
		Listener net.Listener
		Service  *console.Service
		Endpoint *consoleweb.Server
	}

	NodeStats struct {
		Endpoint *nodestats.Endpoint
	}

	GracefulExit struct {
		Chore    *gracefulexit.Chore
		Endpoint *gracefulexit.Endpoint
	}

	Metrics struct {
		Chore *metrics.Chore
	}
}

Satellite contains all the processes needed to run a full Satellite setup.

func (*Satellite) AddProject added in v1.6.1 

func (system *Satellite) AddProject(ctx context.Context, ownerID uuid.UUID, name string) (_ *console.Project, err error)

AddProject adds project to a satellite and makes specified user an owner.

func (*Satellite) AddUser added in v1.6.1 

func (system *Satellite) AddUser(ctx context.Context, newUser console.CreateUser, maxNumberOfProjects int) (_ *console.User, err error)

AddUser adds user to a satellite. Password from newUser will be always overridden by FullName to have known password which can be used automatically.

func (*Satellite) Addr added in v1.1.1 

func (system *Satellite) Addr() string

Addr returns the public address from the Satellite system API.

func (*Satellite) Close added in v1.1.1 

func (system *Satellite) Close() error

Close closes all the subsystems in the Satellite system.

func (*Satellite) ConsoleURL added in v1.32.1 

func (system *Satellite) ConsoleURL() string

ConsoleURL returns the console URL.

func (*Satellite) ID added in v1.1.1 

func (system *Satellite) ID() storj.NodeID

ID returns the ID of the Satellite system.

func (*Satellite) Label added in v1.17.1 

func (system *Satellite) Label() string

Label returns name for debugger.

func (*Satellite) NodeURL added in v1.3.2 

func (system *Satellite) NodeURL() storj.NodeURL

NodeURL returns the storj.NodeURL from the Satellite system API.

func (*Satellite) PrivateAddr added in v1.1.1 

func (system *Satellite) PrivateAddr() string

PrivateAddr returns the private address from the Satellite system API.

func (*Satellite) Run added in v1.1.1 

func (system *Satellite) Run(ctx context.Context) (err error)

Run runs all the subsystems in the Satellite system.

func (*Satellite) URL added in v1.1.1 

func (system *Satellite) URL() string

URL returns the node url from the Satellite system API.

func (*Satellite) UserContext added in v1.58.1 

func (system *Satellite) UserContext(ctx context.Context, userID uuid.UUID) (_ context.Context, err error)

UserContext creates context with user.

type StorageNode added in v1.1.1 

type StorageNode struct {
	Name   string
	Config storagenode.Config
	*storagenode.Peer
	// contains filtered or unexported fields
}

StorageNode contains all the processes needed to run a full StorageNode setup.

func (*StorageNode) APIKey added in v1.42.2 

func (system *StorageNode) APIKey() string

APIKey returns the API key of the node.

func (*StorageNode) Label added in v1.17.1 

func (system *StorageNode) Label() string

Label returns name for debugger.

func (*StorageNode) NodeURL added in v1.3.2 

func (system *StorageNode) NodeURL() storj.NodeURL

NodeURL returns the storj.NodeURL.

func (*StorageNode) URL added in v1.1.1 

func (system *StorageNode) URL() string

URL returns the node url as a string. 

type Uplink struct {
	Log      *zap.Logger
	Identity *identity.FullIdentity
	Dialer   rpc.Dialer
	Config   uplink.Config

	APIKey map[storj.NodeID]*macaroon.APIKey
	Access map[storj.NodeID]*uplink.Access
	User   map[storj.NodeID]UserLogin

	// Projects is indexed by the satellite number.
	Projects []*Project
}

Uplink is a registered user on all satellites, which contains the necessary accesses and project info.

func (*Uplink) Addr 

func (client *Uplink) Addr() string

Addr returns uplink address.

func (*Uplink) CopyObject added in v1.61.1 

func (client *Uplink) CopyObject(ctx context.Context, satellite *Satellite, oldBucket, oldKey, newBucket, newKey string) (err error)

CopyObject copies an object.

func (*Uplink) CreateBucket 

func (client *Uplink) CreateBucket(ctx context.Context, satellite *Satellite, bucketName string) (err error)

CreateBucket creates a new bucket.

func (*Uplink) DeleteBucket added in v0.33.2 

func (client *Uplink) DeleteBucket(ctx context.Context, satellite *Satellite, bucketName string) (err error)

DeleteBucket deletes a bucket.

func (*Uplink) DeleteObject added in v0.33.2 

func (client *Uplink) DeleteObject(ctx context.Context, satellite *Satellite, bucketName string, path storj.Path) (err error)

DeleteObject deletes an object at the path in a bucket.

func (*Uplink) DialMetainfo 

func (client *Uplink) DialMetainfo(ctx context.Context, destination Peer, apikey *macaroon.APIKey) (_ *metaclient.Client, err error)

DialMetainfo dials destination with apikey and returns metainfo Client.

func (*Uplink) DialPiecestore 

func (client *Uplink) DialPiecestore(ctx context.Context, destination Peer) (_ *piecestore.Client, err error)

DialPiecestore dials destination storagenode and returns a piecestore client.

func (*Uplink) Download 

func (client *Uplink) Download(ctx context.Context, satellite *Satellite, bucketName string, path storj.Path) (_ []byte, err error)

Download data from specific satellite.

func (*Uplink) DownloadStream 

func (client *Uplink) DownloadStream(ctx context.Context, satellite *Satellite, bucketName string, path storj.Path) (_ io.ReadCloser, cleanup func() error, err error)

DownloadStream returns stream for downloading data.

func (*Uplink) DownloadStreamRange 

func (client *Uplink) DownloadStreamRange(ctx context.Context, satellite *Satellite, bucketName string, path storj.Path, start, limit int64) (_ io.ReadCloser, cleanup func() error, err error)

DownloadStreamRange returns stream for downloading data.

func (*Uplink) GetProject 

func (client *Uplink) GetProject(ctx context.Context, satellite *Satellite) (_ *uplink.Project, err error)

GetProject returns a uplink.Project which allows interactions with a specific project.

func (*Uplink) ID 

func (client *Uplink) ID() storj.NodeID

ID returns uplink id.

func (*Uplink) ListBuckets added in v1.21.1 

func (client *Uplink) ListBuckets(ctx context.Context, satellite *Satellite) (_ []*uplink.Bucket, err error)

ListBuckets returns a list of all buckets in a project.

func (*Uplink) ListObjects added in v1.31.1 

func (client *Uplink) ListObjects(ctx context.Context, satellite *Satellite, bucketName string) (_ []*uplink.Object, err error)

ListObjects returns a list of all objects in a bucket.

func (*Uplink) OpenProject added in v1.19.1 

func (client *Uplink) OpenProject(ctx context.Context, satellite *Satellite) (_ *uplink.Project, err error)

OpenProject opens project with predefined access grant and gives access to pure uplink API.

func (*Uplink) Shutdown 

func (client *Uplink) Shutdown() error

Shutdown shuts down all uplink dependencies.

func (*Uplink) Upload 

func (client *Uplink) Upload(ctx context.Context, satellite *Satellite, bucket string, path storj.Path, data []byte) (err error)

Upload data to specific satellite.

func (*Uplink) UploadWithExpiration 

func (client *Uplink) UploadWithExpiration(ctx context.Context, satellite *Satellite, bucketName string, path storj.Path, data []byte, expiration time.Time) (err error)

UploadWithExpiration data to specific satellite and expiration time.

type UserLogin added in v1.32.1 

type UserLogin struct {
	Email    string
	Password string
}

UserLogin contains information about the user login.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.