testplanet

package
v1.5.1-rc Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 19, 2020 License: AGPL-3.0 Imports: 96 Imported by: 6

Documentation

Overview

Package testplanet implements the full network wiring for testing

Index

Constants

This section is empty.

Variables

View Source 
var DisablePeerCAWhitelist = Reconfigure{
	Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
		config.Server.UsePeerCAWhitelist = false
	},
	StorageNode: func(index int, config *storagenode.Config) {
		config.Server.UsePeerCAWhitelist = false
	},
}

DisablePeerCAWhitelist returns a `Reconfigure` that sets `UsePeerCAWhitelist` for all node types that use kademlia.

View Source 
var ReconfigureRS = func(minThreshold, repairThreshold, successThreshold, totalThreshold int) func(log *zap.Logger, index int, config *satellite.Config) {
	return func(log *zap.Logger, index int, config *satellite.Config) {
		config.Metainfo.RS.MinThreshold = minThreshold
		config.Metainfo.RS.RepairThreshold = repairThreshold
		config.Metainfo.RS.SuccessThreshold = successThreshold
		config.Metainfo.RS.TotalThreshold = totalThreshold
	}
}

ReconfigureRS returns function to change satellite redundancy scheme values

View Source 
var ShortenOnlineWindow = Reconfigure{
	Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
		config.Overlay.Node.OnlineWindow = 1 * time.Second
	},
}

ShortenOnlineWindow returns a `Reconfigure` that sets the NodeSelection OnlineWindow to 1 second, meaning a connection failure leads to marking the nodes as offline

Functions

func Run 

func Run(t *testing.T, config Config, test func(t *testing.T, ctx *testcontext.Context, planet *Planet))

Run runs testplanet in multiple configurations.

Types

type AccessConfig added in v0.35.2 

type AccessConfig struct {
	Accesses map[string]string `internal:"true"`
	Access   string            `help:"the serialized access, or name of the access to use" default:"" basic-help:"true"`

	// used for backward compatibility
	Scopes map[string]string `internal:"true"` // deprecated
	Scope  string            `internal:"true"` // deprecated

	Legacy // Holds on to legacy configuration values
}

AccessConfig holds information about which accesses exist and are selected.

func (AccessConfig) GetAccess added in v0.35.2 

func (a AccessConfig) GetAccess() (_ *libuplink.Scope, err error)

GetAccess returns the appropriate access for the config.

func (AccessConfig) GetNamedAccess added in v0.35.2 

func (a AccessConfig) GetNamedAccess(name string) (_ *libuplink.Scope, err error)

GetNamedAccess returns named access if exists.

func (AccessConfig) GetNewAccess added in v0.35.2 

func (a AccessConfig) GetNewAccess() (_ *uplink.Access, err error)

GetNewAccess returns the appropriate access for the config.

type ClientConfig added in v0.35.2 

type ClientConfig struct {
	MaxInlineSize memory.Size   `help:"max inline segment size in bytes" default:"4KiB"`
	SegmentSize   memory.Size   `help:"the size of a segment in bytes" default:"64MiB"`
	DialTimeout   time.Duration `help:"timeout for dials" default:"0h2m00s"`
}

ClientConfig is a configuration struct for the uplink that controls how to talk to the rest of the network.

type Config 

type Config struct {
	SatelliteCount   int
	StorageNodeCount int
	UplinkCount      int

	IdentityVersion *storj.IDVersion
	Reconfigure     Reconfigure

	Name        string
	NonParallel bool
}

Config describes planet configuration

type DatabaseConfig added in v1.1.1 

type DatabaseConfig struct {
	SatelliteDB        string
	SatellitePointerDB string
}

DatabaseConfig defines connection strings for database.

type DefaultReferralManagerServer added in v0.27.0 

type DefaultReferralManagerServer struct {
	// contains filtered or unexported fields
}

DefaultReferralManagerServer implements the default behavior of a mock referral manager

func (*DefaultReferralManagerServer) GetTokens added in v0.27.0 

func (server *DefaultReferralManagerServer) GetTokens(ctx context.Context, req *pb.GetTokensRequest) (*pb.GetTokensResponse, error)

GetTokens implements a mock GetTokens endpoint that returns a number of referral tokens. By default, it returns 0 tokens.

func (*DefaultReferralManagerServer) RedeemToken added in v0.27.0 

func (server *DefaultReferralManagerServer) RedeemToken(ctx context.Context, req *pb.RedeemTokenRequest) (*pb.RedeemTokenResponse, error)

RedeemToken implements a mock RedeemToken endpoint.

func (*DefaultReferralManagerServer) SetTokenCount added in v0.27.0 

func (server *DefaultReferralManagerServer) SetTokenCount(tokenCount int)

SetTokenCount sets the number of tokens GetTokens endpoint should return.

type EncryptionConfig added in v0.35.2 

type EncryptionConfig struct {
	DataType int `help:"Type of encryption to use for content and metadata (2=AES-GCM, 3=SecretBox)" default:"2"`
	PathType int `help:"Type of encryption to use for paths (1=Unencrypted, 2=AES-GCM, 3=SecretBox)" default:"2"`
}

EncryptionConfig is a configuration struct that keeps details about encrypting segments

type Legacy added in v0.35.2 

type Legacy struct {
	Client struct {
		APIKey        string `default:"" help:"the api key to use for the satellite (deprecated)" noprefix:"true" deprecated:"true"`
		SatelliteAddr string `` /* 133-byte string literal not displayed */
	}
	Enc struct {
		EncryptionKey     string `help:"the root key for encrypting the data which will be stored in KeyFilePath (deprecated)" setup:"true" deprecated:"true"`
		KeyFilepath       string `help:"the path to the file which contains the root key for encrypting the data (deprecated)" deprecated:"true"`
		EncAccessFilepath string `help:"the path to a file containing a serialized encryption access (deprecated)" deprecated:"true"`
	}
}

Legacy holds deprecated configuration values

type Peer 

type Peer interface {
	ID() storj.NodeID
	Addr() string
	URL() string
	NodeURL() storj.NodeURL
	Local() overlay.NodeDossier

	Run(context.Context) error
	Close() error
}

Peer represents one of StorageNode or Satellite

type Planet 

type Planet struct {
	VersionControl *versioncontrol.Peer
	Satellites     []*Satellite
	StorageNodes   []*StorageNode
	Uplinks        []*Uplink

	ReferralManager *server.Server
	// contains filtered or unexported fields
}

Planet is a full storj system setup.

func NewCustom 

func NewCustom(log *zap.Logger, config Config, satelliteDatabases satellitedbtest.SatelliteDatabases) (*Planet, error)

NewCustom creates a new full system with the specified configuration.

func (*Planet) FindNode added in v0.35.2 

func (planet *Planet) FindNode(nodeID storj.NodeID) *StorageNode

FindNode is a helper to retrieve a storage node record by its node ID.

func (*Planet) Identities 

func (planet *Planet) Identities() *testidentity.Identities

Identities returns the identity provider for this planet.

func (*Planet) NewIdentity 

func (planet *Planet) NewIdentity() (*identity.FullIdentity, error)

NewIdentity creates a new identity for a node

func (*Planet) NewListener 

func (planet *Planet) NewListener() (net.Listener, error)

NewListener creates a new listener

func (*Planet) NewVersionConfig 

func (planet *Planet) NewVersionConfig() checker.Config

NewVersionConfig returns the Version Config for this planet with tuned metrics.

func (*Planet) NewVersionInfo 

func (planet *Planet) NewVersionInfo() version.Info

NewVersionInfo returns the Version Info for this planet with tuned metrics.

func (*Planet) Shutdown 

func (planet *Planet) Shutdown() error

Shutdown shuts down all the nodes and deletes temporary directories.

func (*Planet) Size 

func (planet *Planet) Size() int

Size returns number of nodes in the network

func (*Planet) Start 

func (planet *Planet) Start(ctx context.Context)

Start starts all the nodes.

func (*Planet) StopNodeAndUpdate added in v1.5.2 

func (planet *Planet) StopNodeAndUpdate(ctx context.Context, node *StorageNode) error

StopNodeAndUpdate stops storage node and updates satellite overlay.

func (*Planet) StopPeer 

func (planet *Planet) StopPeer(peer Peer) error

StopPeer stops a single peer in the planet

func (*Planet) WaitForStorageNodeDeleters added in v1.4.1 

func (planet *Planet) WaitForStorageNodeDeleters(ctx context.Context)

WaitForStorageNodeDeleters calls the Wait method on each storagenode's PieceDeleter.

func (*Planet) WaitForStorageNodeEndpoints added in v0.35.2 

func (planet *Planet) WaitForStorageNodeEndpoints(ctx context.Context) error

WaitForStorageNodeEndpoints waits for storage node endpoints to finish their work. The call will return an error if they have not been completed after 1 minute.

func (*Planet) WriteWhitelist 

func (planet *Planet) WriteWhitelist(version storj.IDVersion) (string, error)

WriteWhitelist writes the pregenerated signer's CA cert to a "CA whitelist", PEM-encoded.

type Project added in v1.3.2 

type Project struct {
	ID    uuid.UUID
	Owner ProjectOwner

	Satellite Peer
	APIKey    string

	RawAPIKey *macaroon.APIKey
	// contains filtered or unexported fields
}

Project contains all necessary information about a user.

func (*Project) DialMetainfo added in v1.3.2 

func (project *Project) DialMetainfo(ctx context.Context) (*metainfo.Client, error)

DialMetainfo dials the satellite with the appropriate api key.

type ProjectOwner added in v1.3.2 

type ProjectOwner struct {
	ID    uuid.UUID
	Email string
}

ProjectOwner contains information about the project owner.

type RSConfig added in v0.35.2 

type RSConfig struct {
	MaxBufferMem     memory.Size `help:"maximum buffer memory (in bytes) to be allocated for read buffers" default:"4MiB" hidden:"true"`
	ErasureShareSize memory.Size `help:"the size of each new erasure share in bytes" default:"256B" hidden:"true"`
	MinThreshold     int         `help:"the minimum pieces required to recover a segment. k." releaseDefault:"29" devDefault:"4" hidden:"true"`
	RepairThreshold  int         `help:"the minimum safe pieces before a repair is triggered. m." releaseDefault:"35" devDefault:"6" hidden:"true"`
	SuccessThreshold int         `help:"the desired total pieces for a segment. o." releaseDefault:"80" devDefault:"8" hidden:"true"`
	MaxThreshold     int         `help:"the largest amount of pieces to encode to. n." releaseDefault:"110" devDefault:"10" hidden:"true"`
}

RSConfig is a configuration struct that keeps details about default redundancy strategy information

type Reconfigure 

type Reconfigure struct {
	SatelliteDB        func(log *zap.Logger, index int, db satellite.DB) (satellite.DB, error)
	SatellitePointerDB func(log *zap.Logger, index int, db metainfo.PointerDB) (metainfo.PointerDB, error)
	Satellite          func(log *zap.Logger, index int, config *satellite.Config)

	ReferralManagerServer func(log *zap.Logger) pb.DRPCReferralManagerServer

	StorageNodeDB func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error)
	StorageNode   func(index int, config *storagenode.Config)
	UniqueIPCount int

	Identities func(log *zap.Logger, version storj.IDVersion) *testidentity.Identities
}

Reconfigure allows to change node configurations

type Satellite added in v1.1.1 

type Satellite struct {
	Config satellite.Config

	Core     *satellite.Core
	API      *satellite.API
	Repairer *satellite.Repairer
	Admin    *satellite.Admin
	GC       *satellite.GarbageCollection

	Log      *zap.Logger
	Identity *identity.FullIdentity
	DB       satellite.DB

	Dialer rpc.Dialer

	Server *server.Server

	Version *versionchecker.Service

	Contact struct {
		Service  *contact.Service
		Endpoint *contact.Endpoint
	}

	Overlay struct {
		DB        overlay.DB
		Service   *overlay.Service
		Inspector *overlay.Inspector
	}

	Metainfo struct {
		Database  metainfo.PointerDB
		Service   *metainfo.Service
		Endpoint2 *metainfo.Endpoint
		Loop      *metainfo.Loop
	}

	Inspector struct {
		Endpoint *inspector.Endpoint
	}

	Orders struct {
		DB       orders.DB
		Endpoint *orders.Endpoint
		Service  *orders.Service
		Chore    *orders.Chore
	}

	Repair struct {
		Checker   *checker.Checker
		Repairer  *repairer.Service
		Inspector *irreparable.Inspector
	}
	Audit struct {
		Queue    *audit.Queue
		Worker   *audit.Worker
		Chore    *audit.Chore
		Verifier *audit.Verifier
		Reporter *audit.Reporter
	}

	GarbageCollection struct {
		Service *gc.Service
	}

	ExpiredDeletion struct {
		Chore *expireddeletion.Chore
	}

	DBCleanup struct {
		Chore *dbcleanup.Chore
	}

	Accounting struct {
		Tally          *tally.Service
		Rollup         *rollup.Service
		ProjectUsage   *accounting.Service
		ReportedRollup *reportedrollup.Chore
	}

	LiveAccounting struct {
		Cache accounting.Cache
	}

	Mail struct {
		Service *mailservice.Service
	}

	Vouchers struct {
		Endpoint *vouchers.Endpoint
	}

	Console struct {
		Listener net.Listener
		Service  *console.Service
		Endpoint *consoleweb.Server
	}

	Marketing struct {
		Listener net.Listener
		Endpoint *marketingweb.Server
	}

	NodeStats struct {
		Endpoint *nodestats.Endpoint
	}

	GracefulExit struct {
		Chore    *gracefulexit.Chore
		Endpoint *gracefulexit.Endpoint
	}

	Metrics struct {
		Chore *metrics.Chore
	}

	DowntimeTracking struct {
		DetectionChore  *downtime.DetectionChore
		EstimationChore *downtime.EstimationChore
		Service         *downtime.Service
	}
}

Satellite contains all the processes needed to run a full Satellite setup

func (*Satellite) Addr added in v1.1.1 

func (system *Satellite) Addr() string

Addr returns the public address from the Satellite system API.

func (*Satellite) Close added in v1.1.1 

func (system *Satellite) Close() error

Close closes all the subsystems in the Satellite system

func (*Satellite) ID added in v1.1.1 

func (system *Satellite) ID() storj.NodeID

ID returns the ID of the Satellite system.

func (*Satellite) Local added in v1.1.1 

func (system *Satellite) Local() overlay.NodeDossier

Local returns the peer local node info from the Satellite system API.

func (*Satellite) NodeURL added in v1.3.2 

func (system *Satellite) NodeURL() storj.NodeURL

NodeURL returns the storj.NodeURL from the Satellite system API.

func (*Satellite) PrivateAddr added in v1.1.1 

func (system *Satellite) PrivateAddr() string

PrivateAddr returns the private address from the Satellite system API.

func (*Satellite) Run added in v1.1.1 

func (system *Satellite) Run(ctx context.Context) (err error)

Run runs all the subsystems in the Satellite system

func (*Satellite) URL added in v1.1.1 

func (system *Satellite) URL() string

URL returns the node url from the Satellite system API.

type StorageNode added in v1.1.1 

type StorageNode struct {
	Config storagenode.Config
	*storagenode.Peer
}

StorageNode contains all the processes needed to run a full StorageNode setup.

func (*StorageNode) NodeURL added in v1.3.2 

func (system *StorageNode) NodeURL() storj.NodeURL

NodeURL returns the storj.NodeURL.

func (*StorageNode) URL added in v1.1.1 

func (system *StorageNode) URL() string

URL returns the node url as a string. 

type Uplink struct {
	Log              *zap.Logger
	Info             pb.Node
	Identity         *identity.FullIdentity
	Dialer           rpc.Dialer
	StorageNodeCount int

	APIKey map[storj.NodeID]*macaroon.APIKey

	// Projects is indexed by the satellite number.
	Projects []*Project
}

Uplink is a general purpose

func (*Uplink) Addr 

func (client *Uplink) Addr() string

Addr returns uplink address

func (*Uplink) CreateBucket 

func (client *Uplink) CreateBucket(ctx context.Context, satellite *Satellite, bucketName string) error

CreateBucket creates a new bucket

func (*Uplink) DeleteBucket added in v0.33.2 

func (client *Uplink) DeleteBucket(ctx context.Context, satellite *Satellite, bucketName string) error

DeleteBucket deletes a bucket.

func (*Uplink) DeleteObject added in v0.33.2 

func (client *Uplink) DeleteObject(ctx context.Context, satellite *Satellite, bucketName string, path storj.Path) error

DeleteObject deletes an object at the path in a bucket

func (*Uplink) DialMetainfo 

func (client *Uplink) DialMetainfo(ctx context.Context, destination Peer, apikey *macaroon.APIKey) (*metainfo.Client, error)

DialMetainfo dials destination with apikey and returns metainfo Client

func (*Uplink) DialPiecestore 

func (client *Uplink) DialPiecestore(ctx context.Context, destination Peer) (*piecestore.Client, error)

DialPiecestore dials destination storagenode and returns a piecestore client.

func (*Uplink) Download 

func (client *Uplink) Download(ctx context.Context, satellite *Satellite, bucketName string, path storj.Path) ([]byte, error)

Download data from specific satellite

func (*Uplink) DownloadStream 

func (client *Uplink) DownloadStream(ctx context.Context, satellite *Satellite, bucketName string, path storj.Path) (_ io.ReadCloser, cleanup func() error, err error)

DownloadStream returns stream for downloading data

func (*Uplink) DownloadStreamRange 

func (client *Uplink) DownloadStreamRange(ctx context.Context, satellite *Satellite, bucketName string, path storj.Path, start, limit int64) (_ io.ReadCloser, cleanup func() error, err error)

DownloadStreamRange returns stream for downloading data

func (*Uplink) GetConfig 

func (client *Uplink) GetConfig(satellite *Satellite) UplinkConfig

GetConfig returns a default config for a given satellite.

func (*Uplink) GetNewProject added in v1.3.2 

func (client *Uplink) GetNewProject(ctx context.Context, satellite *Satellite) (*uplink.Project, error)

GetNewProject returns a uplink.Project which allows interactions with a specific project

func (*Uplink) GetProject 

func (client *Uplink) GetProject(ctx context.Context, satellite *Satellite) (*libuplink.Project, error)

GetProject returns a libuplink.Project which allows interactions with a specific project

func (*Uplink) GetProjectAndBucket 

func (client *Uplink) GetProjectAndBucket(ctx context.Context, satellite *Satellite, bucketName string, clientCfg UplinkConfig) (_ *libuplink.Project, _ *libuplink.Bucket, err error)

GetProjectAndBucket returns a libuplink.Project and Bucket which allows interactions with a specific project and its buckets

func (*Uplink) ID 

func (client *Uplink) ID() storj.NodeID

ID returns uplink id

func (*Uplink) Local 

func (client *Uplink) Local() pb.Node

Local returns uplink info 

func (client *Uplink) NewLibuplink(ctx context.Context) (*libuplink.Uplink, error)

NewLibuplink creates a libuplink.Uplink object with the testplanet Uplink config

func (*Uplink) Shutdown 

func (client *Uplink) Shutdown() error

Shutdown shuts down all uplink dependencies

func (*Uplink) Upload 

func (client *Uplink) Upload(ctx context.Context, satellite *Satellite, bucket string, path storj.Path, data []byte) error

Upload data to specific satellite

func (*Uplink) UploadWithClientConfig 

func (client *Uplink) UploadWithClientConfig(ctx context.Context, satellite *Satellite, clientConfig UplinkConfig, bucketName string, path storj.Path, data []byte) (err error)

UploadWithClientConfig uploads data to specific satellite with custom client configuration

func (*Uplink) UploadWithExpiration 

func (client *Uplink) UploadWithExpiration(ctx context.Context, satellite *Satellite, bucketName string, path storj.Path, data []byte, expiration time.Time) error

UploadWithExpiration data to specific satellite and expiration time

type UplinkConfig added in v0.35.2 

type UplinkConfig struct {
	AccessConfig
	Client ClientConfig
	RS     RSConfig
	Enc    EncryptionConfig
	TLS    tlsopts.Config
}

UplinkConfig uplink configuration

func (UplinkConfig) GetEncryptionParameters added in v0.35.2 

func (c UplinkConfig) GetEncryptionParameters() storj.EncryptionParameters

GetEncryptionParameters returns the configured encryption scheme for new uploads Blocksize should align with the stripe size therefore multiples of stripes should fit in every encryption block. Instead of lettings users configure this multiple value, we hardcode stripesPerBlock as 2 for simplicity.

func (UplinkConfig) GetPathCipherSuite added in v0.35.2 

func (c UplinkConfig) GetPathCipherSuite() storj.CipherSuite

GetPathCipherSuite returns the cipher suite used for path encryption for bucket objects

func (UplinkConfig) GetRedundancyScheme added in v0.35.2 

func (c UplinkConfig) GetRedundancyScheme() storj.RedundancyScheme

GetRedundancyScheme returns the configured redundancy scheme for new uploads

func (UplinkConfig) GetSegmentSize added in v0.35.2 

func (c UplinkConfig) GetSegmentSize() memory.Size

GetSegmentSize returns the segment size set in uplink config

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.