generator

README

General

The generator plugin is able to generate sample records with its source. It has no destination and trying to use it will result in an error.

The data is generated in JSON format. The JSON objects themselves are generated using a field specification, which is explained in more details in the Configuration section below.

The plugin is great for getting started with Conduit but also for certain types of performance tests.

How to build?

Run make build-generator-plugin.

Testing

Run make test to run all the unit tests.

Configuration

recordCount The number of records to be generated. If a negative value is used, the source will generate records until stopped.

readTime The time it takes to 'read' a record. It can be used to simulate latency and, for example, simulate slow or fast sources.

fields A comma-separated list of name:type tokens, where type can be: int, string, time, bool. An example is: id:int,name:string,joined:time,admin:bool.

Documentation

Index

• Constants
• type Config
  • func Parse(config plugins.Config) (Config, error)
• type Source
  • func (s *Source) Ack(ctx context.Context, position record.Position) error
  • func (s *Source) Open(ctx context.Context, config plugins.Config) error
  • func (s *Source) Read(ctx context.Context, p record.Position) (record.Record, error)
  • func (s *Source) Teardown() error
  • func (s *Source) Validate(cfg plugins.Config) error
• type Spec
  • func (s Spec) Specify() (plugins.Specification, error)

Constants

const (
	RecordCount = "recordCount"
	ReadTime    = "readTime"
	Fields      = "fields"
)

Types

type Config

type Config struct {
	RecordCount int64
	ReadTime    time.Duration
	Fields      map[string]string
}

func Parse

func Parse(config plugins.Config) (Config, error)

type Source

type Source struct {
	Config Config
	// contains filtered or unexported fields
}

Source connector

func (*Source) Ack

func (s *Source) Ack(ctx context.Context, position record.Position) error

func (*Source) Open

func (s *Source) Open(ctx context.Context, config plugins.Config) error

func (*Source) Read

func (s *Source) Read(ctx context.Context, p record.Position) (record.Record, error)

func (*Source) Teardown

func (s *Source) Teardown() error

func (*Source) Validate

func (s *Source) Validate(cfg plugins.Config) error

type Spec

type Spec struct{}

func (Spec) Specify

func (s Spec) Specify() (plugins.Specification, error)

Specify returns the Plugin's Specification