connectors

Documentation

Index

• Variables
• func Register(name string, connector Connector)
• type Connector
• type Env
• type FileIterator
  • func ConsumeAsIterator(ctx context.Context, env *Env, source *Source) (FileIterator, error)
• type PropertySchema
  • func (ps PropertySchema) ValidateType(val any) bool
• type PropertySchemaType
• type SamplePolicy
• type Source
  • func (s *Source) PropertiesEquals(o *Source) bool
  • func (s *Source) Validate() error
• type Spec

Variables

var Connectors = make(map[string]Connector)

Connectors tracks all registered connector drivers.

Functions

func Register

func Register(name string, connector Connector)

Register tracks a connector driver.

Types

type Connector

type Connector interface {
	Spec() Spec

	ConsumeAsIterator(ctx context.Context, env *Env, source *Source) (FileIterator, error)
}

Connector is a driver for ingesting data from an external system.

type Env

type Env struct {
	RepoDriver string
	RepoDSN    string
	// user provided env variables kept with keys converted to uppercase
	Variables            map[string]string
	AllowHostCredentials bool
}

Env contains contextual information for a source, such as the repo it came from and (in the future) secrets configured by the user.

type FileIterator

type FileIterator interface {
	// Close do cleanup and release resources
	Close() error
	// NextBatch returns a list of file downloaded from external sources
	// NextBatch cleanups file created in previous batch
	NextBatch(limit int) ([]string, error)
	// HasNext can be utlisied to check if iterator has more elements left
	HasNext() bool
}

FileIterator provides ways to iteratively ingest files downloaded from external sources Clients should call close once they are done with iterator to release any resources

func ConsumeAsIterator

func ConsumeAsIterator(ctx context.Context, env *Env, source *Source) (FileIterator, error)

type PropertySchema

type PropertySchema struct {
	Key         string
	Type        PropertySchemaType
	Required    bool
	DisplayName string
	Description string
	Placeholder string
	Hint        string
	Href        string
}

PropertySchema provides the schema for a property supported by a connector.

func (PropertySchema) ValidateType

func (ps PropertySchema) ValidateType(val any) bool

ValidateType checks that val has the correct type.

type PropertySchemaType

type PropertySchemaType int

PropertySchemaType is an enum of types supported for connector properties.

const (
	UnspecifiedPropertyType PropertySchemaType = iota
	StringPropertyType
	NumberPropertyType
	BooleanPropertyType
	InformationalPropertyType
)

type SamplePolicy

type SamplePolicy struct {
	Strategy string
	Sample   float32
	Limit    int
}

SamplePolicy tells the connector to only ingest a sample of data from the source. Support for it is currently not implemented.

type Source

type Source struct {
	Name          string
	Connector     string
	ExtractPolicy *runtimev1.Source_ExtractPolicy
	Properties    map[string]any
	Timeout       int32
}

Source represents a dataset to ingest using a specific connector (like a connector instance).

func (*Source) PropertiesEquals

func (s *Source) PropertiesEquals(o *Source) bool

func (*Source) Validate

func (s *Source) Validate() error

Validate checks the source's properties against its connector's spec.

type Spec

type Spec struct {
	DisplayName string
	Description string
	Properties  []PropertySchema
}

Spec provides metadata about a connector and the properties it supports.