connectors

Documentation

Index

• Variables
• func RecordDownloadMetrics(ctx context.Context, m *DownloadMetrics)
• func Register(name string, connector Connector)
• type Connector
• type DownloadMetrics
• type Env
• type FileIterator
  • func ConsumeAsIterator(ctx context.Context, env *Env, source *Source) (FileIterator, error)
• type PropertySchema
  • func (ps PropertySchema) ValidateType(val any) bool
• type PropertySchemaType
• type SamplePolicy
• type Source
  • func (s *Source) PropertiesEquals(o *Source) bool
  • func (s *Source) Validate() error
• type Spec
• type VariableSchema

Variables

var Connectors = make(map[string]Connector)

Connectors tracks all registered connector drivers.

var ErrIngestionLimitExceeded = fmt.Errorf("connectors: source ingestion exceeds limit")

Functions

func RecordDownloadMetrics

func RecordDownloadMetrics(ctx context.Context, m *DownloadMetrics)

func Register

func Register(name string, connector Connector)

Register tracks a connector driver.

Types

type Connector

type Connector interface {
	Spec() Spec

	ConsumeAsIterator(ctx context.Context, env *Env, source *Source) (FileIterator, error)

	// HasAnonymousAccess returns true if external system can be accessed without credentials
	HasAnonymousAccess(ctx context.Context, env *Env, source *Source) (bool, error)
}

Connector is a driver for ingesting data from an external system.

type DownloadMetrics

type DownloadMetrics struct {
	Connector string
	Ext       string
	Partial   bool
	Duration  time.Duration
	Size      int64
}

type Env

type Env struct {
	RepoDriver string
	RepoRoot   string
	// user provided env variables kept with keys converted to uppercase
	Variables           map[string]string
	AllowHostAccess     bool
	StorageLimitInBytes int64
}

Env contains contextual information for a source, such as the repo it came from and (in the future) secrets configured by the user.

type FileIterator

type FileIterator interface {
	// Close do cleanup and release resources
	Close() error
	// NextBatch returns a list of file downloaded from external sources
	// NextBatch cleanups file created in previous batch
	NextBatch(limit int) ([]string, error)
	// HasNext can be utlisied to check if iterator has more elements left
	HasNext() bool
}

FileIterator provides ways to iteratively ingest files downloaded from external sources Clients should call close once they are done with iterator to release any resources

func ConsumeAsIterator

func ConsumeAsIterator(ctx context.Context, env *Env, source *Source) (FileIterator, error)

type PropertySchema

type PropertySchema struct {
	Key         string
	Type        PropertySchemaType
	Required    bool
	DisplayName string
	Description string
	Placeholder string
	Hint        string
	Href        string
}

PropertySchema provides the schema for a property supported by a connector.

func (PropertySchema) ValidateType

func (ps PropertySchema) ValidateType(val any) bool

ValidateType checks that val has the correct type.

type PropertySchemaType

type PropertySchemaType int

PropertySchemaType is an enum of types supported for connector properties.

const (
	UnspecifiedPropertyType PropertySchemaType = iota
	StringPropertyType
	NumberPropertyType
	BooleanPropertyType
	InformationalPropertyType
)

type SamplePolicy

type SamplePolicy struct {
	Strategy string
	Sample   float32
	Limit    int
}

SamplePolicy tells the connector to only ingest a sample of data from the source. Support for it is currently not implemented.

type Source

type Source struct {
	Name          string
	Connector     string
	ExtractPolicy *runtimev1.Source_ExtractPolicy
	Properties    map[string]any
	Timeout       int32
}

Source represents a dataset to ingest using a specific connector (like a connector instance).

func (*Source) PropertiesEquals

func (s *Source) PropertiesEquals(o *Source) bool

func (*Source) Validate

func (s *Source) Validate() error

Validate checks the source's properties against its connector's spec.

type Spec

type Spec struct {
	DisplayName        string
	Description        string
	Properties         []PropertySchema
	ConnectorVariables []VariableSchema
	Help               string
}

Spec provides metadata about a connector and the properties it supports.

type VariableSchema

type VariableSchema struct {
	Key           string
	Default       string
	Help          string
	Secret        bool
	ValidateFunc  func(any interface{}) error
	TransformFunc func(any interface{}) interface{}
}