connectors

Documentation

Index

• Variables
• func ConsumeAsFile(ctx context.Context, env *Env, source *Source) (string, error)
• func Register(name string, connector Connector)
• type Connector
• type Env
• type PropertySchema
  • func (ps PropertySchema) ValidateType(val any) bool
• type PropertySchemaType
• type SamplePolicy
• type Source
  • func (s *Source) PropertiesEquals(o *Source) bool
  • func (s *Source) Validate() error
• type Spec

Variables

var Connectors = make(map[string]Connector)

Connectors tracks all registered connector drivers.

Functions

func ConsumeAsFile

func ConsumeAsFile(ctx context.Context, env *Env, source *Source) (string, error)

func Register

func Register(name string, connector Connector)

Register tracks a connector driver.

Types

type Connector

type Connector interface {
	Spec() Spec

	ConsumeAsFile(ctx context.Context, env *Env, source *Source) (string, error)
}

Connector is a driver for ingesting data from an external system.

type Env

type Env struct {
	RepoDriver string
	RepoDSN    string
}

Env contains contextual information for a source, such as the repo it came from and (in the future) secrets configured by the user.

type PropertySchema

type PropertySchema struct {
	Key         string
	Type        PropertySchemaType
	Required    bool
	DisplayName string
	Description string
	Placeholder string
	Hint        string
	Href        string
}

PropertySchema provides the schema for a property supported by a connector.

func (PropertySchema) ValidateType

func (ps PropertySchema) ValidateType(val any) bool

ValidateType checks that val has the correct type.

type PropertySchemaType

type PropertySchemaType int

PropertySchemaType is an enum of types supported for connector properties.

const (
	UnspecifiedPropertyType PropertySchemaType = iota
	StringPropertyType
	NumberPropertyType
	BooleanPropertyType
	InformationalPropertyType
)

type SamplePolicy

type SamplePolicy struct {
	Strategy string
	Sample   float32
	Limit    int
}

SamplePolicy tells the connector to only ingest a sample of data from the source. Support for it is currently not implemented.

type Source

type Source struct {
	Name         string
	Connector    string
	SamplePolicy *SamplePolicy
	Properties   map[string]any
}

Source represents a dataset to ingest using a specific connector (like a connector instance).

func (*Source) PropertiesEquals

func (s *Source) PropertiesEquals(o *Source) bool

func (*Source) Validate

func (s *Source) Validate() error

Validate checks the source's properties against its connector's spec.

type Spec

type Spec struct {
	DisplayName string
	Description string
	Properties  []PropertySchema
}

Spec provides metadata about a connector and the properties it supports.