Documentation ¶
Index ¶
- Variables
- func Drop(driver string, config map[string]any, logger *zap.Logger) error
- func NewPermissionDeniedError(msg string) error
- func RecordDownloadMetrics(ctx context.Context, m *DownloadMetrics)
- func Register(name string, driver Driver)
- func RegisterAsConnector(name string, driver Driver)
- type CatalogEntry
- type CatalogStore
- type Dialect
- type DownloadMetrics
- type Driver
- type FileIterator
- type FileStore
- type Handle
- type InformationSchema
- type IngestionSummary
- type Instance
- type NoOpProgress
- type OLAPStore
- type ObjectStore
- type ObjectType
- type PermissionDeniedError
- type Progress
- type ProgressUnit
- type PropertySchema
- type PropertySchemaType
- type QueryOption
- type RegistryStore
- type RepoObjectStat
- type RepoStore
- type Resource
- type Result
- type RowIterator
- type SQLStore
- type Spec
- type Statement
- type Table
- type TransferOptions
- type Transporter
- type WatchCallback
- type WatchEvent
- type WithConnectionFunc
Constants ¶
This section is empty.
Variables ¶
var Connectors = make(map[string]Driver)
Connectors tracks all registered connector drivers.
var Drivers = make(map[string]Driver)
Drivers is a registry of drivers.
var ErrDropNotSupported = errors.New("driver: drop not supported")
ErrDropNotSupported indicates the driver doesn't support dropping its underlying store.
var ErrFileAlreadyExists = errors.New("file already exists")
var ErrInconsistentControllerVersion = errors.New("controller: inconsistent version")
ErrInconsistentControllerVersion is returned from Controller when an unexpected controller version is observed in the DB. An unexpected controller version will only be observed if multiple controllers are running simultanesouly (split brain).
var ErrIngestionLimitExceeded = fmt.Errorf("connectors: source ingestion exceeds limit")
var ErrIteratorDone = errors.New("empty iterator")
var ErrNotFound = errors.New("driver: not found")
ErrNotFound indicates the resource wasn't found.
var ErrResourceAlreadyExists = errors.New("controller: resource already exists")
ErrResourceAlreadyExists is returned from catalog functions when attempting to create a resource that already exists.
var ErrResourceNotFound = errors.New("controller: resource not found")
ErrResourceNotFound is returned from catalog functions when a referenced resource does not exist.
var ErrUnsupportedConnector = errors.New("drivers: connector not supported")
ErrUnsupportedConnector is returned from Ingest for unsupported connectors.
Functions ¶
func Drop ¶ added in v0.27.0
Drop tears down a store. Drivers that do not support it return ErrDropNotSupported.
func NewPermissionDeniedError ¶ added in v0.30.0
func RecordDownloadMetrics ¶ added in v0.30.0
func RecordDownloadMetrics(ctx context.Context, m *DownloadMetrics)
func RegisterAsConnector ¶ added in v0.30.0
RegisterAsConnector tracks a connector driver.
Types ¶
type CatalogEntry ¶ added in v0.16.0
type CatalogEntry struct { Name string Type ObjectType Object proto.Message Path string Embedded bool BytesIngested int64 Parents []string Children []string CreatedOn time.Time UpdatedOn time.Time RefreshedOn time.Time }
CatalogEntry represents one object in the catalog, such as a source.
func (*CatalogEntry) GetMetricsView ¶ added in v0.16.0
func (e *CatalogEntry) GetMetricsView() *runtimev1.MetricsView
func (*CatalogEntry) GetModel ¶ added in v0.16.0
func (e *CatalogEntry) GetModel() *runtimev1.Model
func (*CatalogEntry) GetSource ¶ added in v0.16.0
func (e *CatalogEntry) GetSource() *runtimev1.Source
func (*CatalogEntry) GetTable ¶ added in v0.16.0
func (e *CatalogEntry) GetTable() *runtimev1.Table
func (*CatalogEntry) IsMetricsView ¶ added in v0.33.0
func (e *CatalogEntry) IsMetricsView() bool
type CatalogStore ¶
type CatalogStore interface { FindEntries(ctx context.Context, t ObjectType) ([]*CatalogEntry, error) FindEntry(ctx context.Context, name string) (*CatalogEntry, error) CreateEntry(ctx context.Context, entry *CatalogEntry) error UpdateEntry(ctx context.Context, entry *CatalogEntry) error DeleteEntry(ctx context.Context, name string) error DeleteEntries(ctx context.Context) error NextControllerVersion(ctx context.Context) (int64, error) CheckControllerVersion(ctx context.Context, v int64) error FindResources(ctx context.Context) ([]Resource, error) CreateResource(ctx context.Context, v int64, r Resource) error UpdateResource(ctx context.Context, v int64, r Resource) error DeleteResource(ctx context.Context, v int64, k, n string) error DeleteResources(ctx context.Context) error }
CatalogStore is implemented by drivers capable of storing catalog info for a specific instance.
type DownloadMetrics ¶ added in v0.30.0
type Driver ¶
type Driver interface { Spec() Spec // Open opens a new connection to an underlying store. Open(config map[string]any, shared bool, client activity.Client, logger *zap.Logger) (Handle, error) // Drop tears down a store. Drivers that do not support it return ErrDropNotSupported. Drop(config map[string]any, logger *zap.Logger) error // HasAnonymousSourceAccess returns true if external system can be accessed without credentials HasAnonymousSourceAccess(ctx context.Context, src map[string]any, logger *zap.Logger) (bool, error) }
Driver represents an underlying DB.
type FileIterator ¶ added in v0.30.0
type FileIterator interface { // Close do cleanup and release resources Close() error // NextBatch returns a list of file downloaded from external sources // and cleanups file created in previous batch NextBatch(limit int) ([]string, error) // NextBatchSize returns a list of file downloaded from external sources // such that the size of all files is less than equal to sizeInBytes // and cleanups file created in previous batch NextBatchSize(sizeInBytes int64) ([]string, error) // HasNext can be utlisied to check if iterator has more elements left HasNext() bool // Size returns size of data downloaded in unit. // Returns 0,false if not able to compute size in given unit Size(unit ProgressUnit) (int64, bool) // KeepFilesUntilClose marks the iterator to keep the files until close is called. // This is used when the entire list of files is used at once in certain cases. KeepFilesUntilClose(keepFilesUntilClose bool) }
FileIterator provides ways to iteratively download files from external sources Clients should call close once they are done with iterator to release any resources
type Handle ¶ added in v0.32.0
type Handle interface { // Driver type (like "duckdb") Driver() string // Config used to open the Connection Config() map[string]any // Migrate prepares the connection for use. It will be called before the connection is first used. // (Not to be confused with migrating artifacts, which is handled by the runtime and tracked in the catalog.) Migrate(ctx context.Context) error // MigrationStatus returns the connection's current and desired migration version (if applicable) MigrationStatus(ctx context.Context) (current int, desired int, err error) // Close closes the connection Close() error // AsRegistry returns a AsRegistry if the driver can serve as such, otherwise returns false. // The registry is responsible for tracking instances and repos. AsRegistry() (RegistryStore, bool) // AsCatalogStore returns a AsCatalogStore if the driver can serve as such, otherwise returns false. // A catalog is used to store state about migrated/deployed objects (such as sources and metrics views). AsCatalogStore(instanceID string) (CatalogStore, bool) // AsRepoStore returns a AsRepoStore if the driver can serve as such, otherwise returns false. // A repo stores file artifacts (either in a folder or virtualized in a database). AsRepoStore(instanceID string) (RepoStore, bool) // AsOLAP returns an AsOLAP if the driver can serve as such, otherwise returns false. // OLAP stores are where we actually store, transform, and query users' data. AsOLAP(instanceID string) (OLAPStore, bool) // AsObjectStore returns an ObjectStore if the driver can serve as such, otherwise returns false. AsObjectStore() (ObjectStore, bool) // AsFileStore returns a Filetore if the driver can serve as such, otherwise returns false. AsFileStore() (FileStore, bool) // AsTransporter optionally returns an implementation for moving data between two connectors. // One of the input connections may be the Connection itself. // Examples: // a) myDuckDB.AsTransporter(myGCS, myDuckDB) // b) myBeam.AsTransporter(myGCS, myS3) // In the future AsTransporter(from Handle, to Handle) (Transporter, bool) // AsSQLStore returns a SQLStore if the driver can serve as such, otherwise returns false. AsSQLStore() (SQLStore, bool) }
Handle represents a connection to an underlying DB. It should implement one or more of RegistryStore, CatalogStore, RepoStore, and OLAPStore.
type InformationSchema ¶
type InformationSchema interface { All(ctx context.Context) ([]*Table, error) Lookup(ctx context.Context, name string) (*Table, error) }
InformationSchema contains information about existing tables in an OLAP driver.
type IngestionSummary ¶ added in v0.24.0
type IngestionSummary struct {
BytesIngested int64
}
IngestionSummary is details about ingestion
type Instance ¶
type Instance struct { // Identifier ID string // Driver name to connect to for OLAP OLAPConnector string // Driver name for reading/editing code artifacts RepoConnector string // EmbedCatalog tells the runtime to store the instance's catalog in its OLAP store instead // of in the runtime's metadata store. Currently only supported for the duckdb driver. EmbedCatalog bool `db:"embed_catalog"` // CreatedOn is when the instance was created CreatedOn time.Time `db:"created_on"` // UpdatedOn is when the instance was last updated in the registry UpdatedOn time.Time `db:"updated_on"` // Variables contains user-provided variables Variables map[string]string `db:"variables"` // Instance specific connectors Connectors []*runtimev1.Connector `db:"connectors"` // ProjectVariables contains default variables from rill.yaml // (NOTE: This can always be reproduced from rill.yaml, so it's really just a handy cache of the values.) ProjectVariables map[string]string `db:"project_variables"` // ProjectVariables contains default connectors from rill.yaml ProjectConnectors []*runtimev1.Connector `db:"project_connectors"` // IngestionLimitBytes is total data allowed to ingest across all sources // 0 means there is no limit IngestionLimitBytes int64 `db:"ingestion_limit_bytes"` // Annotations to enrich activity events (like usage tracking) Annotations map[string]string }
Instance represents a single data project, meaning one OLAP connection, one repo connection, and one catalog connection.
func (*Instance) ResolveVariables ¶ added in v0.23.0
ResolveVariables returns the final resolved variables
type NoOpProgress ¶ added in v0.30.0
type NoOpProgress struct{}
func (NoOpProgress) Observe ¶ added in v0.30.0
func (n NoOpProgress) Observe(val int64, unit ProgressUnit)
func (NoOpProgress) Target ¶ added in v0.30.0
func (n NoOpProgress) Target(val int64, unit ProgressUnit)
type OLAPStore ¶
type OLAPStore interface { Dialect() Dialect WithConnection(ctx context.Context, priority int, longRunning, tx bool, fn WithConnectionFunc) error Exec(ctx context.Context, stmt *Statement) error Execute(ctx context.Context, stmt *Statement) (*Result, error) InformationSchema() InformationSchema EstimateSize() (int64, bool) }
OLAPStore is implemented by drivers that are capable of storing, transforming and serving analytical queries.
type ObjectStore ¶ added in v0.30.0
type ObjectType ¶ added in v0.16.0
type ObjectType int
Constants representing the kinds of catalog objects.
const ( ObjectTypeUnspecified ObjectType = 0 ObjectTypeTable ObjectType = 1 ObjectTypeSource ObjectType = 2 ObjectTypeModel ObjectType = 3 ObjectTypeMetricsView ObjectType = 4 )
type PermissionDeniedError ¶ added in v0.30.0
type PermissionDeniedError struct {
// contains filtered or unexported fields
}
func (*PermissionDeniedError) Error ¶ added in v0.30.0
func (e *PermissionDeniedError) Error() string
type Progress ¶ added in v0.30.0
type Progress interface { Target(val int64, unit ProgressUnit) // Observe is used by caller to provide incremental updates Observe(val int64, unit ProgressUnit) }
Progress is an interface for communicating progress info
type ProgressUnit ¶ added in v0.30.0
type ProgressUnit int
const ( ProgressUnitByte ProgressUnit = iota ProgressUnitFile ProgressUnitRecord )
type PropertySchema ¶ added in v0.30.0
type PropertySchema struct { Key string Type PropertySchemaType Required bool DisplayName string Description string Placeholder string // Default can be different from placeholder in the sense that placeholder should not be used as default value. // If a default is set then it should also be used as a placeholder. Default string Hint string Href string Secret bool ValidateFunc func(any interface{}) error TransformFunc func(any interface{}) interface{} }
PropertySchema provides the schema for a property supported by a connector.
func (PropertySchema) ValidateType ¶ added in v0.30.0
func (ps PropertySchema) ValidateType(val any) bool
ValidateType checks that val has the correct type.
type PropertySchemaType ¶ added in v0.30.0
type PropertySchemaType int
PropertySchemaType is an enum of types supported for connector properties.
const ( UnspecifiedPropertyType PropertySchemaType = iota StringPropertyType NumberPropertyType BooleanPropertyType InformationalPropertyType )
type QueryOption ¶ added in v0.33.2
type QueryOption struct { // TotalLimitInBytes rerpresent the max limit on the bytes that should be downloaded in a file TotalLimitInBytes int64 }
type RegistryStore ¶
type RegistryStore interface { FindInstances(ctx context.Context) ([]*Instance, error) FindInstance(ctx context.Context, id string) (*Instance, error) CreateInstance(ctx context.Context, instance *Instance) error DeleteInstance(ctx context.Context, id string) error EditInstance(ctx context.Context, instance *Instance) error }
RegistryStore is implemented by drivers capable of storing and looking up instances and repos.
type RepoObjectStat ¶ added in v0.15.0
type RepoStore ¶
type RepoStore interface { Driver() string // Root returns directory where artifacts are stored. Root() string CommitHash(ctx context.Context) (string, error) ListRecursive(ctx context.Context, glob string) ([]string, error) Get(ctx context.Context, path string) (string, error) Stat(ctx context.Context, path string) (*RepoObjectStat, error) Put(ctx context.Context, path string, reader io.Reader) error Rename(ctx context.Context, fromPath string, toPath string) error Delete(ctx context.Context, path string) error Sync(ctx context.Context) error Watch(ctx context.Context, cb WatchCallback) error }
RepoStore is implemented by drivers capable of storing code artifacts. It mirrors a file system, but may be virtualized by a database for non-local deployments.
type Result ¶ added in v0.15.0
type Result struct { *sqlx.Rows Schema *runtimev1.StructType // contains filtered or unexported fields }
Result wraps the results of query.
func (*Result) Close ¶ added in v0.18.0
Close wraps rows.Close and calls the Result's cleanup function (if it is set). Close should be idempotent.
func (*Result) SetCleanupFunc ¶ added in v0.18.0
SetCleanupFunc sets a function, which will be called when the Result is closed.
type RowIterator ¶ added in v0.31.0
type RowIterator interface { // Schema of the underlying data Schema(ctx context.Context) (*runtimev1.StructType, error) // Next fetches next row Next(ctx context.Context) ([]any, error) // Close closes the iterator and frees resources Close() error // Size returns total size of data downloaded in unit. // Returns 0,false if not able to compute size in given unit Size(unit ProgressUnit) (uint64, bool) }
RowIterator returns an iterator to iterate over result of a sql query
type SQLStore ¶ added in v0.31.0
type SQLStore interface { // Query returns driver.RowIterator to iterate over results row by row Query(ctx context.Context, props map[string]any) (RowIterator, error) // QueryAsFiles downloads results into files and returns an iterator to iterate over them QueryAsFiles(ctx context.Context, props map[string]any, opt *QueryOption, p Progress) (FileIterator, error) }
SQLStore is implemented by drivers capable of running sql queries and generating an iterator to consume results. In future the results can be produced in other formats like arrow as well. May be call it DataWarehouse to differentiate from OLAP or postgres?
type Spec ¶ added in v0.30.0
type Spec struct { DisplayName string Description string ServiceAccountDocs string SourceProperties []PropertySchema ConfigProperties []PropertySchema Help string }
Spec provides metadata about a connector and the properties it supports.
type Statement ¶
type Statement struct { Query string Args []any DryRun bool Priority int LongRunning bool ExecutionTimeout time.Duration }
Statement wraps a query to execute against an OLAP driver.
type Table ¶
type Table struct { Database string DatabaseSchema string Name string View bool Schema *runtimev1.StructType }
Table represents a table in an information schema.
type TransferOptions ¶ added in v0.34.1
type TransferOptions struct { AllowHostAccess bool RepoRoot string LimitInBytes int64 Progress Progress AcquireConnector func(string) (Handle, func(), error) }
TransferOptions provide execution context for Transporter.Transfer
type Transporter ¶ added in v0.30.0
type Transporter interface {
Transfer(ctx context.Context, srcProps, sinkProps map[string]any, opts *TransferOptions) error
}
Transporter implements logic for moving data between two connectors (the actual connector objects are provided in AsTransporter)
type WatchCallback ¶ added in v0.30.0
type WatchCallback func(event []WatchEvent)
type WatchEvent ¶ added in v0.30.0
type WithConnectionFunc ¶ added in v0.18.0
type WithConnectionFunc func(wrappedCtx context.Context, ensuredCtx context.Context, conn *sql.Conn) error
WithConnectionFunc is a callback function that provides a context to be used in further OLAP store calls to enforce affinity to a single connection. It also provides pointers to the actual database/sql and database/sql/driver connections. It's called with two contexts: wrappedCtx wraps the input context (including cancellation), and ensuredCtx wraps a background context (ensuring it can never be cancelled).