runtime

package
v0.52.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 19, 2024 License: Apache-2.0 Imports: 49 Imported by: 0

README

runtime

The runtime a data infrastructure proxy and orchestrator – our data plane. It connects to data infrastructure and is responsible for dashboard queries, parsing code files, reconciling infra state, implementing connectors, enforcing (row-based) access policies, scheduling tasks, triggering reports, and much more.

It's designed as a modular component that can be embedded in local applications (as it is into Rill Developer) or deployed stand-alone in a cloud environment.

Code structure

The base directory contains a Runtime type that represents the lifecycle of the runtime. It ties together the sub-directories:

  • client contains a Go client library for connecting to a runtime server.
  • compilers contains logic for parsing Rill projects (incomplete, currently mostly implemented in services instead).
  • connectors contains connector implementations.
  • drivers contains interfaces and drivers for external data infrastructure that the runtime interfaces with (like DuckDB and Druid).
  • pkg contains utility libraries.
  • queries contains pre-defined analytical queries that the runtime can serve (used for profiling and dashboards).
  • server contains a server that implements the runtime's APIs.
  • testruntime contains helper functions for initializing a test runtime with test data.

Development

Developing the local application

Run rill devtool local. You need to stop and restart it using ctrl+C when you make code changes.

Developing for cloud

In one terminal, start a full cloud development environment except the runtime:

rill devtool start cloud --except runtime

In a separate terminal, start a runtime server:

go run ./cli runtime start

Optionally, deploy a seed project:

rill devtool seed cloud
Running tests

You can run all tests using:

go test ./runtime/...

Configuration

The runtime server is configured using environment variables parsed in cli/cmd/runtime/start.go.

Adding a new endpoint

We define our APIs using gRPC and use gRPC-Gateway to map the RPCs to a RESTful API. See proto/README.md for details.

To add a new endpoint:

  1. Describe the endpoint in proto/rill/runtime/v1/api.proto
  2. Re-generate gRPC and OpenAPI interfaces by running make proto.generate
  3. Copy the new handler signature from the RuntimeServiceServer interface in proto/gen/rill/runtime/v1/api_grpc_pb.go
  4. Paste the handler signature and implement it in a relevant file in runtime/server/

Adding a new analytical query endpoint

  1. Add a new endpoint for the query by following the steps in the section above ("Adding a new endpoint")
  2. Implement the query in runtime/queries by following the instructions in runtime/queries/README.md

Using a DuckDB nightly build

The following steps apply for macOS, but a similar approach should work for Linux.

  1. Download the latest DuckDB nightly from Github from the "Artifacts" section on the newest workflow run here)
  2. Unzip the downloaded archive and copy the libduckdb.dylib file in the libduckdb-osx-universal folder to /usr/local/lib
  • You must use the command-line to copy the file. If you touch it using the Finder, macOS will quarantine it. To remove a quarantine, run: xattr -d com.apple.quarantine libduckdb.dylib.
  1. DuckDB usually does not support older file formats, so delete the stage.db and stage.db.wal files in your dev-project
  2. Add the flag -tags=duckdb_use_lib when running go run or go build to use the nightly build of DuckDB
  • If testing the local frontend, you need to temporarily set it in the dev-runtime script in package.json
  • For details, see Linking DuckDB

Note: DuckDB often makes breaking changes to its APIs, so you may encounter other errors when using a dev version of DuckDB.

Documentation

Index

Constants

View Source 
const (
	ResourceKindProjectParser  string = "rill.runtime.v1.ProjectParser"
	ResourceKindSource         string = "rill.runtime.v1.Source"
	ResourceKindModel          string = "rill.runtime.v1.Model"
	ResourceKindMetricsView    string = "rill.runtime.v1.MetricsView"
	ResourceKindExplore        string = "rill.runtime.v1.Explore"
	ResourceKindMigration      string = "rill.runtime.v1.Migration"
	ResourceKindReport         string = "rill.runtime.v1.Report"
	ResourceKindAlert          string = "rill.runtime.v1.Alert"
	ResourceKindPullTrigger    string = "rill.runtime.v1.PullTrigger"
	ResourceKindRefreshTrigger string = "rill.runtime.v1.RefreshTrigger"
	ResourceKindBucketPlanner  string = "rill.runtime.v1.BucketPlanner"
	ResourceKindTheme          string = "rill.runtime.v1.Theme"
	ResourceKindComponent      string = "rill.runtime.v1.Component"
	ResourceKindCanvas         string = "rill.runtime.v1.Canvas"
	ResourceKindAPI            string = "rill.runtime.v1.API"
	ResourceKindConnector      string = "rill.runtime.v1.Connector"
)

Built-in resource kinds

Variables

View Source 
var BuiltinAPIs = map[string]*runtimev1.API{}

BuiltinAPIs is a map of built-in APIs (i.e. predefined APIs that are not created dynamically from a project's YAML files.)

View Source 
var ErrAINotConfigured = fmt.Errorf("an AI service is not configured for this instance")
View Source 
var ErrAdminNotConfigured = fmt.Errorf("an admin service is not configured for this instance")
View Source 
var ErrForbidden = errors.New("action not allowed")
View Source 
var GlobalProjectParserName = &runtimev1.ResourceName{Kind: ResourceKindProjectParser, Name: "parser"}

GlobalProjectParserName is the name of the instance-global project parser resource that is created for each new instance.

View Source 
var ReconcilerInitializers = make(map[string]ReconcilerInitializer)

ReconcilerInitializers is a registry of reconciler initializers for different resource kinds. There can be only one reconciler per resource kind.

View Source 
var ResolvedSecurityClosed = &ResolvedSecurity{
	access:      nil,
	fieldAccess: nil,
	rowFilter:   "",
	queryFilter: nil,
}

ResolvedSecurityClosed is a ResolvedSecurity that denies access.

View Source 
var ResolvedSecurityOpen = &ResolvedSecurity{
	access:      &truth,
	fieldAccess: nil,
	rowFilter:   "",
	queryFilter: nil,
}

ResolvedSecurityOpen is a ResolvedSecurity that allows access with no restrictions.

View Source 
var ResolverInitializers = make(map[string]ResolverInitializer)

ResolverInitializers tracks resolver initializers by name.

Functions

func PrettifyReconcileStatus added in v0.49.0 

func PrettifyReconcileStatus(s runtimev1.ReconcileStatus) string

PrettifyReconcileStatus returns the reconcile status in a user-friendly format suitable for printing.

func PrettifyResourceKind added in v0.49.0 

func PrettifyResourceKind(k string) string

PrettifyResourceKind returns the resource kind in a user-friendly format suitable for printing.

func RegisterBuiltinAPI added in v0.42.0 

func RegisterBuiltinAPI(name, resolver string, resolverProps map[string]any)

RegisterBuiltinAPI adds a built-in API with the given name that invokes the given resolver and resolver properties.

func RegisterReconcilerInitializer added in v0.32.0 

func RegisterReconcilerInitializer(resourceKind string, initializer ReconcilerInitializer)

RegisterReconciler registers a reconciler initializer for a specific resource kind

func RegisterResolverInitializer added in v0.42.0 

func RegisterResolverInitializer(name string, initializer ResolverInitializer)

RegisterResolverInitializer registers a resolver initializer by name.

func ResolveConnectorProperties added in v0.49.0 

func ResolveConnectorProperties(environment string, vars map[string]string, c *runtimev1.Connector) (map[string]string, error)

ResolveConnectorProperties resolves templating in the provided connector's properties. It always returns a clone of the properties, even if no templating is found, so the output is safe for further mutations.

func ResourceNameFromCompiler added in v0.42.0 

func ResourceNameFromCompiler(name compilerv1.ResourceName) *runtimev1.ResourceName

ResourceNameFromCompiler converts a compiler resource name to a runtime resource name.

func ResourceNameToCompiler added in v0.42.0 

func ResourceNameToCompiler(name *runtimev1.ResourceName) compilerv1.ResourceName

ResourceNameToCompiler converts a runtime resource name to a compiler resource name.

Types

type ConnectorConfig added in v0.43.0 

type ConnectorConfig struct {
	Driver  string
	Preset  map[string]string
	Project map[string]string
	Env     map[string]string
}

ConnectorConfig holds and resolves connector configuration. We support three levels of configuration: 1. Preset: provided when creating the instance (or set by the system, such as allow_host_access). Cannot be overridden. 2. Project: defined in the rill.yaml file. Can be overridden by the env. 3. Env: defined in the instance's variables (in the format "connector.name.var").

func (*ConnectorConfig) Resolve added in v0.43.0 

func (c *ConnectorConfig) Resolve() map[string]any

Resolve returns the final resolved connector configuration. It guarantees that all keys in the result are lowercase.

func (*ConnectorConfig) ResolveStrings added in v0.43.0 

func (c *ConnectorConfig) ResolveStrings() map[string]string

ResolveString is similar to Resolve, but it returns a map of strings.

type Controller added in v0.32.0 

type Controller struct {
	Runtime    *Runtime
	InstanceID string
	Logger     *zap.Logger
	Activity   *activity.Client
	// contains filtered or unexported fields
}

Controller manages the catalog for a single instance and runs reconcilers to migrate the catalog (and related resources in external databases) into the desired state. For information about how the controller schedules reconcilers, see `runtime/reconcilers/README.md`.

func NewController added in v0.32.0 

func NewController(ctx context.Context, rt *Runtime, instanceID string, logger *zap.Logger, ac *activity.Client) (*Controller, error)

NewController creates a new Controller

func (*Controller) AcquireConn added in v0.32.0 

func (c *Controller) AcquireConn(ctx context.Context, connector string) (drivers.Handle, func(), error)

AcquireOLAP gets a handle for a connector in the controller's instance.

func (*Controller) AcquireOLAP added in v0.32.0 

func (c *Controller) AcquireOLAP(ctx context.Context, connector string) (drivers.OLAPStore, func(), error)

AcquireOLAP gets an OLAP handle for a connector in the controller's instance.

func (*Controller) Cancel added in v0.33.0 

func (c *Controller) Cancel(ctx context.Context, name *runtimev1.ResourceName) error

Cancel cancels the current invocation of a resource's reconciler (if it's running). It does not re-enqueue the resource for reconciliation.

func (*Controller) Create added in v0.32.0 

func (c *Controller) Create(ctx context.Context, name *runtimev1.ResourceName, refs []*runtimev1.ResourceName, owner *runtimev1.ResourceName, paths []string, hidden bool, r *runtimev1.Resource) error

Create creates a resource and enqueues it for reconciliation. If a resource with the same name is currently being deleted, the deletion will be cancelled.

func (*Controller) Delete added in v0.32.0 

func (c *Controller) Delete(ctx context.Context, name *runtimev1.ResourceName) error

Delete soft-deletes a resource and enqueues it for reconciliation (with DeletedOn != nil). Once the deleting reconciliation has been completed, the resource will be hard deleted. If Delete is called from the resource's own reconciler, the resource will be hard deleted immediately (and the calling reconcile's ctx will be canceled immediately).

func (*Controller) Flush added in v0.32.0 

func (c *Controller) Flush(ctx context.Context) error

Flush forces a flush of the controller's catalog changes to persistent storage.

func (*Controller) Get added in v0.32.0 

func (c *Controller) Get(ctx context.Context, name *runtimev1.ResourceName, clone bool) (*runtimev1.Resource, error)

Get returns a resource by name. Soft-deleted resources (i.e. resources where DeletedOn != nil) are not returned.

func (*Controller) List added in v0.32.0 

func (c *Controller) List(ctx context.Context, kind, path string, clone bool) ([]*runtimev1.Resource, error)

List returns a list of resources of the specified kind. If kind is empty, all resources are returned. Soft-deleted resources (i.e. resources where DeletedOn != nil) are not returned.

func (*Controller) Lock added in v0.32.0 

func (c *Controller) Lock(ctx context.Context)

Lock locks the controller's catalog and delays scheduling of new reconciliations until the lock is released. It can only be called from within a reconciler invocation. While the lock is held, resources can only be edited by a caller using the ctx passed to Lock.

func (*Controller) Reconcile added in v0.33.0 

func (c *Controller) Reconcile(ctx context.Context, name *runtimev1.ResourceName) error

Reconcile enqueues a resource for reconciliation. If the resource is currently reconciling, the current reconciler will be cancelled first.

func (*Controller) Run added in v0.32.0 

func (c *Controller) Run(ctx context.Context) error

Run starts and runs the controller's event loop. It returns when ctx is cancelled or an unrecoverable error occurs. Before returning, it closes the controller, so it must only be called once. The event loop schedules/invokes resource reconciliation and periodically flushes catalog changes to persistent storage. The implementation centers around these internal functions: enqueue, processQueue (uses markPending, trySchedule, invoke), and processCompletedInvocation. See their docstrings for further details.

func (*Controller) Subscribe added in v0.33.1 

func (c *Controller) Subscribe(ctx context.Context, fn SubscribeCallback) error

Subscribe registers a callback that will receive resource update events. The same callback function will not be invoked concurrently. The callback function is invoked under a lock and must not call the controller.

func (*Controller) Unlock added in v0.32.0 

func (c *Controller) Unlock(ctx context.Context)

Unlock releases the lock acquired by Lock.

func (*Controller) UpdateError added in v0.32.0 

func (c *Controller) UpdateError(ctx context.Context, name *runtimev1.ResourceName, reconcileErr error) error

UpdateError updates a resource's error. Unlike UpdateMeta and UpdateSpec, it does not cancel or enqueue reconciliation for the resource.

func (*Controller) UpdateMeta added in v0.32.0 

func (c *Controller) UpdateMeta(ctx context.Context, name *runtimev1.ResourceName, refs []*runtimev1.ResourceName, owner *runtimev1.ResourceName, paths []string) error

UpdateMeta updates a resource's meta fields and enqueues it for reconciliation. If called from outside the resource's reconciler and the resource is currently reconciling, the current reconciler will be cancelled first.

func (*Controller) UpdateName added in v0.33.0 

func (c *Controller) UpdateName(ctx context.Context, name, newName, owner *runtimev1.ResourceName, paths []string) error

UpdateName renames a resource and updates annotations, and enqueues it for reconciliation. If called from outside the resource's reconciler and the resource is currently reconciling, the current reconciler will be cancelled first.

func (*Controller) UpdateSpec added in v0.32.0 

func (c *Controller) UpdateSpec(ctx context.Context, name *runtimev1.ResourceName, r *runtimev1.Resource) error

UpdateSpec updates a resource's spec and enqueues it for reconciliation. If called from outside the resource's reconciler and the resource is currently reconciling, the current reconciler will be cancelled first.

func (*Controller) UpdateState added in v0.32.0 

func (c *Controller) UpdateState(ctx context.Context, name *runtimev1.ResourceName, r *runtimev1.Resource) error

UpdateState updates a resource's state. It can only be called from within the resource's reconciler. NOTE: Calls to UpdateState succeed even if ctx is cancelled. This enables cancelled reconcilers to update state before finishing.

func (*Controller) WaitUntilIdle added in v0.35.0 

func (c *Controller) WaitUntilIdle(ctx context.Context, ignoreHidden bool) error

WaitUntilIdle returns when the controller is idle (i.e. no reconcilers are pending or running).

type ExportOptions added in v0.29.1 

type ExportOptions struct {
	Format       runtimev1.ExportFormat
	Priority     int
	PreWriteHook func(filename string) error
}

type Health added in v0.47.0 

type Health struct {
	HangingConn     error
	Registry        error
	InstancesHealth map[string]*InstanceHealth
}

type InstanceHealth added in v0.47.0 

type InstanceHealth struct {
	Controller string `json:"controller"`
	// OLAP error can be cached
	OLAP string `json:"olap"`
	Repo string `json:"repo"`
	// MetricsViews errors can be cached
	MetricsViews      map[string]InstanceHealthMetricsViewError `json:"metrics_views"`
	ParseErrCount     int                                       `json:"parse_error_count"`
	ReconcileErrCount int                                       `json:"reconcile_error_count"`

	// cached health check information can be used if controller version is same and metrics view spec is same
	ControllerVersion int64 `json:"controller_version"`
}

InstanceHealth contains health information for a single instance. The information about OLAP and metrics views is cached in the catalog. We want to avoid hitting the underlying OLAP engine when OLAP engine can scale to zero when no queries are generated within TTL. We do not want to keep it running just to check health. In such cases, we use the cached health information.

func (*InstanceHealth) To added in v0.47.0 

func (h *InstanceHealth) To() *runtimev1.InstanceHealth

type InstanceHealthMetricsViewError added in v0.51.0 

type InstanceHealthMetricsViewError struct {
	Err     string `json:"err"`
	Version int64  `json:"version"`
}

type Options added in v0.16.0 

type Options struct {
	MetastoreConnector           string
	SystemConnectors             []*runtimev1.Connector
	ConnectionCacheSize          int
	QueryCacheSizeBytes          int64
	SecurityEngineCacheSize      int
	ControllerLogBufferCapacity  int
	ControllerLogBufferSizeBytes int64
	AllowHostAccess              bool
}

type Query added in v0.16.0 

type Query interface {
	// Key should return a cache key that uniquely identifies the query
	Key() string
	// Deps should return the resource names that the query targets.
	// It's used to invalidate cached queries when the underlying data changes.
	// If a dependency doesn't exist, it is ignored. (So if the underlying resource kind is unknown, it can return all possible dependency names.)
	Deps() []*runtimev1.ResourceName
	// MarshalResult should return the query result and estimated cost in bytes for caching
	MarshalResult() *QueryResult
	// UnmarshalResult should populate a query with a cached result
	UnmarshalResult(v any) error
	// Resolve should execute the query against the instance's infra.
	// Error can be nil along with a nil result in general, i.e. when a model contains no rows aggregation results can be nil.
	Resolve(ctx context.Context, rt *Runtime, instanceID string, priority int) error
	// Export resolves the query and serializes the result to the writer.
	Export(ctx context.Context, rt *Runtime, instanceID string, w io.Writer, opts *ExportOptions) error
}

type QueryResult added in v0.26.0 

type QueryResult struct {
	Value any
	Bytes int64
}

type ReconcileResult added in v0.32.0 

type ReconcileResult struct {
	Err       error
	Retrigger time.Time
}

ReconcileResult propagates results from a reconciler invocation

type Reconciler added in v0.32.0 

type Reconciler interface {
	Close(ctx context.Context) error
	AssignSpec(from, to *runtimev1.Resource) error
	AssignState(from, to *runtimev1.Resource) error
	ResetState(r *runtimev1.Resource) error
	Reconcile(ctx context.Context, n *runtimev1.ResourceName) ReconcileResult
}

Reconciler implements reconciliation logic for all resources of a specific kind. Reconcilers are managed and invoked by a Controller.

type ReconcilerInitializer added in v0.32.0 

type ReconcilerInitializer func(*Controller) Reconciler

ReconcilerInitializer is a function that initializes a new reconciler for a specific controller

type ResolveOptions added in v0.42.0 

type ResolveOptions struct {
	InstanceID         string
	Resolver           string
	ResolverProperties map[string]any
	Args               map[string]any
	Claims             *SecurityClaims
}

ResolveOptions are the options passed to the runtime's Resolve method.

type ResolvedSecurity added in v0.47.0 

type ResolvedSecurity struct {
	// contains filtered or unexported fields
}

ResolvedSecurity represents the resolved security rules for a given claims against a specific resource.

func (*ResolvedSecurity) CanAccess added in v0.47.0 

func (r *ResolvedSecurity) CanAccess() bool

CanAccess returns whether the resource can be accessed.

func (*ResolvedSecurity) CanAccessAllFields added in v0.47.0 

func (r *ResolvedSecurity) CanAccessAllFields() bool

CanAccessAllFields returns whether all fields in the resource are allowed.

func (*ResolvedSecurity) CanAccessField added in v0.47.0 

func (r *ResolvedSecurity) CanAccessField(field string) bool

CanAccessField evaluates whether a specific field in the resource is allowed.

func (*ResolvedSecurity) QueryFilter added in v0.47.0 

func (r *ResolvedSecurity) QueryFilter() *runtimev1.Expression

QueryFilter returns a query expression to apply when querying the resource.

func (*ResolvedSecurity) RowFilter added in v0.47.0 

func (r *ResolvedSecurity) RowFilter() string

RowFilter returns a raw SQL expression to apply to the WHERE clause when querying the resource.

type Resolver added in v0.42.0 

type Resolver interface {
	// Close is called when done with the resolver.
	// Note that the Resolve method may not have been called when Close is called (in case of cache hits or validation failures).
	Close() error
	// Cacheable indicates whether the resolver's results can be cached.
	Cacheable() bool
	// Key that can be used for caching. It can be a large string since the value will be hashed.
	// The key should include all the properties and args that affect the output.
	// It does not need to include the instance ID or resolver name, as those are added separately to the cache key.
	Key() string
	// Refs access by the resolver. The output may be approximate, i.e. some of the refs may not exist.
	// The output should avoid duplicates and be stable between invocations.
	Refs() []*runtimev1.ResourceName
	// Validate the properties and args without running any expensive operations.
	Validate(ctx context.Context) error
	// ResolveInteractive resolves data for interactive use (e.g. API requests or alerts).
	ResolveInteractive(ctx context.Context) (ResolverResult, error)
	// ResolveExport resolve data for export (e.g. downloads or reports).
	ResolveExport(ctx context.Context, w io.Writer, opts *ResolverExportOptions) error
}

Resolver represents logic, such as a SQL query, that produces output data. Resolvers are used to evaluate API requests, alerts, reports, etc.

A resolver has two levels of configuration: static properties and dynamic arguments. For example, a SQL resolver has a static property for the SQL query and dynamic arguments for the query parameters. The static properties are usually declared in advance, such as in the YAML for a custom API, whereas the dynamic arguments are provided just prior to execution, such as in an API request.

type ResolverExportOptions added in v0.42.0 

type ResolverExportOptions struct {
	// Format is the format to export the result in.
	Format runtimev1.ExportFormat
	// PreWriteHook is a function that is called after the export has been prepared, but before the first bytes are output to the io.Writer.
	PreWriteHook func(filename string) error
}

ResolverExportOptions are the options passed to a resolver's ResolveExport method.

type ResolverInitializer added in v0.42.0 

type ResolverInitializer func(ctx context.Context, opts *ResolverOptions) (Resolver, error)

ResolverInitializer is a function that initializes a resolver.

type ResolverOptions added in v0.42.0 

type ResolverOptions struct {
	Runtime    *Runtime
	InstanceID string
	Properties map[string]any
	Args       map[string]any
	Claims     *SecurityClaims
	ForExport  bool
}

ResolverOptions are the options passed to a resolver initializer.

type ResolverResult added in v0.42.0 

type ResolverResult interface {
	// Close should be called to release resources
	Close() error
	// Schema is the schema for the Data
	Schema() *runtimev1.StructType
	// Next returns the next row of data. It returns io.EOF when there are no more rows.
	Next() (map[string]any, error)
	// MarshalJSON is a convenience method to serialize the result to JSON.
	MarshalJSON() ([]byte, error)
}

ResolverResult is the result of a resolver's execution.

func NewDriverResolverResult added in v0.48.0 

func NewDriverResolverResult(result *drivers.Result) ResolverResult

NewDriverResolverResult creates a ResolverResult from a drivers.Result.

func NewMapsResolverResult added in v0.48.0 

func NewMapsResolverResult(result []map[string]any, schema *runtimev1.StructType) ResolverResult

NewMapsResolverResult creates a ResolverResult from a slice of maps.

type Runtime 

type Runtime struct {
	Email *email.Client

	Logger *zap.Logger
	// contains filtered or unexported fields
}

func New 

func New(ctx context.Context, opts *Options, logger *zap.Logger, st *storage.Client, ac *activity.Client, emailClient *email.Client) (*Runtime, error)

func (*Runtime) AI added in v0.41.0 

func (r *Runtime) AI(ctx context.Context, instanceID string) (drivers.AIService, func(), error)

func (*Runtime) APIForName added in v0.42.0 

func (r *Runtime) APIForName(ctx context.Context, instanceID, name string) (*runtimev1.API, error)

APIForName returns the API with the given name for the given instance. It gives precedence to built-in APIs over project-specific dynamically created APIs.

func (*Runtime) AcquireHandle added in v0.32.0 

func (r *Runtime) AcquireHandle(ctx context.Context, instanceID, connector string) (drivers.Handle, func(), error)

AcquireHandle returns instance specific handle

func (*Runtime) AcquireSystemHandle added in v0.32.0 

func (r *Runtime) AcquireSystemHandle(ctx context.Context, connector string) (drivers.Handle, func(), error)

func (*Runtime) Admin added in v0.37.0 

func (r *Runtime) Admin(ctx context.Context, instanceID string) (drivers.AdminService, func(), error)

func (*Runtime) AllowHostAccess added in v0.27.0 

func (r *Runtime) AllowHostAccess() bool

func (*Runtime) Catalog added in v0.16.0 

func (r *Runtime) Catalog(ctx context.Context, instanceID string) (drivers.CatalogStore, func(), error)

func (*Runtime) Close added in v0.18.0 

func (r *Runtime) Close() error

func (*Runtime) ConnectorConfig added in v0.43.0 

func (r *Runtime) ConnectorConfig(ctx context.Context, instanceID, name string) (*ConnectorConfig, error)

func (*Runtime) Controller added in v0.33.1 

func (r *Runtime) Controller(ctx context.Context, instanceID string) (*Controller, error)

Controller returns the controller for the given instance. If the controller is currently initializing, the call will wait until the controller is ready. If the controller has closed with a fatal error, that error will be returned here until it's restarted.

func (*Runtime) CreateInstance 

func (r *Runtime) CreateInstance(ctx context.Context, inst *drivers.Instance) error

CreateInstance creates a new instance and starts a controller for it.

func (*Runtime) DataDir added in v0.47.0 

func (r *Runtime) DataDir(instanceID string, elem ...string) (string, error)

DataDir returns the path to a persistent data directory for the given instance. The directory is created if it doesn't exist. Storage usage in the returned directory will be reported in the instance's heartbeat events.

func (*Runtime) DeleteFile added in v0.16.0 

func (r *Runtime) DeleteFile(ctx context.Context, instanceID, path string, force bool) error

func (*Runtime) DeleteInstance added in v0.16.0 

func (r *Runtime) DeleteInstance(ctx context.Context, instanceID string) error

DeleteInstance deletes an instance and stops its controller.

func (*Runtime) EditInstance added in v0.23.0 

func (r *Runtime) EditInstance(ctx context.Context, inst *drivers.Instance, restartController bool) error

EditInstance edits an existing instance. If restartController is true, the instance's controller will be re-opened and all cached connections for the instance will be evicted. Until the controller and connections have been closed and re-opened, calls related to the instance may return transient errors.

func (*Runtime) GetFile added in v0.16.0 

func (r *Runtime) GetFile(ctx context.Context, instanceID, path string) (string, time.Time, error)

func (*Runtime) GetInstanceAttributes added in v0.32.0 

func (r *Runtime) GetInstanceAttributes(ctx context.Context, instanceID string) []attribute.KeyValue

GetInstanceAttributes fetches an instance and converts its annotations to attributes nil is returned if an error occurred or instance was not found

func (*Runtime) Health added in v0.47.0 

func (r *Runtime) Health(ctx context.Context) (*Health, error)

func (*Runtime) Instance added in v0.35.0 

func (r *Runtime) Instance(ctx context.Context, instanceID string) (*drivers.Instance, error)

Instance looks up an instance by ID. Instances are cached in-memory, so this is a cheap operation.

func (*Runtime) InstanceConfig added in v0.43.0 

func (r *Runtime) InstanceConfig(ctx context.Context, instanceID string) (drivers.InstanceConfig, error)

InstanceConfig returns the instance's dynamic configuration.

func (*Runtime) InstanceHealth added in v0.47.0 

func (r *Runtime) InstanceHealth(ctx context.Context, instanceID string) (*InstanceHealth, error)

func (*Runtime) InstanceLogger added in v0.40.0 

func (r *Runtime) InstanceLogger(ctx context.Context, instanceID string) (*zap.Logger, error)

InstanceLogger returns a logger scoped for the given instance. Logs emitted to the logger will also be available in the instance's log buffer.

func (*Runtime) InstanceLogs added in v0.40.0 

func (r *Runtime) InstanceLogs(ctx context.Context, instanceID string) (*logbuffer.Buffer, error)

InstanceLogs returns an in-memory buffer of recent logs related to the given instance.

func (*Runtime) Instances added in v0.35.0 

func (r *Runtime) Instances(ctx context.Context) ([]*drivers.Instance, error)

Instances returns all instances managed by the runtime.

func (*Runtime) ListFiles added in v0.16.0 

func (r *Runtime) ListFiles(ctx context.Context, instanceID, glob string) ([]drivers.DirEntry, error)

func (*Runtime) MakeDir added in v0.43.0 

func (r *Runtime) MakeDir(ctx context.Context, instanceID, path string) error

func (*Runtime) OLAP added in v0.16.0 

func (r *Runtime) OLAP(ctx context.Context, instanceID, connector string) (drivers.OLAPStore, func(), error)

OLAP returns a handle for an OLAP data store. The connector argument is optional. If not provided, the instance's default OLAP connector is used.

func (*Runtime) PutFile added in v0.16.0 

func (r *Runtime) PutFile(ctx context.Context, instanceID, path string, blob io.Reader, create, createOnly bool) error

func (*Runtime) Query added in v0.16.0 

func (r *Runtime) Query(ctx context.Context, instanceID string, query Query, priority int) error

func (*Runtime) RenameFile added in v0.16.0 

func (r *Runtime) RenameFile(ctx context.Context, instanceID, fromPath, toPath string) error

func (*Runtime) Repo added in v0.16.0 

func (r *Runtime) Repo(ctx context.Context, instanceID string) (drivers.RepoStore, func(), error)

func (*Runtime) Resolve added in v0.42.0 

func (r *Runtime) Resolve(ctx context.Context, opts *ResolveOptions) (ResolverResult, error)

Resolve resolves a query using the given options. The caller must call Close on the result when done consuming it.

func (*Runtime) ResolveSecurity added in v0.47.0 

func (r *Runtime) ResolveSecurity(instanceID string, claims *SecurityClaims, res *runtimev1.Resource) (*ResolvedSecurity, error)

func (*Runtime) TempDir added in v0.47.0 

func (r *Runtime) TempDir(instanceID string, elem ...string) (string, error)

TempDir returns the path to a temporary directory for the given instance. The directory is created if it doesn't exist. The TempDir is a fixed location. The caller is responsible for cleaning up after use. The TempDir may be cleared after restarts. Storage usage in the returned directory will be reported in the instance's heartbeat events.

func (*Runtime) UpdateInstanceConnector added in v0.46.0 

func (r *Runtime) UpdateInstanceConnector(ctx context.Context, instanceID, name string, connector *runtimev1.ConnectorSpec) error

UpdateInstanceConnector upserts or removes a connector from an instance If connector is nil, the connector is removed; otherwise, it is upserted

func (*Runtime) UpdateInstanceWithRillYAML added in v0.45.0 

func (r *Runtime) UpdateInstanceWithRillYAML(ctx context.Context, instanceID string, parser *rillv1.Parser, restartController bool) error

func (*Runtime) WaitUntilIdle added in v0.35.0 

func (r *Runtime) WaitUntilIdle(ctx context.Context, instanceID string, ignoreHidden bool) error

WaitUntilIdle waits until the instance's controller is idle (not reconciling any resources).

type SecurityClaims added in v0.47.0 

type SecurityClaims struct {
	// UserAttributes about the current user (or service account). Usually exposed through templating as {{ .user }}.
	UserAttributes map[string]any
	// AdditionalRules are optional security rules to apply *in addition* to the built-in rules and the rules defined on the requested resource.
	// These are currently leveraged by the admin service to enforce restrictions for magic auth tokens.
	AdditionalRules []*runtimev1.SecurityRule
	// SkipChecks enables completely skipping all security checks. Used in local development.
	SkipChecks bool
}

SecurityClaims represents contextual information for the enforcement of security rules.

func (*SecurityClaims) Admin added in v0.47.0 

func (c *SecurityClaims) Admin() bool

Admin is a convenience function for extracting an "admin" bool from the user attributes.

func (*SecurityClaims) MarshalJSON added in v0.47.0 

func (c *SecurityClaims) MarshalJSON() ([]byte, error)

MarshalJSON serializes the SecurityClaims to JSON. It serializes the AdditionalRules using protojson.

func (*SecurityClaims) UnmarshalJSON added in v0.47.0 

func (c *SecurityClaims) UnmarshalJSON(data []byte) error

UnmarshalJSON deserializes the SecurityClaims from JSON. It deserializes the AdditionalRules using protojson.

func (*SecurityClaims) UserID added in v0.47.0 

func (c *SecurityClaims) UserID() string

UserID is a convenience function for extracting an "id" string from the user attributes. Note that the ID may not correspond to an actual user, but could also be a service ID or similar.

type SubscribeCallback added in v0.33.1 

type SubscribeCallback func(e runtimev1.ResourceEvent, n *runtimev1.ResourceName, r *runtimev1.Resource)

SubscribeCallback is the callback type passed to Subscribe.

Source Files

View all Source files

Directories

Path Synopsis
compilers
rillv1
admin
athena
azure
bigquery
blob
clickhouse
druid
druid/druidsqldriver
druid/retrier
duckdb
duckdb/extensions
file
gcs
https
mock
mock/object_store
mysql
pinot
pinot/sqldriver
postgres
redshift
s3
salesforce
slack
snowflake
sqlite
pkg
activity
archive
arrayutil
bufferutil
conncache
container
ctxsync
dag
debugserver
duckdbsql
duration
Package duration copied as it is from github.com/senseyeio/duration
 Package duration copied as it is from github.com/senseyeio/duration
email
env
examples
expressionpb
fieldselectorpb
fileutil
formatter
gcputil
globutil
graceful
httputil
jsonval
logbuffer
logutil
mapstructureutil
metricssql
middleware
observability
openapiutil
pathutil
pbutil
priorityqueue
ratelimit
rduckdb
schedule
securetoken
singleflight
Package singleflight provides a duplicate function call suppression mechanism.
 Package singleflight provides a duplicate function call suppression mechanism.
sqlparse
timeutil
typepb
auth

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.