plugin

package
v5.10.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 13, 2024 License: Apache-2.0 Imports: 64 Imported by: 399

Documentation

Overview

Package plugin provides data structures and functions that enable a plugin to read data from an API and stream it into Postgres tables by way of Steampipe's [foreign data wrapper] (FDW).

Index

Constants

View Source 
const (
	EnvDiagnosticsLevel       = "STEAMPIPE_DIAGNOSTIC_LEVEL"
	EnvLegacyDiagnosticsLevel = "STEAMPIPE_DIAGNOSTICS_LEVEL"
	DiagnosticsAll            = "ALL"
	DiagnosticsNone           = "NONE"
)
View Source 
const (
	// Require values
	Required = "required"
	Optional = "optional"
	AnyOf    = "any_of"
)
View Source 
const (
	SchemaModeStatic  = "static"
	SchemaModeDynamic = "dynamic"
)
View Source 
const (
	UnrecognizedRemotePluginMessage       = "Unrecognized remote plugin message:"
	UnrecognizedRemotePluginMessageSuffix = "\nThis usually means"
	PluginStartupFailureMessage           = "Plugin startup failed: "
)
Serve creates and starts the GRPC server which serves the plugin,

passing callback functions to implement each of the plugin interface functions:

  • SetConnectionConfig

  • SetAllConnectionConfigs

  • UpdateConnectionConfigs

  • GetSchema

  • Execute

    It is called from the main function of the plugin.

Variables

View Source 
var ValidDiagnosticsLevels = map[string]struct{}{
	DiagnosticsAll:  {},
	DiagnosticsNone: {},
}

Functions

func DiagsToError 

func DiagsToError(prefix string, diags hcl.Diagnostics) error

DiagsToError converts error diags into a single error deprecated this is duplicated from pipe-fittings - only exists here until AWS plugin is updated to latest sdk so we can reference pipe-fittings

func DiagsToWarnings added in v5.1.0 

func DiagsToWarnings(diags hcl.Diagnostics) []string

DiagsToWarnings converts warning diags into a list of warning strings deprecated this is duplicated from pipe-fittings - only exists here until AWS plugin is updated to latest sdk so we can reference pipe-fittings

func GetFreeMemInterval 

func GetFreeMemInterval() int64

func GetMatrixItem 

func GetMatrixItem(ctx context.Context) map[string]interface{}

GetMatrixItem retrieves the matrix item from the context

func GetValidOperators added in v5.2.0 

func GetValidOperators() []string

GetValidOperators returns the list of operators which are valid in the context of KeyColumn

func IsCancelled 

func IsCancelled(ctx context.Context) bool

IsCancelled returns whether the context has been cancelled.

To use: 

for _, i := range items {
	d.StreamListItem(ctx, i)
	if plugin.IsCancelled(ctx) {
		return nil, nil
	}
}

Plugin examples:

func IsReservedColumnName added in v5.2.0 

func IsReservedColumnName(columnName string) bool

func Logger 

func Logger(ctx context.Context) hclog.Logger

Logger retrieves the hclog.Logger from the context.

Log at trace level: 

plugin.Logger(ctx).Trace("Code execution starts here")

Log at error level with single data: 

plugin.Logger(ctx).Error("hackernews_item.itemList", "query_error", err)

Log at info level with single data: 

plugin.Logger(ctx).Info("listGreeting", "number", i)

Log at warn level with multiple data: 

plugin.Logger(ctx).Warn("getDomain", "invalid_name", err, "query_response", resp)

func NewConnectionUpdateData added in v5.4.1 

func NewConnectionUpdateData() *connectionUpdateData

func RetryHydrate 

func RetryHydrate(ctx context.Context, d *QueryData, hydrateData *HydrateData, hydrate HydrateFunc, retryConfig *RetryConfig) (hydrateResult interface{}, err error)

RetryHydrate function invokes the hydrate function with retryable errors and retries the function until the maximum attempts before throwing error

func Serve 

func Serve(opts *ServeOpts)

func Server added in v5.8.0 

func Server(opts *ServeOpts) *grpc.PluginServer

func SqlLike added in v5.10.0 

func SqlLike(input, pattern string, caseSensitive bool) bool

SqlLike simulates SQL LIKE pattern matching using fnmatch, with an option for case sensitivity.

func ValidateDiagnosticsEnvVar added in v5.6.0 

func ValidateDiagnosticsEnvVar() error

func ValidateSchemaMode added in v5.2.0 

func ValidateSchemaMode(m string) error

func WrapHydrate 

func WrapHydrate(hydrate namedHydrateFunc, ignoreConfig *IgnoreConfig) namedHydrateFunc

WrapHydrate is a higher order function which returns a HydrateFunc that handles Ignorable errors.

Types

type Column 

type Column struct {
	// column name
	Name string
	// column type
	Type proto.ColumnType
	// column description
	Description string
	// explicitly specify the function which populates this data
	// - this is only needed if any of the default hydrate functions will NOT return this column
	Hydrate HydrateFunc
	// the default column value
	Default interface{}
	//  a list of transforms to generate the column value
	Transform *transform.ColumnTransforms
	// contains filtered or unexported fields
}

Column defines a column of a table.

A column may be populated by a List or Get call. It may alternatively define its own HydrateFunc that makes an additional API call for each row.

A column may transform the data it receives using one or more transform functions.

To define a column populated by a List or Get call: 

func itemCols() []*plugin.Column {
    return []*plugin.Column{
        {Name: "id", Type: proto.ColumnType_INT, Description: "The item's unique id."},
    }
}

To define a column populated by a HydrateFunc: 

Columns: awsColumns([]*plugin.Column{
	{
		Name:        "permissions_boundary_arn",
		Description: "The ARN of the policy used to set the permissions boundary for the user.",
		Type:        proto.ColumnType_STRING,
		Hydrate:     getAwsIamUserData,
	},
}

To define columns that transform the data: 

Columns: awsColumns([]*plugin.Column{
	{
		Name:        "mfa_enabled",
		Description: "The MFA status of the user.",
		Type:        proto.ColumnType_BOOL,
		Hydrate:     getAwsIamUserMfaDevices,
		Transform:   transform.From(handleEmptyUserMfaStatus),
	},
	{
		Name:        "login_profile",
		Description: "Contains the user name and password create date for a user.",
		Type:        proto.ColumnType_JSON,
		Hydrate:     getAwsIamUserLoginProfile,
		Transform:   transform.FromValue(),
	},
	...
}

Examples:

func (*Column) ToColumnValue added in v5.7.0 

func (c *Column) ToColumnValue(val any) (*proto.Column, error)

ToColumnValue converts a value of unknown type to a valid protobuf column value.type

type Connection 

type Connection struct {
	Name string
	// the connection config
	// NOTE: we always pass and store connection config BY VALUE
	Config any
}

Connection is a struct which is used to store connection config.

The connection config is parsed and stored as plugin.Plugin.Connection.

The connection may be retrieved by the plugin by calling: plugin.QueryData.Connection

Plugin examples:

type ConnectionConfigInstanceFunc 

type ConnectionConfigInstanceFunc func() any

ConnectionConfigInstanceFunc is a function type which returns 'any'.

It is used to implement plugin.ConnectionConfigSchema.NewInstance.

type ConnectionConfigSchema 

type ConnectionConfigSchema struct {
	Schema map[string]*schema.Attribute
	// function which returns an instance of a connection config struct
	NewInstance ConnectionConfigInstanceFunc
}

ConnectionConfigSchema is a struct that defines custom arguments in the plugin spc file that are passed to the plugin as plugin.Connection.Config.

A plugin that uses custom connection config must set plugin.Plugin.ConnectionConfigSchema.

Usage: 

p := &plugin.Plugin{
	Name: "steampipe-plugin-hackernews",
	ConnectionConfigSchema: &plugin.ConnectionConfigSchema{
		NewInstance: ConfigInstance,
		Schema:      ConfigSchema,
	},
	...
}

var ConfigSchema = map[string]*schema.Attribute{
	"max_items": {
		Type: schema.TypeInt,
	},
}

func ConfigInstance() any {
	return &hackernewsConfig{}
}

Plugin examples:

func (*ConnectionConfigSchema) Validate 

func (c *ConnectionConfigSchema) Validate() []string

Validate validates the connection config

type ConnectionData 

type ConnectionData struct {
	// map of all the tables in the connection schema, keyed by the table name
	TableMap map[string]*Table
	// Connection struct contains the _parsed_ connection config object as an interface
	Connection *Connection
	// the connection schema
	Schema *grpc.PluginSchema
	// connection specific filewatcher to watch for the changes in files (if needed)
	Watcher *filewatcher.FileWatcher
	Plugin  *Plugin

	// map of aggregated tables, keyed by connection name
	// NOTE: only set for aggregator connections
	AggregatedTablesByConnection map[string]map[string]*Table
	// contains filtered or unexported fields
}

ConnectionData is the data stored by the plugin which is connection dependent.

func NewConnectionData added in v5.2.0 

func NewConnectionData(c *Connection, p *Plugin, config *proto.ConnectionConfig) *ConnectionData

type ConnectionKeyColumn added in v5.10.0 

type ConnectionKeyColumn struct {
	Hydrate HydrateFunc
	Name    string
}

ConnectionKeyColumn is a struct that defines a column that has a value which maps 1-1 to a Steampipe connection and so can be used to filter connections when executing an aggregator query

These columns are treated as (optional) KeyColumns. This means they are taken into account in the query planning - so it is more likely that the scans will be planned to use these as quals where possible.

When executing an aggregator query, if one of these columns is used as a qual, the execution code will filter the list of connections by comparing the qual value with the column value returned by the ConnectionKeyColumn Hydrate function.

The Hydrate function is a function that returns the value fo the column for a given connection.

type CreatePlugin 

type CreatePlugin func(context.Context, string) (*Plugin, error)

type DefaultConcurrencyConfig 

type DefaultConcurrencyConfig struct {
	// sets how many HydrateFunc calls can run concurrently in total
	TotalMaxConcurrency int
	// sets the default for how many calls to each HydrateFunc can run concurrently
	DefaultMaxConcurrency int
}

Deprecated

type ErrorPredicate 

type ErrorPredicate func(error) bool

ErrorPredicate is a function type which accepts error as an input and returns a boolean value.

type ErrorPredicateWithContext 

type ErrorPredicateWithContext func(context.Context, *QueryData, *HydrateData, error) bool

ErrorPredicateWithContext is a function type which accepts context, query data, hydrate data and error as an input and returns a boolean value.

Plugin examples:

type ForceImport 

type ForceImport string

ForceImport is a mechanism to ensure godoc can reference all required packages

type GetConfig 

type GetConfig struct {
	// the hydrate function which is called first when performing a 'get' call.
	// if this returns 'not found', no further hydrate functions are called
	Hydrate HydrateFunc
	// key or keys which are used to uniquely identify rows - used to determine whether  a query is a 'get' call
	KeyColumns KeyColumnSlice
	// a function which will return whenther to ignore a given error
	IgnoreConfig *IgnoreConfig
	// a function which will return whenther to retry the call if an error is returned
	RetryConfig *RetryConfig
	Tags        map[string]string

	// Deprecated: use IgnoreConfig
	ShouldIgnoreError ErrorPredicate
	MaxConcurrency    int
	// contains filtered or unexported fields
}

A GetConfig defines how to get a single row of a table:

A GetConfig with KeyColumns: 

Get: &plugin.GetConfig{
	KeyColumns: plugin.SingleColumn("id"),
	Hydrate:    getItem,
}

A GetConfig with IgnoreConfig: 

Get: &plugin.GetConfig{
	KeyColumns: 	plugin.SingleColumn("id"),
	Hydrate:    	getItem,
	IgnoreConfig:   &plugin.IgnoreConfig{ShouldIgnoreErrorFunc: shouldIgnoreError},
}

A GetConfig with RetryConfig: 

Get: &plugin.GetConfig{
	KeyColumns: 	plugin.SingleColumn("id"),
	Hydrate:    	getItem,
	RetryConfig:    &plugin.RetryConfig{
		ShouldRetryErrorFunc: shouldRetryError,
	},
}

A GetConfig with all fields specified: 

Get: &plugin.GetConfig{
	KeyColumns:     plugin.SingleColumn("id"),
	Hydrate:        getItem,
	RetryConfig:    &plugin.RetryConfig{
		ShouldRetryErrorFunc: shouldRetryError,
	},
	IgnoreConfig:   &plugin.IgnoreConfig{ShouldIgnoreErrorFunc: shouldIgnoreError},
	MaxConcurrency: 50,
}

Plugin examples:

func (*GetConfig) Validate 

func (c *GetConfig) Validate(table *Table) []string

type HydrateConfig 

type HydrateConfig struct {
	Func HydrateFunc
	// a function which will return whenther to ignore a given error
	IgnoreConfig *IgnoreConfig
	// a function which will return whenther to retry the call if an error is returned
	RetryConfig *RetryConfig
	Depends     []HydrateFunc

	// tags - used to resolve the rate limiter for this hydrate call
	// for example:
	// "service": "s3"
	//
	// when resolving a rate limiter for a hydrate call, a map of scope values is automatically populated:
	// - the table and  connection name
	// - values specified in the hydrate config
	// - matrix quals (with values as string)
	// this map is then used to find a rate limiter
	Tags map[string]string

	MaxConcurrency int

	// Deprecated: use IgnoreConfig
	ShouldIgnoreError ErrorPredicate
	// contains filtered or unexported fields
}

HydrateConfig defines how to run a HydrateFunc:

It's not valid to have a HydrateConfig for a HydrateFunc that is specified in a GetConfig.

A HydrateConfig with IgnoreConfig: 

HydrateConfig: []plugin.HydrateConfig{
	{
	Func:           getRetentionPeriod,
	IgnoreConfig:   &plugin.IgnoreConfig{ShouldIgnoreErrorFunc: shouldIgnoreError},
	}

A HydrateConfig with MaxConcurrency: 

HydrateConfig: []plugin.HydrateConfig{
	{
	Func:           getRetentionPeriod,
	MaxConcurrency: 50,
	IgnoreConfig:   &plugin.IgnoreConfig{ShouldIgnoreErrorFunc: shouldIgnoreError},
	}

A HydrateConfig with all fields specified: 

HydrateConfig: []plugin.HydrateConfig{
	{
	Func:           getRetentionPeriod,
	MaxConcurrency: 50,
	IgnoreConfig:   &plugin.IgnoreConfig{ShouldIgnoreErrorFunc: shouldIgnoreError},
	RetryConfig:    &plugin.RetryConfig{
		ShouldRetryErrorFunc: shouldRetryError,
	}

Steampipe parallelizes hydrate functions as much as possible. Sometimes, however, one hydrate function requires the output from another. 

return &plugin.Table{
		Name: "hydrate_columns_dependency",
		List: &plugin.ListConfig{
			Hydrate: hydrateList,
		},
		HydrateConfig: []plugin.HydrateConfig{
			{
				Func:    hydrate2,
				Depends: []plugin.HydrateFunc{hydrate1},
			},
		},
		Columns: []*plugin.Column{
			{Name: "id", Type: proto.ColumnType_INT},
			{Name: "hydrate_column_1", Type: proto.ColumnType_STRING, Hydrate: hydrate1},
			{Name: "hydrate_column_2", Type: proto.ColumnType_STRING, Hydrate: hydrate2},
		},
	}

Here, hydrate function hydrate2 is dependent on hydrate1. This means hydrate2 will not execute until hydrate1 has completed and the results are available. hydrate2 can refer to the results from hydrate1 as follows: 

func hydrate2(ctx context.Context, d *plugin.QueryData, h *plugin.HydrateData) (interface{}, error) {
		// NOTE: in this case we know the output of hydrate1 is map[string]interface{} so we cast it accordingly.
		// the data should be cast to th appropriate type
	hydrate1Results := h.HydrateResults["hydrate1"].(map[string]interface{})
.....
}

Note that:

  • Multiple dependencies are supported.
  • Circular dependencies will be detected and cause a validation failure.
  • The Get and List hydrate functions ***CANNOT*** have dependencies.

Examples:

func (*HydrateConfig) String 

func (c *HydrateConfig) String() string

type HydrateData 

type HydrateData struct {
	// if there was a parent-child list call, store the parent list item
	ParentItem     interface{}
	Item           interface{}
	HydrateResults map[string]interface{}
}

HydrateData contains the input data passed to every hydrate function

type HydrateDependencies deprecated 

type HydrateDependencies struct {
	Func    HydrateFunc
	Depends []HydrateFunc
}

Deprecated: Use HydrateConfig instead.

type HydrateFunc 

type HydrateFunc func(context.Context, *QueryData, *HydrateData) (interface{}, error)

HydrateFunc is a function that gathers data to build table rows. Typically this would make an API call and return the raw API output.

List and Get are special hydrate functions.

  • List returns data for all rows. Almost all tables will have a List function.

  • Get returns data for a single row. In order to filter as cheaply as possible a Get function should be implemented if the API supports fetching single items by key.

A column may require data not returned by the List or Get calls and an additional API call will be required. A HydrateFunc that wraps this API call can be specified in the Column definition.

You could do this the hard way by looping through the List API results and enriching each item by making an additional API call. However the SDK does all this for you.

func (HydrateFunc) Memoize added in v5.2.0 

func (f HydrateFunc) Memoize(opts ...MemoizeOption) HydrateFunc

Memoize ensures the HydrateFunc results are saved in the connection.ConnectionCache.

Use it to reduce the number of API calls if the HydrateFunc is used by multiple tables.

NOTE: this should only be used to memoize a function which will be manually invoked and requires caching It should NOT be used to memoize a hydrate function being passed to a table definition.

func (HydrateFunc) WithCache 

func (f HydrateFunc) WithCache(args ...HydrateFunc) HydrateFunc

WithCache deprecated: use Memoize

type IgnoreConfig 

type IgnoreConfig struct {
	ShouldIgnoreErrorFunc ErrorPredicateWithContext
	// Deprecated: used ShouldIgnoreErrorFunc
	ShouldIgnoreError ErrorPredicate
}

IgnoreConfig defines errors to ignore. When that happens, an empty row is returned.

If a HydrateFunc has specific errors that should not block query execution, set plugin.GetConfig.IgnoreConfig, plugin.ListConfig.IgnoreConfig or plugin.HydrateConfig.IgnoreConfig.

For errors common to many HydrateFuncs, you can define a default IgnoreConfig by setting plugin.DefaultGetConfig.

Ignore errors from a HydrateFunc that has a GetConfig: 

Get: &plugin.GetConfig{
	IgnoreConfig: &plugin.IgnoreConfig{
		ShouldIgnoreErrorFunc: isIgnorableErrorPredicate([]string{"Request_ResourceNotFound", "Invalid object identifier"}),
	},
	...
},

Ignore errors from a HydrateFunc that has a ListConfig: 

List: &plugin.ListConfig{
	IgnoreConfig: &plugin.IgnoreConfig{
		ShouldIgnoreErrorFunc: isIgnorableErrorPredicate([]string{"Request_UnsupportedQuery"}),
	},
	...
},

Ignore errors from a HydrateFunc that has a HydrateConfig: 

HydrateConfig: []plugin.HydrateConfig{
	IgnoreConfig: &plugin.IgnoreConfig{
		ShouldIgnoreErrorFunc: isIgnorableErrorPredicate([]string{"Request_UnsupportedQuery"}),
	},
	...
},

Ignore errors that may occur in many HydrateFuncs: 

DefaultIgnoreConfig: &plugin.DefaultIgnoreConfig{
	IgnoreConfig: &plugin.IgnoreConfig{
		ShouldIgnoreErrorFunc: isIgnorableErrorPredicate([]string{"Request_ResourceNotFound"}),
	},
	...
},

Plugin examples:

func (*IgnoreConfig) DefaultTo 

func (c *IgnoreConfig) DefaultTo(other *IgnoreConfig)

func (*IgnoreConfig) String 

func (c *IgnoreConfig) String() string

type KeyColumn 

type KeyColumn struct {
	Name       string
	Operators  []string
	Require    string
	CacheMatch string
}

KeyColumn is a struct representing the definition of a column used to filter and Get and List calls.

At least one key column must be defined for a Get call. They are optional for List calls.

Operators

This property specifies the accepted operators (from a possible set: "=", "<>", "<", "<=", ">", ">=", "~~", "!~~", "~~*", "!~~*", "~", "~*", "!~", "!~*", "@>", "<@", "?", "?|", "?&", "@?", "@@")

Require

This property determines whether the column is required or optional. Possible values:

"required"

The key column must be provided as a query qualifier (i.e. in a where clause in the query).

"optional"

The key column is optional but if provided it will be used to filter the results.

"any_of"

Any one of the given columns must be provided.

CacheMatch

This property determines the logic used by the query results cache to determine whether a cached value matches a given query. Possible values:

"subset" [default value]

A cached item is considered a match (i.e. a cache hit) if the qual for the query is a subset of the quals for the cached item.

For example, is the cached qual is "val < 100", and the query qual is "val < 50", this would be considered a qual subset so would lead to a cache match.

"exact"

A cached item is considered a match ONLY if the qual for the cached item is the same as as the qual for the query.

This is used for columns which are only populated if the qual for that column is passed. A common pattern is to provide a "filter" column, which is populated using the qual value provided. This filter value is used when making the API call the fetch the data. If no filter qual is provided, then the filter column returned by the plugin is empty.

This breaks the subset logic as if there is a cached data with no qual for the filter column, this cached data would contain null values for the filter column. This data would be considered a superset of the data returned from a query which provides a filter qual, which is incorrect as the data returned if a filter qual is passed would include a non null filter column.

The solution is to set CacheMatch="exact".

Plugin examples:

func (*KeyColumn) InitialiseOperators 

func (k *KeyColumn) InitialiseOperators()

InitialiseOperators adds a default '=' operator is no operators are set, and converts "!=" to "<>".

func (*KeyColumn) SingleEqualsQual 

func (k *KeyColumn) SingleEqualsQual() bool

SingleEqualsQual returns whether this key column has a single = operator.

func (*KeyColumn) String 

func (k *KeyColumn) String() string

func (*KeyColumn) ToProtobuf 

func (k *KeyColumn) ToProtobuf() *proto.KeyColumn

ToProtobuf converts the KeyColumn to a protobuf object.

func (*KeyColumn) Validate 

func (k *KeyColumn) Validate() []string

Validate ensures 'Operators' and 'Require' are valid. It returns a list of validation error strings.

type KeyColumnEqualsQualMap 

type KeyColumnEqualsQualMap map[string]*proto.QualValue

KeyColumnEqualsQualMap is a map of column name to qual value. It is used to represent a set of columns which have a single 'equals' qual.

func (KeyColumnEqualsQualMap) GetListQualValues 

func (m KeyColumnEqualsQualMap) GetListQualValues() map[string]*proto.QualValueList

GetListQualValues returns a map of all qual values with a List value

func (KeyColumnEqualsQualMap) String 

func (m KeyColumnEqualsQualMap) String() string

type KeyColumnQualMap 

type KeyColumnQualMap map[string]*KeyColumnQuals

KeyColumnQualMap is a map of KeyColumnQuals keyed by column name

func NewKeyColumnQualValueMap 

func NewKeyColumnQualValueMap(qualMap map[string]*proto.Quals, keyColumns KeyColumnSlice) KeyColumnQualMap

NewKeyColumnQualValueMap creates a KeyColumnQualMap from a qual map and a KeyColumnSlice

func (KeyColumnQualMap) GetListQualValues 

func (m KeyColumnQualMap) GetListQualValues() quals.QualSlice

GetListQualValues returns a slice of any quals we have which have a list value

func (KeyColumnQualMap) GetUnsatisfiedKeyColumns 

func (m KeyColumnQualMap) GetUnsatisfiedKeyColumns(columns KeyColumnSlice) KeyColumnSlice

func (KeyColumnQualMap) String 

func (m KeyColumnQualMap) String() string

func (KeyColumnQualMap) ToEqualsQualValueMap 

func (m KeyColumnQualMap) ToEqualsQualValueMap() map[string]*proto.QualValue

ToEqualsQualValueMap converts a KeyColumnQualMap to a column-qual value map, including only the

func (KeyColumnQualMap) ToProtoQualMap 

func (m KeyColumnQualMap) ToProtoQualMap() map[string]*proto.Quals

ToQualMap converts the map into a map of column to *proto.Quals used for cache indexes

func (KeyColumnQualMap) ToQualMap 

func (m KeyColumnQualMap) ToQualMap() map[string]quals.QualSlice

ToQualMap converts the map into a simpler map of column to []Quals this is used in the TransformData (needed to avoid the transform package needing to reference plugin)

type KeyColumnQuals 

type KeyColumnQuals struct {
	Name  string
	Quals quals.QualSlice
}

KeyColumnQuals defines all qualifiers for a column.

Use it in a table definition, by way of the plugin.QueryData object.

The query writer must specify the qualifiers, in a WHERE or a JOIN clause, in order to limit the number of API calls that Steampipe makes to satisfy the query. 

func listUser(ctx context.Context, d *plugin.QueryData, h *plugin.HydrateData) (interface{}, error) {
	var item User
	var id string

	if h.Item != nil {
		user := h.Item.(*User)
		id = user.ID
	} else {
		quals := d.KeyColumnQuals
		id = quals["id"].GetStringValue()
	}
	...
}

Examples:

func (KeyColumnQuals) SatisfiesKeyColumn 

func (k KeyColumnQuals) SatisfiesKeyColumn(keyColumn *KeyColumn) bool

func (KeyColumnQuals) SingleEqualsQual 

func (k KeyColumnQuals) SingleEqualsQual() bool

type KeyColumnSlice 

type KeyColumnSlice []*KeyColumn

KeyColumnSlice is an array of KeyColumn

func AllColumns 

func AllColumns(columns []string) KeyColumnSlice

AllColumns creates a KeyColumnSlice based on a slice of column names, each with a single equals operator and Require=Required

func AnyColumn 

func AnyColumn(columns []string) KeyColumnSlice

AnyColumn Columns creates a KeyColumnSlice based on a slice of column names, each with a single equals operator and Require=AnyOf

func NewEqualsKeyColumnSlice 

func NewEqualsKeyColumnSlice(columns []string, require string) KeyColumnSlice

NewEqualsKeyColumnSlice creates a KeyColumnSlice from a list of column names, each with a single equals operator

func OptionalColumns 

func OptionalColumns(columns []string) KeyColumnSlice

OptionalColumns Columns creates a KeyColumnSlice based on a slice of column names, with a single equals operator and Require=Optional

func SingleColumn 

func SingleColumn(column string) KeyColumnSlice

SingleColumn creates a KeyColumnSlice based on a column name The created slice has a single KeyColumn using a single equals operator and Require=Required

func (KeyColumnSlice) AllEquals 

func (k KeyColumnSlice) AllEquals() bool

AllEquals returns whether all KeyColumns only use equals operators

func (KeyColumnSlice) Find 

func (k KeyColumnSlice) Find(name string) *KeyColumn

Find looks for a key column with the given name and returns it if found

func (KeyColumnSlice) IsAnyOf 

func (k KeyColumnSlice) IsAnyOf() bool

IsAnyOf returns whether all key columns have Require == AnyOf

func (KeyColumnSlice) String 

func (k KeyColumnSlice) String() string

func (KeyColumnSlice) StringSlice 

func (k KeyColumnSlice) StringSlice() []string

StringSlice converts a KeyColumnSlice to a slice of strings

func (KeyColumnSlice) ToProtobuf 

func (k KeyColumnSlice) ToProtobuf() []*proto.KeyColumn

ToProtobuf converts the KeyColumnSlice to a slice of proto.KeyColumn

func (KeyColumnSlice) Validate 

func (k KeyColumnSlice) Validate() []string

Validate validates all child columns

type ListConfig 

type ListConfig struct {
	// the list function, this should stream the list results back using the QueryData object and return nil
	Hydrate HydrateFunc
	// key or keys which are used to uniquely identify rows - used to optimise the list call
	KeyColumns KeyColumnSlice
	// the parent list function - if we list items with a parent-child relationship, this will list the parent items
	ParentHydrate HydrateFunc
	// a function which will return whenther to ignore a given error
	IgnoreConfig *IgnoreConfig
	// a function which will return whenther to retry the call if an error is returned
	RetryConfig *RetryConfig

	Tags       map[string]string
	ParentTags map[string]string

	// Deprecated: Use IgnoreConfig
	ShouldIgnoreError ErrorPredicate
	// contains filtered or unexported fields
}

ListConfig defines how to return all rows in the table:

To define a table's List function: 

func tableHackernewsItem(ctx context.Context) *plugin.Table {
	return &plugin.Table{
		Name:        "hackernews_item",
		Description: "This table includes the most recent items posted to Hacker News.",
		List: &plugin.ListConfig{
			Hydrate: itemList,
		},
		...
	}
}

Examples:

func (*ListConfig) Validate 

func (c *ListConfig) Validate(table *Table) []string

type MatrixItemFunc 

type MatrixItemFunc func(context.Context, *Connection) []map[string]interface{}

Deprecated use MatrixItemMapFunc

type MatrixItemMapFunc 

type MatrixItemMapFunc func(context.Context, *QueryData) []map[string]interface{}

MatrixItemMapFunc is a callback function which may be implemented by the plugin to provide a map of matrix_items to execute the query with.

matrix_items are a powerful way of executing the same query multiple times in parallel for a set of parameters.

Plugin examples:

type MemoizeConfiguration added in v5.2.0 

type MemoizeConfiguration struct {
	GetCacheKeyFunc HydrateFunc
	Ttl             time.Duration
}

type MemoizeOption added in v5.2.0 

type MemoizeOption = func(config *MemoizeConfiguration)

type NewPluginOptions 

type NewPluginOptions struct {
	ConnectionName   string
	ConnectionConfig string
}

type Plugin 

type Plugin struct {
	Name   string
	Logger hclog.Logger
	// TableMap is a map of all the tables in the plugin, keyed by the table name
	// NOTE: it must be NULL for plugins with dynamic schema
	TableMap         map[string]*Table
	TableMapFunc     TableMapFunc
	DefaultTransform *transform.ColumnTransforms

	// ConnectionKeyColumns ia a list of [ConnectionKeyColumn] structs which define columns whose value map
	// directly to a Steampipe connection. These are used to filter connections when executing an aggregator query.
	// See [ConnectionKeyColumn] for more details.
	ConnectionKeyColumns []ConnectionKeyColumn

	// deprecated - use RateLimiters to control concurrency
	DefaultConcurrency  *DefaultConcurrencyConfig
	DefaultRetryConfig  *RetryConfig
	DefaultIgnoreConfig *IgnoreConfig

	// rate limiter definitions - these are (optionally) defined by the plugin author
	// and do NOT include any config overrides
	RateLimiters []*rate_limiter.Definition

	// deprecated - use DefaultRetryConfig and DefaultIgnoreConfig
	DefaultGetConfig *GetConfig
	// deprecated - use DefaultIgnoreConfig
	DefaultShouldIgnoreError ErrorPredicate
	// every table must implement these columns
	RequiredColumns        []*Column
	ConnectionConfigSchema *ConnectionConfigSchema
	// ConnectionConfigChangedFunc is a callback function which is called from UpdateConnectionConfigs
	// when any connection configs have changed
	ConnectionConfigChangedFunc func(ctx context.Context, p *Plugin, old, new *Connection) error

	// map of connection data (schema, config)
	// keyed by connection name
	ConnectionMap map[string]*ConnectionData

	// is this a static or dynamic schema
	SchemaMode string
	// callback function which is called when any watched source file(s) gets changed
	WatchedFileChangedFunc func(ctx context.Context, p *Plugin, connection *Connection, events []fsnotify.Event)

	// config for hydrate functions which are used across multiple tables
	HydrateConfig []HydrateConfig
	// contains filtered or unexported fields
}

Plugin is the primary struct that defines a Steampipe GRPC plugin.

Set plugin name using plugin.Plugin.Name.

The tables provided by the plugin are specified by setting either plugin.Plugin.TableMap or plugin.Plugin.TableMapFunc:

If the plugin uses custom connection config, it must define a plugin.ConnectionConfigSchema.

Various default behaviours can be defined:

Required columns can be specified by setting plugin.Plugin.RequiredColumns.

Plugin examples:

func (*Plugin) ClearConnectionCache 

func (p *Plugin) ClearConnectionCache(ctx context.Context, connectionName string) error

ClearConnectionCache clears the connection cache for the given connection.

func (*Plugin) ClearQueryCache 

func (p *Plugin) ClearQueryCache(ctx context.Context, connectionName string) error

ClearQueryCache clears the query cache for the given connection.

func (*Plugin) ConnectionSchemaChanged 

func (p *Plugin) ConnectionSchemaChanged(connection *Connection) error

ConnectionSchemaChanged sends a message to the plugin-manager that the schema of this plugin has changed

This should be called from the plugin implementation of plugin.Plugin.WatchedFileChangedFunc if a change in watched source files has changed the plugin schema.

type PluginFunc 

type PluginFunc func(context.Context) *Plugin

type QueryColumn 

type QueryColumn struct {
	*Column
	// contains filtered or unexported fields
}

QueryColumn is struct storing column name and resolved hydrate name (including List/Get call) this is used in the query data when the hydrate function has been resolved

func NewQueryColumn 

func NewQueryColumn(column *Column, hydrateName string) *QueryColumn

type QueryContext 

type QueryContext struct {
	Columns      []string
	UnsafeQuals  map[string]*proto.Quals
	Limit        *int64
	CacheEnabled bool
	CacheTTL     int64
}

QueryContext contains important query properties:

  • The columns requested.

  • All quals specified (not just key column quals).

  • The limit.

  • Cache properties.

func NewQueryContext 

func NewQueryContext(p *proto.QueryContext, limit *proto.NullableInt, cacheEnabled bool, cacheTTL int64, table *Table) *QueryContext

NewQueryContext maps from a proto.QueryContext to a plugin.QueryContext.

func (*QueryContext) GetLimit 

func (q *QueryContext) GetLimit() int64

GetLimit converts plugin.QueryContext.Limit from a *int64 to an int64 (where -1 means no limit).

type QueryData 

type QueryData struct {

	// The table this query is associated with
	Table *Table

	// a map of key columns which have a _single equals qual_
	// this is intended for use by Get calls only
	// key: column name, value: QualValue
	//  (this will also be populated for a list call if list key columns are specified -
	//  however this usage is deprecated and provided for legacy reasons only)
	EqualsQuals KeyColumnEqualsQualMap
	// a map of all KeyColumnQuals which were specified in the query. ([key_columns]
	Quals KeyColumnQualMap
	// is this a 'get' or a 'list' call
	FetchType fetchType
	// query context data passed from postgres - this includes the requested columns and the quals
	QueryContext *QueryContext
	// connection details - the connection name and any config declared in the connection config file
	Connection *Connection
	// Matrix is an array of parameter maps (MatrixItems)
	// the list/get calls with be executed for each element of this array
	Matrix []map[string]interface{}

	// object to handle caching of connection specific data
	// deprecated use ConnectionCache
	ConnectionManager *connection_manager.Manager
	ConnectionCache   *connection_manager.ConnectionCache

	// streaming funcs
	StreamListItem func(context.Context, ...interface{})

	// deprecated - plugins should no longer call StreamLeafListItem directly and should just call StreamListItem
	// event for the child list of a parent child list call
	StreamLeafListItem func(context.Context, ...interface{})
	// contains filtered or unexported fields
}

QueryData is passed to all HydrateFunc calls. It contains all required information about the executing query:

Plugin examples:

func (*QueryData) EqualsQualString 

func (d *QueryData) EqualsQualString(key string) string

EqualsQualString looks for the specified key column quals and if it exists, return the value as a string

func (*QueryData) GetSourceFiles 

func (d *QueryData) GetSourceFiles(source string) ([]string, error)

GetSourceFiles accept a source path downloads files if necessary, and returns a list of local file paths

func (*QueryData) RowsRemaining 

func (d *QueryData) RowsRemaining(ctx context.Context) int64

RowsRemaining returns how many rows are required to complete the query

  • if no limit has been parsed from the query, this will return math.MaxInt32 (meaning an unknown number of rows remain)
  • if there is a limit, it will return the number of rows required to reach this limit
  • if the context has been cancelled, it will return zero

func (*QueryData) WaitForListRateLimit added in v5.6.0 

func (d *QueryData) WaitForListRateLimit(ctx context.Context)

type RetryConfig 

type RetryConfig struct {
	ShouldRetryErrorFunc ErrorPredicateWithContext
	// deprecated use ShouldRetryErrorFunc
	ShouldRetryError ErrorPredicate

	// Maximum number of retry operation to be performed. Default set to 10.
	MaxAttempts int64
	// Algorithm for the backoff. Supported values: Fibonacci, Exponential, and Constant. Default set to Fibonacci.
	BackoffAlgorithm string
	// Starting interval. Default set to 100ms.
	RetryInterval int64
	// Set a maximum on the duration (in ms) returned from the next backoff.
	CappedDuration int64
	// Sets a maximum on the total amount of time (in ms) a backoff should execute.
	MaxDuration int64
}

RetryConfig retries HydrateFunc errors.

If a HydrateFunc returns an error in the first attempt but resolves itself in a future attempt, for instance API rate limit or throttling errors, set plugin.GetConfig.RetryConfig, plugin.ListConfig.RetryConfig or plugin.HydrateConfig.RetryConfig.

For errors common to many HydrateFuncs, you can define a default RetryConfig by setting plugin.DefaultGetConfig.

Retry errors from a HydrateFunc that has a GetConfig: 

Get: &plugin.GetConfig{
	RetryConfig: &plugin.RetryConfig{
		ShouldRetryError: shouldRetryError,
	},
	...
},

Retry errors from a HydrateFunc that has a ListConfig: 

List: &plugin.ListConfig{
	RetryConfig: &plugin.RetryConfig{
		ShouldRetryError: shouldRetryError,
	},
	...
},

Retry errors from a HydrateFunc that has a HydrateConfig: 

HydrateConfig: []plugin.HydrateConfig{
	RetryConfig: &plugin.RetryConfig{
		ShouldRetryError: shouldRetryError,
	},
	...
},

Retry errors that may occur in many HydrateFuncs: 

DefaultIgnoreConfig: &plugin.DefaultIgnoreConfig{
	RetryConfig: &plugin.RetryConfig{
		ShouldRetryError: shouldRetryError,
	},
	...
},

func (*RetryConfig) DefaultTo 

func (c *RetryConfig) DefaultTo(other *RetryConfig)

func (*RetryConfig) GetListRetryConfig 

func (c *RetryConfig) GetListRetryConfig() *RetryConfig

GetListRetryConfig wraps the ShouldRetry function with an additional check of the rows streamed (as we cannot retry errors in the list hydrate function after streaming has started)

func (*RetryConfig) String 

func (c *RetryConfig) String() string

type ServeOpts 

type ServeOpts struct {
	PluginName string
	PluginFunc PluginFunc
}

ServeOpts are the configurations to serve a plugin.

type SteampipeMetadata added in v5.8.0 

type SteampipeMetadata struct {
	SdkVersion string `json:"sdk_version"`
}

type Table 

type Table struct {
	Name string
	// table description
	Description string
	// column definitions
	Columns []*Column
	// the function used to list table rows
	List *ListConfig
	// the function used to efficiently retrieve a row by id
	Get *GetConfig
	// deprecated
	// the function used when retrieving data for multiple 'matrix items', e.g. regions
	GetMatrixItem     MatrixItemFunc
	GetMatrixItemFunc MatrixItemMapFunc
	// default transform applied to all columns
	DefaultTransform *transform.ColumnTransforms
	// function controlling default error handling behaviour
	DefaultIgnoreConfig *IgnoreConfig
	DefaultRetryConfig  *RetryConfig
	// the parent plugin object
	Plugin *Plugin
	// Deprecated: used HydrateConfig
	HydrateDependencies []HydrateDependencies
	// Config for any required hydrate functions, including dependencies between hydrate functions,
	// error handling and concurrency behaviour
	HydrateConfig []HydrateConfig
	// cache options - allows disabling of cache for this table
	Cache *TableCacheOptions

	// tags used to provide scope values for all child hydrate calls
	// (may be used for more in future)
	Tags map[string]string

	// deprecated - use DefaultIgnoreConfig
	DefaultShouldIgnoreError ErrorPredicate
	// contains filtered or unexported fields
}

| Table defines the properties of a plugin table:

func (*Table) GetSchema 

func (t *Table) GetSchema() (*proto.TableSchema, error)

GetSchema returns the proto.TableSchema, which defines the columns returned by the table.

Note: an additional '_ctx' column is added to all table schemas. This contains Steampipe specific data. (Currently this is populated with the connection name.)

type TableCacheOptions 

type TableCacheOptions struct {
	Enabled bool
}

TableCacheOptions provides a mechanism to disable caching for a specific table.

It is useful in cases where the table returns a huge volume of data cheaply.

Use TableCacheOptions to override the .cache off property of the CLI.

type TableMapData 

type TableMapData struct {
	Connection      *Connection
	ConnectionCache *connection.ConnectionCache
	// contains filtered or unexported fields
}

func (*TableMapData) GetSourceFiles added in v5.3.0 

func (d *TableMapData) GetSourceFiles(source string) ([]string, error)

GetSourceFiles accept a source path downloads files if necessary, and returns a list of local file paths

type TableMapFunc 

type TableMapFunc func(ctx context.Context, d *TableMapData) (map[string]*Table, error)

TableMapFunc is callback function which can be used to populate plugin.Plugin.TableMap and allows the connection config to be used in the table creation (connection config is not available at plugin creation time).

This callback function should be implemented by the plugin writer for dynamic plugins.

Plugin examples:

Source Files

View all Source files

Directories

Path Synopsis
Package context_key provides keys used to retrieve items from the context
 Package context_key provides keys used to retrieve items from the context
Package quals is the SDK representation of a SQL query qualifier, i.e.
 Package quals is the SDK representation of a SQL query qualifier, i.e.
Package schema provides types used to define the plugin.ConnectionConfigSchema
 Package schema provides types used to define the plugin.ConnectionConfigSchema
Package transform defines functions that modify plugin.Column values.
 Package transform defines functions that modify plugin.Column values.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.