schema

package
v0.13.22 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 6, 2022 License: MPL-2.0 Imports: 9 Imported by: 189

Documentation

Overview

Package schema defines types supported by tables in source plugins

Index

Constants

This section is empty.

Variables

View Source 
var CqIDColumn = Column{
	Name:        "_cq_id",
	Type:        TypeUUID,
	Description: "Internal CQ ID of the row",
	Resolver:    cqUUIDResolver(),
}

These columns are managed and populated by the source plugins

View Source 
var CqParentIDColumn = Column{
	Name:          "_cq_parent_id",
	Type:          TypeUUID,
	Description:   "Internal CQ ID of the parent row",
	Resolver:      parentCqUUIDResolver(),
	IgnoreInTests: true,
}
View Source 
var CqSourceNameColumn = Column{
	Name:        "_cq_source_name",
	Type:        TypeString,
	Description: "Internal CQ row that references the source plugin name data was retrieved",
}
View Source 
var CqSyncTimeColumn = Column{
	Name:        "_cq_sync_time",
	Type:        TypeTimestamp,
	Description: "Internal CQ row of when sync was started (this will be the same for all rows in a single fetch)",
}

These columns are managed and populated by the destination plugin.

Functions

func CQTypesFromValues added in v0.13.16 

func CQTypesFromValues(table *Table, values interface{}) (cqtypes.CQTypes, error)

CQTypesFromValues tries best effort to convert a slice of values to CQTypes based on the provided table columns.

func ValidateTable 

func ValidateTable(t *Table) error

Types

type ClientMeta 

type ClientMeta interface {
	ID() string
}

type Column 

type Column struct {
	// Name of column
	Name string `json:"name,omitempty"`
	// Value Type of column i.e String, UUID etc'
	Type ValueType `json:"type,omitempty"`
	// Description about column, this description is added as a comment in the database
	Description string `json:"-"`
	// Column Resolver allows to set you own data based on resolving this can be an API call or setting multiple embedded values etc'
	Resolver ColumnResolver `json:"-"`
	// Creation options allow modifying how column is defined when table is created
	CreationOptions ColumnCreationOptions `json:"creation_options,omitempty"`
	// IgnoreInTests is used to skip verifying the column is non-nil in integration tests.
	// By default, integration tests perform a fetch for all resources in cloudquery's test account, and
	// verify all columns are non-nil.
	// If IgnoreInTests is true, verification is skipped for this column.
	// Used when it is hard to create a reproducible environment with this column being non-nil (e.g. various error columns).
	IgnoreInTests bool `json:"-"`
}

Column definition for Table

type ColumnCreationOptions 

type ColumnCreationOptions struct {
	PrimaryKey bool `json:"primary_key,omitempty"`
}

ColumnCreationOptions allow modification of how column is defined when table is created

type ColumnList 

type ColumnList []Column

func (ColumnList) Get 

func (c ColumnList) Get(name string) *Column

func (ColumnList) Index added in v0.13.16 

func (c ColumnList) Index(col string) int

func (ColumnList) Names 

func (c ColumnList) Names() []string

type ColumnResolver 

type ColumnResolver func(ctx context.Context, meta ClientMeta, resource *Resource, c Column) error

ColumnResolver is called for each row received in TableResolver's data fetch. execution holds all relevant information regarding execution as well as the Column called. resource holds the current row we are resolving the column for.

func ParentColumnResolver added in v0.11.0 

func ParentColumnResolver(name string) ColumnResolver

ParentColumnResolver resolves a column from the parent's table data, if name isn't set the column will be set to null

func PathResolver 

func PathResolver(path string) ColumnResolver

PathResolver resolves a field in the Resource.Item

Examples: PathResolver("Field") PathResolver("InnerStruct.Field") PathResolver("InnerStruct.InnerInnerStruct.Field")

type DestinationResource added in v0.13.16 

type DestinationResource struct {
	TableName string          `json:"table_name"`
	Data      cqtypes.CQTypes `json:"data"`
}

This struct is what we send over the wire to destination. We dont want to reuse the same struct as otherwise we will have to comment on fields which don't get sent over the wire but still accessible code wise

type LengthTableValidator 

type LengthTableValidator struct{}

func (LengthTableValidator) Validate 

func (LengthTableValidator) Validate(t *Table) error

type Meta 

type Meta struct {
	LastUpdate time.Time `json:"last_updated"`
	FetchID    string    `json:"fetch_id,omitempty"`
}

type Multiplexer added in v0.7.1 

type Multiplexer func(meta ClientMeta) []ClientMeta

type Resource 

type Resource struct {
	// Original resource item that wa from prior resolve
	Item interface{}
	// Set if this is an embedded table
	Parent *Resource
	// internal fields
	Table *Table
	// contains filtered or unexported fields
}

Resource represents a row in it's associated table, it carries a reference to the original item, and automatically generates an Id based on Table's Columns. Resource data can be accessed by the Get and Set methods

func NewResourceData 

func NewResourceData(t *Table, parent *Resource, item interface{}) *Resource

func (*Resource) Columns 

func (r *Resource) Columns() []string

func (*Resource) Get 

func (r *Resource) Get(columnName string) cqtypes.CQType

func (*Resource) GetItem added in v0.13.15 

func (r *Resource) GetItem() interface{}

func (*Resource) GetValues added in v0.13.16 

func (r *Resource) GetValues() cqtypes.CQTypes

func (*Resource) ID added in v0.5.0 

func (r *Resource) ID() uuid.UUID

func (*Resource) Set 

func (r *Resource) Set(columnName string, value interface{}) error

Set sets a column with value. This does validation and conversion to one of concrete CQTypes. it returns an error just for backward compatibility and panics in case it fails

func (*Resource) SetItem added in v0.0.11 

func (r *Resource) SetItem(item interface{})

Override original item (this is useful for apis that follow list/details pattern)

func (*Resource) ToDestinationResource added in v0.13.16 

func (r *Resource) ToDestinationResource() DestinationResource

type Resources 

type Resources []*Resource

func (Resources) ColumnNames 

func (rr Resources) ColumnNames() []string

func (Resources) GetIds 

func (rr Resources) GetIds() []uuid.UUID

func (Resources) TableName 

func (rr Resources) TableName() string

type RowResolver 

type RowResolver func(ctx context.Context, meta ClientMeta, resource *Resource) error

type Status added in v0.13.16 

type Status byte
const (
	Undefined Status = iota
	Null
	Present
)

type SyncSummary added in v0.12.0 

type SyncSummary struct {
	Resources uint64
	Errors    uint64
	Panics    uint64
}

This is deprecated

type Table 

type Table struct {
	// Name of table
	Name string `json:"name"`
	// table description
	Description string `json:"description"`
	// Columns are the set of fields that are part of this table
	Columns ColumnList `json:"columns"`
	// Relations are a set of related tables defines
	Relations Tables `json:"relations"`
	// Resolver is the main entry point to fetching table data and
	Resolver TableResolver `json:"-"`
	// Multiplex returns re-purposed meta clients. The sdk will execute the table with each of them
	Multiplex Multiplexer `json:"-"`
	// PostResourceResolver is called after all columns have been resolved, but before the Resource is sent to be inserted. The ordering of resolvers is:
	//  (Table) Resolver → PreResourceResolver → ColumnResolvers → PostResourceResolver
	PostResourceResolver RowResolver `json:"-"`
	// PreResourceResolver is called before all columns are resolved but after Resource is created. The ordering of resolvers is:
	//  (Table) Resolver → PreResourceResolver → ColumnResolvers → PostResourceResolver
	PreResourceResolver RowResolver `json:"-"`

	// IgnoreInTests is used to exclude a table from integration tests.
	// By default, integration tests fetch all resources from cloudquery's test account, and verify all tables
	// have at least one row.
	// When IgnoreInTests is true, integration tests won't fetch from this table.
	// Used when it is hard to create a reproducible environment with a row in this table.
	IgnoreInTests bool `json:"ignore_in_tests"`

	// Parent is the parent table in case this table is called via parent table (i.e. relation)
	Parent *Table `json:"-"`
}

func (*Table) Column 

func (t *Table) Column(name string) *Column

func (*Table) OverwriteOrAddColumn added in v0.13.0 

func (t *Table) OverwriteOrAddColumn(column *Column)

If the column with the same name exists, overwrites it. Otherwise, adds the column to the beginning of the table.

func (*Table) PrimaryKeys added in v0.0.4 

func (t *Table) PrimaryKeys() []string

func (*Table) TableNames 

func (t *Table) TableNames() []string

func (*Table) ValidateColumnNames added in v0.13.11 

func (t *Table) ValidateColumnNames() error

func (*Table) ValidateDuplicateColumns added in v0.0.4 

func (t *Table) ValidateDuplicateColumns() error

func (*Table) ValidateName added in v0.13.11 

func (t *Table) ValidateName() error

type TableResolver 

type TableResolver func(ctx context.Context, meta ClientMeta, parent *Resource, res chan<- interface{}) error

TableResolver is the main entry point when a table is sync is called.

Table resolver has 3 main arguments: - meta(ClientMeta): is the client returned by the plugin.Provider Configure call - parent(Resource): resource is the parent resource in case this table is called via parent table (i.e. relation) - res(chan interface{}): is a channel to pass results fetched by the TableResolver

type TableValidator 

type TableValidator interface {
	Validate(t *Table) error
}

type Tables added in v0.0.4 

type Tables []*Table

func (Tables) Get added in v0.13.15 

func (tt Tables) Get(name string) *Table

Get return table by name

func (Tables) TableNames added in v0.0.4 

func (tt Tables) TableNames() []string

func (Tables) ValidateColumnNames added in v0.13.11 

func (tt Tables) ValidateColumnNames() error

func (Tables) ValidateDuplicateColumns added in v0.0.4 

func (tt Tables) ValidateDuplicateColumns() error

func (Tables) ValidateDuplicateTables added in v0.0.6 

func (tt Tables) ValidateDuplicateTables() error

func (Tables) ValidateTableNames added in v0.13.11 

func (tt Tables) ValidateTableNames() error

type ValueType 

type ValueType int
const (
	TypeInvalid ValueType = iota
	TypeBool
	TypeInt
	TypeFloat
	TypeUUID
	TypeString
	TypeByteArray
	TypeStringArray
	TypeIntArray
	TypeTimestamp
	TypeJSON
	TypeUUIDArray
	TypeInet
	TypeInetArray
	TypeCIDR
	TypeCIDRArray
	TypeMacAddr
	TypeMacAddrArray
	TypeTimeInterval
)

func (ValueType) String 

func (v ValueType) String() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.