optimus

README

optimus

Usage

source:
  type: optimus
  config:
    host: optimus.com:80

Inputs

Key	Value	Example	Description
host	string	optimus.com:80	Optimus' GRPC host	required
max_size_in_mb	int	45	Max megabytes for GRPC client to receive message. Default to 45.

Outputs

Field	Sample Value
resource.urn	optimus::https://optimus-host.com/project.namespace.job
resource.name	job-name
resource.service	optimus
resource.description	Sample job description
ownership.owners[0].urn	john_doe@example.com
ownership.owners[0].name	john_doe@example.com
lineage.upstreams[].urn	bigquery::project/dataset/table
lineage.upstreams[].type	table
lineage.upstreams[].service	bigquery
lineage.downstreams[0].urn	bigquery::project/dataset/table
lineage.downstreams[0].type	table
lineage.downstreams[0].service	bigquery
properties.attributes	{}

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation

Index

• Constants
• type Client
• type Config
• type Extractor
  • func New(logger log.Logger, client Client) *Extractor
  • func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error
  • func (e *Extractor) Init(ctx context.Context, config plugins.Config) (err error)

Constants

const (
	GRPCMaxClientSendSizeMB      = 45
	GRPCMaxClientRecvSizeMB      = 45
	GRPCMaxRetry            uint = 3
)

Types

type Client

type Client interface {
	pb.NamespaceServiceClient
	pb.ProjectServiceClient
	pb.JobSpecificationServiceClient
	pb.JobRunServiceClient
	Connect(ctx context.Context, host string, maxSizeInMB int) error
	Close() error
}

type Config

type Config struct {
	Host        string `mapstructure:"host" validate:"required"`
	MaxSizeInMB int    `mapstructure:"max_size_in_mb"`
}

Config holds the set of configuration for the bigquery extractor

type Extractor

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

Extractor manages the communication with the bigquery service

func New

func New(logger log.Logger, client Client) *Extractor

func (*Extractor) Extract

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error

Extract checks if the table is valid and extracts the table schema

func (*Extractor) Init

func (e *Extractor) Init(ctx context.Context, config plugins.Config) (err error)

Init initializes the extractor