stream2bq

package
v0.4.0-rc23 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 5, 2021 License: Apache-2.0 Imports: 29 Imported by: 0

Documentation

Overview

Package stream2bq streams PubSub message into BigQuery tables

It can stream into 3 RAM tables: 1) assets 2) compliance states 3) violations.

Triggered by

Messages in related PubSub topics.

Instances

- one per AssetType for resource, all streaming to assets table.

- one for compliance states, streaming to complianteStates table.

- one for violations, streaming to violations table.

Output

Streaming into BigQuery tables.

Cardinality

One-one, one pubsub message - one stream inserted in BigQuery.

Automatic retrying

Yes.

Implementation example

package p
import (
    "context"

    "github.com/BrunoReboul/ram/services/stream2bq"
    "github.com/BrunoReboul/ram/utilities/ram"
)
var global stream2bq.Global
var ctx = context.Background()

// EntryPoint is the function to be executed for each cloud function occurence
func EntryPoint(ctxEvent context.Context, PubSubMessage gps.PubSubMessage) error {
    return stream2bq.EntryPoint(ctxEvent, PubSubMessage, &global)
}

func init() {
    stream2bq.Initialize(ctx, &global)
}

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func EntryPoint

func EntryPoint(ctxEvent context.Context, PubSubMessage gps.PubSubMessage, global *Global) error

EntryPoint is the function to be executed for each cloud function occurence

func Initialize

func Initialize(ctx context.Context, global *Global) (err error)

Initialize is to be executed in the init() function of the cloud function to optimize the cold start

Types

type Global

type Global struct {
	PubSubID string
	// contains filtered or unexported fields
}

Global structure for global variables to optimize the cloud function performances

type InstanceDeployment added in v0.0.24

type InstanceDeployment struct {
	DumpTimestamp time.Time `yaml:"dumpTimestamp"`
	Core          *deploy.Core
	Settings      struct {
		Service struct {
			GSU gsu.Parameters
			IAM iamgt.Parameters
			GCB gcb.Parameters
			GCF gcf.Parameters
		}
		Instance struct {
			GCF      gcf.Event
			Bigquery struct {
				TableName string `yaml:"tableName"`
			}
		}
	}
}

InstanceDeployment settings and artifacts structure

func NewInstanceDeployment added in v0.0.24

func NewInstanceDeployment() *InstanceDeployment

NewInstanceDeployment create deployment structure with default settings set

func (*InstanceDeployment) Deploy added in v0.0.24

func (instanceDeployment *InstanceDeployment) Deploy() (err error)

Deploy a service instance

func (*InstanceDeployment) ReadValidate added in v0.0.24

func (instanceDeployment *InstanceDeployment) ReadValidate() (err error)

ReadValidate reads and validates service and instance settings

func (*InstanceDeployment) Situate added in v0.0.24

func (instanceDeployment *InstanceDeployment) Situate() (err error)

Situate complement settings taking in account the situation for service and instance settings

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL