stream2bq

Documentation

Overview

Package stream2bq streams PubSub message into BigQuery tables

It can stream into 3 RAM tables: 1) assets 2) compliance states 3) violations.

Triggered by

Messages in related PubSub topics.

Instances

- one per AssetType for resource, all streaming to assets table.

- one for compliance states, streaming to complianteStates table.

- one for violations, streaming to violations table.

Output

Streaming into BigQuery tables.

Cardinality

One-one, one pubsub message - one stream inserted in BigQuery.

Automatic retrying

Yes.

Implementation example

package p
import (
    "context"

    "github.com/BrunoReboul/ram/services/stream2bq"
    "github.com/BrunoReboul/ram/utilities/ram"
)
var global stream2bq.Global
var ctx = context.Background()

// EntryPoint is the function to be executed for each cloud function occurence
func EntryPoint(ctxEvent context.Context, PubSubMessage ram.PubSubMessage) error {
    return stream2bq.EntryPoint(ctxEvent, PubSubMessage, &global)
}

func init() {
    stream2bq.Initialize(ctx, &global)
}

Index

• func EntryPoint(ctxEvent context.Context, PubSubMessage ram.PubSubMessage, global *Global) error
• func Initialize(ctx context.Context, global *Global)
• type Global
• type InstanceDeployment
  • func NewInstanceDeployment() *InstanceDeployment
  • func (instanceDeployment *InstanceDeployment) Deploy() (err error)
  • func (instanceDeployment *InstanceDeployment) ReadValidate() (err error)
  • func (instanceDeployment *InstanceDeployment) Situate() (err error)

Functions

func EntryPoint

func EntryPoint(ctxEvent context.Context, PubSubMessage ram.PubSubMessage, global *Global) error

EntryPoint is the function to be executed for each cloud function occurence

func Initialize

func Initialize(ctx context.Context, global *Global)

Initialize is to be executed in the init() function of the cloud function to optimize the cold start

Types

type Global

type Global struct {
	// contains filtered or unexported fields
}

Global structure for global variables to optimize the cloud function performances

type InstanceDeployment

type InstanceDeployment struct {
	DumpTimestamp time.Time `yaml:"dumpTimestamp"`
	Core          *deploy.Core
	Settings      struct {
		Service struct {
			GSU gsu.Parameters
			IAM iamgt.Parameters
			GCB gcb.Parameters
			GCF gcf.Parameters
		}
		Instance struct {
			GCF      gcf.Event
			Bigquery struct {
				TableName string `yaml:"tableName"`
			}
		}
	}
}

InstanceDeployment settings and artifacts structure

func NewInstanceDeployment

func NewInstanceDeployment() *InstanceDeployment

NewInstanceDeployment create deployment structure with default settings set

func (*InstanceDeployment) Deploy

func (instanceDeployment *InstanceDeployment) Deploy() (err error)

Deploy a service instance

func (*InstanceDeployment) ReadValidate

func (instanceDeployment *InstanceDeployment) ReadValidate() (err error)

ReadValidate reads and validates service and instance settings

func (*InstanceDeployment) Situate

func (instanceDeployment *InstanceDeployment) Situate() (err error)

Situate complement settings taking in account the situation for service and instance settings