stream2bq

Documentation

Overview

Package stream2bq stream from PubSub to BigQuery 1) assets 2) compliance states 3) violations - Triggered by: Messages in related PubSub topics - Instances: one per Big Query table

• assets
• compliance states
• violations

- Output: Streming into BigQuery tables - Cardinality: one-one, one pubsub message - one stream insert in BQ - Automatic retrying: yes - Required environment variables:

• ASSETSCOLLECTIONID the name of the FireStore collection grouping all assets documents
• BQ_DATASET name of the Big Query dataset hosting the table
• BQ_TABLE name of the Big Query table where to insert streams
• OWNERLABELKEYNAME key name for the label identifying the asset owner
• VIOLATIONRESOLVERLABELKEYNAMEkey name for the label identifying the asset violation resolver

Index

• func EntryPoint(ctxEvent context.Context, PubSubMessage ram.PubSubMessage, global *Global) error
• func Initialize(ctx context.Context, global *Global)
• type Asset
• type AssetAssetBQ
• type AssetBQ
• type AssetFeedMessageBQ
• type ConstraintConfig
• type ConstraintConfigBQ
• type ConstraintMetadata
• type ConstraintMetadataBQ
• type FeedMessage
• type FeedMessageBQ
• type FunctionConfig
• type Global
• type NonCompliance
• type NonComplianceBQ
• type Parameters
• type Spec
• type SpecBQ
• type Violation
• type ViolationBQ

Functions

func EntryPoint

func EntryPoint(ctxEvent context.Context, PubSubMessage ram.PubSubMessage, global *Global) error

EntryPoint is the function to be executed for each cloud function occurence

func Initialize

func Initialize(ctx context.Context, global *Global)

Initialize is to be executed in the init() function of the cloud function to optimize the cold start

Types

type Asset

type Asset struct {
	Name                    string          `json:"name"`
	Owner                   string          `json:"owner"`
	ViolationResolver       string          `json:"violationResolver"`
	AncestryPathDisplayName string          `json:"ancestryPathDisplayName"`
	AncestryPath            string          `json:"ancestryPath"`
	AncestorsDisplayName    json.RawMessage `json:"ancestorsDisplayName"`
	Ancestors               json.RawMessage `json:"ancestors"`
	AssetType               string          `json:"assetType"`
	IamPolicy               json.RawMessage `json:"iamPolicy"`
	Resource                json.RawMessage `json:"resource"`
}

Asset Cloud Asset Metadata

type AssetAssetBQ

type AssetAssetBQ struct {
	Name                    string    `json:"name"`
	Owner                   string    `json:"owner"`
	ViolationResolver       string    `json:"violationResolver"`
	AncestryPathDisplayName string    `json:"ancestryPathDisplayName"`
	AncestryPath            string    `json:"ancestryPath"`
	AncestorsDisplayName    []string  `json:"ancestorsDisplayName"`
	Ancestors               []string  `json:"ancestors"`
	AssetType               string    `json:"assetType"`
	Deleted                 bool      `json:"deleted"`
	Timestamp               time.Time `json:"timestamp"`
}

AssetAssetBQ format to persist asset in BQ assets table

type AssetBQ

type AssetBQ struct {
	Name                    string `json:"name"`
	Owner                   string `json:"owner"`
	ViolationResolver       string `json:"violationResolver"`
	AncestryPathDisplayName string `json:"ancestryPathDisplayName"`
	AncestryPath            string `json:"ancestryPath"`
	AncestorsDisplayName    string `json:"ancestorsDisplayName"`
	Ancestors               string `json:"ancestors"`
	AssetType               string `json:"assetType"`
	IamPolicy               string `json:"iamPolicy"`
	Resource                string `json:"resource"`
}

AssetBQ format to persist asset in BQ violations table

type AssetFeedMessageBQ

type AssetFeedMessageBQ struct {
	Asset   AssetAssetBQ `json:"asset"`
	Window  ram.Window   `json:"window"`
	Deleted bool         `json:"deleted"`
	Origin  string       `json:"origin"`
}

AssetFeedMessageBQ Cloud Asset Inventory feed message for asset table

type ConstraintConfig

type ConstraintConfig struct {
	APIVersion string             `json:"apiVersion"`
	Kind       string             `json:"kind"`
	Metadata   ConstraintMetadata `json:"metadata"`
	Spec       Spec               `json:"spec"`
}

ConstraintConfig expose content of the constraint yaml file

type ConstraintConfigBQ

type ConstraintConfigBQ struct {
	Kind     string               `json:"kind"`
	Metadata ConstraintMetadataBQ `json:"metadata"`
	Spec     SpecBQ               `json:"spec"`
}

ConstraintConfigBQ format to persist in BQ

type ConstraintMetadata

type ConstraintMetadata struct {
	Name        string          `json:"name"`
	Annotations json.RawMessage `json:"annotation"`
}

ConstraintMetadata Constraint's metadata

type ConstraintMetadataBQ

type ConstraintMetadataBQ struct {
	Name        string `json:"name"`
	Annotations string `json:"annotation"`
}

ConstraintMetadataBQ format to persist in BQ

type FeedMessage

type FeedMessage struct {
	Asset  Asset      `json:"asset"`
	Window ram.Window `json:"window"`
	Origin string     `json:"origin"`
}

FeedMessage Cloud Asset Inventory feed message

type FeedMessageBQ

type FeedMessageBQ struct {
	Asset  AssetBQ    `json:"asset"`
	Window ram.Window `json:"window"`
	Origin string     `json:"origin"`
}

FeedMessageBQ format to persist in BQ

type FunctionConfig

type FunctionConfig struct {
	FunctionName   string    `json:"functionName"`
	DeploymentTime time.Time `json:"deploymentTime"`
	ProjectID      string    `json:"projectID"`
	Environment    string    `json:"environment"`
}

FunctionConfig function deployment settings

type Global

type Global struct {
	// contains filtered or unexported fields
}

Global structure for global variables to optimize the cloud function performances

type NonCompliance

type NonCompliance struct {
	Message  string          `json:"message"`
	Metadata json.RawMessage `json:"metadata"`
}

NonCompliance form the "deny" rego policy in a <templateName>.rego module

type NonComplianceBQ

type NonComplianceBQ struct {
	Message  string `json:"message"`
	Metadata string `json:"metadata"`
}

NonComplianceBQ form the "deny" rego policy in a <templateName>.rego module

type Parameters

type Parameters map[string]json.RawMessage

Parameters Constraint's settings

type Spec

type Spec struct {
	Severity   string          `json:"severity"`
	Match      json.RawMessage `json:"match"`
	Parameters json.RawMessage `json:"parameters"`
}

Spec Constraint's specifications

type SpecBQ

type SpecBQ struct {
	Severity   string `json:"severity"`
	Match      string `json:"match"`
	Parameters string `json:"parameters"`
}

SpecBQ format to persist in BQ

type Violation

type Violation struct {
	NonCompliance    NonCompliance    `json:"nonCompliance"`
	FunctionConfig   FunctionConfig   `json:"functionConfig"`
	ConstraintConfig ConstraintConfig `json:"constraintConfig"`
	FeedMessage      FeedMessage      `json:"feedMessage"`
	RegoModules      json.RawMessage  `json:"regoModules"`
}

Violation from the "audit" rego policy in "audit.rego" module

type ViolationBQ

type ViolationBQ struct {
	NonCompliance    NonComplianceBQ    `json:"nonCompliance"`
	FunctionConfig   FunctionConfig     `json:"functionConfig"`
	ConstraintConfig ConstraintConfigBQ `json:"constraintConfig"`
	FeedMessage      FeedMessageBQ      `json:"feedMessage"`
	RegoModules      string             `json:"regoModules"`
}

ViolationBQ from the "audit" rego policy in "audit.rego" module