splitdump

Documentation

Overview

Package splitdump nibble large CAI dumps into PubSub asset feed messages

One dump line = one PubSub message.

Triggered by

Google Cloud Storage event when a new dump is delivered.

Instances

Only one.

Output

- PubSub messages formated like Cloud Asset Inventory real-time feed messages.

- Delivered in the same topics as used per CAI real-time.

- Tags to differentiate them from CAI real time feeds.

- Create missing topics en the fly (best effort) in case it does not already exist for real-time.

Cardinality

One-many: one dump is nubbled in many feed messages.

To ensure scallabilty the function is recurssive:

- If dump size > x lines then segments the dumo into child dumps of x lines.

- else nibble the dump, one dump line = one PubSub message.

- x is set through an environment variable, e.g. 1000.

Automatic retrying

Yes.

Is recurssive

Yes.

Implementation example

package p
import (
    "context"

    "github.com/BrunoReboul/ram/services/splitdump"
    "github.com/BrunoReboul/ram/utilities/gcs"
)
var global splitdump.Global
var ctx = context.Background()

// EntryPoint is the function to be executed for each cloud function occurence
func EntryPoint(ctxEvent context.Context, gcsEvent gcs.Event) error {
    return splitdump.EntryPoint(ctxEvent, PubSubMessage, &global)
}

func init() {
    splitdump.Initialize(ctx, &global)
}

Notes

- When Cloud Asset Inventory publishes an export to Cloud Storage:

- 1st Creates an empty dump object.

- 2nd Updates the objects then.

- So, two event notifications. The cloud founction ignore the first one (empty object).

Index

• func EntryPoint(ctxEvent context.Context, gcsEvent gcs.Event, global *Global) error
• func Initialize(ctx context.Context, global *Global) (err error)
• type Global
• type InstanceDeployment
  • func NewInstanceDeployment() *InstanceDeployment
  • func (instanceDeployment *InstanceDeployment) Deploy() (err error)
  • func (instanceDeployment *InstanceDeployment) ReadValidate() (err error)
  • func (instanceDeployment *InstanceDeployment) Situate() (err error)

Functions

func EntryPoint

func EntryPoint(ctxEvent context.Context, gcsEvent gcs.Event, global *Global) error

EntryPoint is the function to be executed for each cloud function occurence

func Initialize

func Initialize(ctx context.Context, global *Global) (err error)

Initialize is to be executed in the init() function of the cloud function to optimize the cold start

Types

type Global

type Global struct {
	PubSubID string
	// contains filtered or unexported fields
}

Global structure for global variables to optimize the cloud function performances

type InstanceDeployment

type InstanceDeployment struct {
	DumpTimestamp time.Time `yaml:"dumpTimestamp"`
	Core          *deploy.Core
	Settings      struct {
		Service struct {
			GSU gsu.Parameters
			IAM iamgt.Parameters
			GCB gcb.Parameters
			GCF gcf.Parameters
		}
		Instance struct {
			SplitThresholdLineNumber   int64 `yaml:"splitThresholdLineNumber"`
			ScannerBufferSizeKiloBytes int   `yaml:"scannerBufferSizeKiloBytes"`
		}
	}
}

InstanceDeployment settings and artifacts structure

func NewInstanceDeployment

func NewInstanceDeployment() *InstanceDeployment

NewInstanceDeployment create deployment structure with default settings set

func (*InstanceDeployment) Deploy

func (instanceDeployment *InstanceDeployment) Deploy() (err error)

Deploy a service instance

func (*InstanceDeployment) ReadValidate

func (instanceDeployment *InstanceDeployment) ReadValidate() (err error)

ReadValidate reads and validates service and instance settings

func (*InstanceDeployment) Situate

func (instanceDeployment *InstanceDeployment) Situate() (err error)

Situate complement settings taking in account the situation for service and instance settings