lib

Documentation

Overview

Package gcf runs a GCF function that triggers in 2 scenarios:

1. completion of prophet-flume job in borg. On triggering it sets up new cloud BT table, scales up BT cluster (if needed) and starts a dataflow job.
2. completion of BT cache ingestion dataflow job. It scales BT cluster down (if needed).

There are two set of trigger functions defined: - ProdBTImportController - PrivateBTImportController which targets on production imports and private imports. The folder structure are different for the two scenarios. The environment variables for deployments are stored in (prod|private)/*.yaml

Dataflow launcher helper functions. launchDataflowJob will either start a Dataflow job using a Flex Template or a Classic Template depending on the template path. Flex templates are json based Dataflow templates and MUST end with ".json".

Index

• Constants
• func DeleteBTTable(ctx context.Context, projectID, instance, table string) error
• func DoesObjectExist(ctx context.Context, path string) (bool, error)
• func JoinURL(base string, paths ...string) string
• func LaunchDataflowJob(ctx context.Context, projectID string, instance string, tableID string, ...) error
• func ParsePath(path string) (string, string, error)
• func SetupBT(ctx context.Context, btProjectID, btInstance, tableID string) error
• func WriteToGCS(ctx context.Context, path, data string) error
• type GCSEvent

Constants

const (

	// NOTE: The following three files represents the state of a BT import. They
	// get written under:
	//
	//		<controlPath>/<TableID>/
	//
	// Init: written by borg to start BT import.
	InitFile = "init.txt"
	// Launched: written by this cloud function to mark launching of BT import job.
	LaunchedFile = "launched.txt"
	// Completed: written by dataflow to mark completion of BT import.
	CompletedFile = "completed.txt"
)

Functions

func DeleteBTTable

func DeleteBTTable(ctx context.Context, projectID, instance, table string) error

func DoesObjectExist

func DoesObjectExist(ctx context.Context, path string) (bool, error)

func JoinURL

func JoinURL(base string, paths ...string) string

joinURL joins url components. path.Join does work well for url, for example gs:// is changaed to gs:/

func LaunchDataflowJob

func LaunchDataflowJob(
	ctx context.Context,
	projectID string,
	instance string,
	tableID string,
	dataPath string,
	controlPath string,
	dataflowTemplate string,
	appProfileID string,
	maxNumWorkers string,
) error

func ParsePath

func ParsePath(path string) (string, string, error)

parsePath returns the GCS bucket and object from path in the form of gs://<bucket>/<object>

func SetupBT

func SetupBT(ctx context.Context, btProjectID, btInstance, tableID string) error

func WriteToGCS

func WriteToGCS(ctx context.Context, path, data string) error

Types

type GCSEvent

type GCSEvent struct {
	Name   string `json:"name"` // File name in the control folder
	Bucket string `json:"bucket"`
}

GCSEvent is the payload of a GCS event.