process

Documentation

Index

• func HandleSyncEvent(ctx context.Context, event *SyncEvent) error
• func HandleSyncTableEvent(ctx context.Context, event *SyncTableEvent) error
• func InvokeBackgroundSync(ctx context.Context, lambdaAPI lambdaiface.LambdaAPI, event *SyncEvent) (err error)
• func SQS(event events.SQSEvent) error
• func Setup()
• type DataCatalogEvent
• type SyncEvent
• type SyncTableEvent

Functions

func HandleSyncEvent

func HandleSyncEvent(ctx context.Context, event *SyncEvent) error

HandleSyncEvent handles a full database sync by invoking all required table sync events in the background

func HandleSyncTableEvent

func HandleSyncTableEvent(ctx context.Context, event *SyncTableEvent) error

HandleSyncTableEvent starts or continues a gluetasks.SyncTablePartitions task. nolint: nakedret

func InvokeBackgroundSync

func InvokeBackgroundSync(ctx context.Context, lambdaAPI lambdaiface.LambdaAPI, event *SyncEvent) (err error)

InvokeBackgroundSync triggers a database sync in the background. The sync is kicked-off using a Lambda call and will be a long running task. This function does not wait for it to finish. If no TraceID is provided this function will try to use the AWS request id.

func SQS

func SQS(event events.SQSEvent) error

func Setup

func Setup()

Types

type DataCatalogEvent

type DataCatalogEvent struct {
	events.SQSEvent
	SyncDatabaseEvent   *SyncEvent
	SyncTablePartitions *SyncTableEvent
}

DataCatalogEvent is the event that this lambda accepts.

type SyncEvent

type SyncEvent struct {
	// An identifier to use in order to keep track of all 'child' Lambda invocations for this sync.
	TraceID string
	// Which databases to sync
	DatabaseNames []string
	// Which log types to sync
	LogTypes []string
	// If set to true the sync will only scan for updates and will not modify Glue partitions
	DryRun bool
}

SyncEvent is a request to start a full database sync in the background.

type SyncTableEvent

type SyncTableEvent struct {
	// Use a common trace id (the SyncDatabase request id) for all events triggered by a sync event.
	// This is used in all logging done by the SyncTablePartitions task to be able to trace all lambda invocations
	// back to their original SyncDatabase request.
	TraceID string
	// NumCalls keeps track of the number of recursive calls for the specific sync table event.
	// It acts as a guard against infinite recursion.
	NumCalls int
	// NumTimeouts keeps track of how many times the last token was retried because of timeout.
	NumTimeouts int
	// Embed the full sync table partitions task state
	// This allows us to continue the task by recursively calling the lambda.
	// The task carries all status information over
	gluetasks.SyncTablePartitions
}

SyncTableEvent initializes or continues a gluetasks.SyncTablePartitions task