monitor

Documentation

Index

• type BatchController
  • func NewBatchController() BatchController
  • func (ctl BatchController) BatchList() (map[string]string, error)
  • func (ctl BatchController) BatchResourcesList(batchID string) ([]string, error)
  • func (ctl BatchController) BatchResourcesSet(batchID string, resourceIDs []string) error
  • func (ctl BatchController) BatchSet(batchID string, timestamp string) error
  • func (ctl BatchController) Close()
  • func (ctl BatchController) Destroy(batchID string) error
  • func (ctl BatchController) ListenAndDestroy()

Types

type BatchController

type BatchController struct {
	Topics topics.Controller
	// contains filtered or unexported fields
}

func NewBatchController

func NewBatchController() BatchController

func (BatchController) BatchList

func (ctl BatchController) BatchList() (map[string]string, error)

BatchList returns a list of the current batches

func (BatchController) BatchResourcesList

func (ctl BatchController) BatchResourcesList(batchID string) ([]string, error)

BatchResourcesList returns the list of resource ids of a batch

func (BatchController) BatchResourcesSet

func (ctl BatchController) BatchResourcesSet(batchID string, resourceIDs []string) error

BatchResourcesSet records the resource ids of a batch in Redis

func (BatchController) BatchSet

func (ctl BatchController) BatchSet(batchID string, timestamp string) error

BatchSet records a batch in Redis

func (BatchController) Close

func (ctl BatchController) Close()

func (BatchController) Destroy

func (ctl BatchController) Destroy(batchID string) error

Destroy deletes a batch by removing its Redis keys and Kafka topics. The Redis counter is not deleted right away for testing purposes but we set an expiration duration.

func (BatchController) ListenAndDestroy

func (ctl BatchController) ListenAndDestroy()

ListenAndDestroy consumes load.* topics and destroy a batch when all resources have been loaded. Each time a new resource instance is loaded, the batch_id counter of the resource type resource_id in Redis is incremented. A batch counter is a Redis hash of key "batch:{batch_id}:counter" containing elements of keys "resource:{resource_id}:extracted" and "resource:{resource_id}:loaded". "resource:{resource_id}:extracted" refers to the number of resources of type {resource_id} extracted. "resource:{resource_id}:loaded" refers to the number of loaded resources of type {resource_id}. The list of resource types of a batch is in a Redis set "batch:{batch_id}:resources"