ds

Documentation

Overview

ds is a package for using Google Cloud Datastore.

Index

• Constants
• Variables
• func DeleteAll(client *datastore.Client, kind Kind, wait bool) (int, error)
• func EnsureNotEmulator()
• func Init(project string, ns string) error
• func InitForTesting(project string, ns string, kinds ...Kind) error
• func InitWithOpt(project string, ns string, opts ...option.ClientOption) error
• func IterKeys(client *datastore.Client, kind Kind, pageSize int) (<-chan *IterKeysItem, error)
• func NewKey(kind Kind) *datastore.Key
• func NewKeyWithParent(kind Kind, parent *datastore.Key) *datastore.Key
• func NewQuery(kind Kind) *datastore.Query
• type IterKeysItem
• type Kind

Constants

const (
	// Android Compile
	ANDROID_COMPILE_NS = "android-compile"

	// Leasing
	LEASING_SERVER_NS = "leasing-server"

	// CT
	CT_NS = "cluster-telemetry"

	// Autoroll
	AUTOROLL_NS          = "autoroll"
	AUTOROLL_INTERNAL_NS = "autoroll-internal"

	// AlertManager
	ALERT_MANAGER_NS = "alert-manager"
)

Namespaces that are used in production, and thus might be backed up.

const (
	// Maximum number of entities which may be inserted or deleted at once.
	MAX_MODIFICATIONS = 500
)

Global constants.

Variables

var (
	// DS is the Cloud Datastore client. Valid after Init() has been called.
	DS *datastore.Client

	// Namespace is the datastore namespace that data will be stored in. Valid after Init() has been called.
	Namespace string
)

var (
	// KindsToBackup is a map from namespace to the list of Kinds to backup.
	// If this value is changed then remember to push a new version of /ds/go/datastore_backup.
	//
	// Note that we try to backup all kinds and all namespaces for every project
	// even if that app isn't running there, which has a better failure mode of
	// possibly backing up too much data rather than too little.
	KindsToBackup = map[string][]Kind{
		AUTOROLL_NS:          {KIND_AUTOROLL_MODE, KIND_AUTOROLL_MODE_ANCESTOR, KIND_AUTOROLL_ROLL, KIND_AUTOROLL_ROLL_ANCESTOR, KIND_AUTOROLL_STATUS, KIND_AUTOROLL_STATUS_ANCESTOR, KIND_AUTOROLL_STRATEGY, KIND_AUTOROLL_STRATEGY_ANCESTOR, KIND_AUTOROLL_UNTHROTTLE, KIND_AUTOROLL_UNTHROTTLE_ANCESTOR},
		AUTOROLL_INTERNAL_NS: {KIND_AUTOROLL_MODE, KIND_AUTOROLL_MODE_ANCESTOR, KIND_AUTOROLL_ROLL, KIND_AUTOROLL_ROLL_ANCESTOR, KIND_AUTOROLL_STATUS, KIND_AUTOROLL_STATUS_ANCESTOR, KIND_AUTOROLL_STRATEGY, KIND_AUTOROLL_STRATEGY_ANCESTOR, KIND_AUTOROLL_UNTHROTTLE, KIND_AUTOROLL_UNTHROTTLE_ANCESTOR},
		ANDROID_COMPILE_NS:   {COMPILE_TASK, ANDROID_COMPILE_INSTANCES},
		LEASING_SERVER_NS:    {TASK},
		CT_NS:                {CAPTURE_SKPS_TASKS, CHROMIUM_ANALYSIS_TASKS, CHROMIUM_BUILD_TASKS, CHROMIUM_PERF_TASKS, LUA_SCRIPT_TASKS, METRICS_ANALYSIS_TASKS, PIXEL_DIFF_TASKS, RECREATE_PAGESETS_TASKS, RECREATE_WEBPAGE_ARCHIVES_TASKS, CLUSTER_TELEMETRY_IDS},
		ALERT_MANAGER_NS:     {INCIDENT_AM, INCIDENT_ACTIVE_PARENT_AM, SILENCE_AM, SILENCE_ACTIVE_PARENT_AM, REMINDER_AM, AUDITLOG_AM},
	}
)

Functions

func DeleteAll

func DeleteAll(client *datastore.Client, kind Kind, wait bool) (int, error)

DeleteAll removes all entities of the given kind. If wait is true it waits until an eventually consistent query of the Kind returns a count of 0. Upon success the number of deleted entities is returned.

Note: This is a very expensive operation if there are many entities of this kind and should be run as an 'offline' task.

func EnsureNotEmulator

func EnsureNotEmulator()

EnsureNotEmulator will panic if it detects the Datastore Emulator is configured.

func Init

func Init(project string, ns string) error

Init the Cloud Datastore Client (DS).

project - The project name, i.e. "google.com:skia-buildbots". ns - The datastore namespace to store data into.

func InitForTesting

func InitForTesting(project string, ns string, kinds ...Kind) error

InitForTesting is an init to call when running tests. It doesn't do any auth as it is expecting to run against the Cloud Datastore Emulator. See https://cloud.google.com/datastore/docs/tools/datastore-emulator

project - The project name, i.e. "google.com:skia-buildbots". ns - The datastore namespace to store data into.

func InitWithOpt

func InitWithOpt(project string, ns string, opts ...option.ClientOption) error

InitWithOpt the Cloud Datastore Client (DS).

project - The project name, i.e. "google.com:skia-buildbots". ns - The datastore namespace to store data into. opt - Options to pass to the client.

func IterKeys

func IterKeys(client *datastore.Client, kind Kind, pageSize int) (<-chan *IterKeysItem, error)

IterKeys iterates all keys of the specified kind in slices of pageSize length.

func NewKey

func NewKey(kind Kind) *datastore.Key

NewKey creates a new indeterminate key of the given kind.

func NewKeyWithParent

func NewKeyWithParent(kind Kind, parent *datastore.Key) *datastore.Key

func NewQuery

func NewQuery(kind Kind) *datastore.Query

NewQuery creates a new query of the given kind with the right namespace.

Types

type IterKeysItem

type IterKeysItem struct {
	Keys []*datastore.Key
	Err  error
}

IterKeysItem is the item returned by the IterKeys function via a channel.

type Kind

type Kind string

Kind of datastore entry.

const (
	// Predict
	FAILURES     Kind = "Failures"
	FLAKY_RANGES Kind = "FlakyRanges"

	// Android Compile
	COMPILE_TASK              Kind = "CompileTask"
	ANDROID_COMPILE_INSTANCES Kind = "AndroidCompileInstances"

	// Leasing
	TASK Kind = "Task"

	// CT
	CAPTURE_SKPS_TASKS              Kind = "CaptureSkpsTasks"
	CHROMIUM_ANALYSIS_TASKS         Kind = "ChromiumAnalysisTasks"
	CHROMIUM_BUILD_TASKS            Kind = "ChromiumBuildTasks"
	CHROMIUM_PERF_TASKS             Kind = "ChromiumPerfTasks"
	LUA_SCRIPT_TASKS                Kind = "LuaScriptTasks"
	METRICS_ANALYSIS_TASKS          Kind = "MetricsAnalysisTasks"
	PIXEL_DIFF_TASKS                Kind = "PixelDiffTasks"
	RECREATE_PAGESETS_TASKS         Kind = "RecreatePageSetsTasks"
	RECREATE_WEBPAGE_ARCHIVES_TASKS Kind = "RecreateWebpageArchivesTasks"
	CLUSTER_TELEMETRY_IDS           Kind = "ClusterTelemetryIDs"

	// Autoroll
	KIND_AUTOROLL_MODE                Kind = "AutorollMode"
	KIND_AUTOROLL_MODE_ANCESTOR       Kind = "AutorollModeAncestor" // Fake; used to force strong consistency for testing's sake.
	KIND_AUTOROLL_ROLL                Kind = "AutorollRoll"
	KIND_AUTOROLL_ROLL_ANCESTOR       Kind = "AutorollRollAncestor" // Fake; used to force strong consistency for testing's sake.
	KIND_AUTOROLL_STATUS              Kind = "AutorollStatus"
	KIND_AUTOROLL_STATUS_ANCESTOR     Kind = "AutorollStatusAncestor" // Fake; used to force strong consistency for testing's sake.
	KIND_AUTOROLL_STRATEGY            Kind = "AutorollStrategy"
	KIND_AUTOROLL_STRATEGY_ANCESTOR   Kind = "AutorollStrategyAncestor" // Fake; used to force strong consistency for testing's sake.
	KIND_AUTOROLL_UNTHROTTLE          Kind = "AutorollUnthrottle"
	KIND_AUTOROLL_UNTHROTTLE_ANCESTOR Kind = "AutorollUnthrottleAncestor" // Fake; used to force strong consistency for testing's sake.

	// AlertManager
	INCIDENT_AM               Kind = "IncidentAm"
	INCIDENT_ACTIVE_PARENT_AM Kind = "IncidentActiveParentAm"
	SILENCE_ACTIVE_PARENT_AM  Kind = "SilenceActiveParentAm"
	SILENCE_AM                Kind = "SilenceAm"
	REMINDER_AM               Kind = "ReminderAm"
	AUDITLOG_AM               Kind = "AuditLogAm"
)

Below are all the Kinds used in all applications. New Kinds should be listed here, and they should all have unique values, because when defining indexes for Cloud Datastore the index config is per project, not pre-namespace. Remember to add them to KindsToBackup if they are to be backed up, and push a new version of /ds/go/datastore_backup.