util

Documentation

Index

• Constants
• func CamelToSnake(str string) string
• func CheckValidDCIDs(dcids []string) bool
• func GetContainedIn(typeRelationJSONFilePath string) (map[TypePair][]string, error)
• func KeyToDcid(key string) (string, error)
• func KeysToSlice(m map[string]bool) []string
• func MergeDedupe(strLists ...[]string) []string
• func ParseBigtableGroup(s string) []string
• func PrintMemUsage()
• func RandomString() string
• func RemoveKeyPrefix(key string) (string, error)
• func Sample(m proto.Message, strategy *SamplingStrategy) proto.Message
• func SnakeToCamel(s string) string
• func StringContainedIn(target string, strs []string) bool
• func StringList(strs []string) string
• func TimeTrack(start time.Time, name string)
• func UnzipAndDecode(contents string) ([]byte, error)
• func ZipAndEncode(contents []byte) (string, error)
• type PlaceStatVar
• type SamplingStrategy
• type TypePair

Constants

const (
	// LimitFactor is the amount to multiply the limit by to make sure certain
	// triples are returned by the BQ query.
	LimitFactor = 1
	// TextType represents text type.
	TextType = "Text"
)

Functions

func CamelToSnake

func CamelToSnake(str string) string

CamelToSnake converts a camel case string to snake case string.

func CheckValidDCIDs

func CheckValidDCIDs(dcids []string) bool

CheckValidDCIDs checks if DCIDs are valid. More criteria will be added as being discovered.

func GetContainedIn

func GetContainedIn(typeRelationJSONFilePath string) (map[TypePair][]string, error)

GetContainedIn returns the contained in relation change given two types.

func KeyToDcid

func KeyToDcid(key string) (string, error)

KeyToDcid ... The Bigtable key is in the form of "x/y/dcid^prop1^prop2^..."

func KeysToSlice

func KeysToSlice(m map[string]bool) []string

KeysToSlice stores the keys of a map in a slice.

func MergeDedupe

func MergeDedupe(strLists ...[]string) []string

MergeDedupe merges a list of string lists and remove duplicate elements.

func ParseBigtableGroup

func ParseBigtableGroup(s string) []string

ParseBigtableGroup retrieves Bigtable names for import group from newline separated list.

func PrintMemUsage

func PrintMemUsage()

PrintMemUsage outputs the current, total and OS memory being used. As well as the number of garage collection cycles completed.

func RandomString

func RandomString() string

RandomString creates a random string with 16 runes.

func RemoveKeyPrefix

func RemoveKeyPrefix(key string) (string, error)

RemoveKeyPrefix removes the prefix of a big query key

func Sample

func Sample(m proto.Message, strategy *SamplingStrategy) proto.Message

Sample constructs a sampled protobuf message based on the sampling strategy. The output is deterministic given the same strategy.

func SnakeToCamel

func SnakeToCamel(s string) string

SnakeToCamel converts a snake case string to camel case string.

func StringContainedIn

func StringContainedIn(target string, strs []string) bool

StringContainedIn returns true if TARGET is contained in STRS

func StringList

func StringList(strs []string) string

StringList formats a list of strings into a comma-separated list with each surrounded with quotes.

func TimeTrack

func TimeTrack(start time.Time, name string)

TimeTrack is used to track function execution time.

func UnzipAndDecode

func UnzipAndDecode(contents string) ([]byte, error)

UnzipAndDecode decompresses the given contents using gzip and decodes it from base64

func ZipAndEncode

func ZipAndEncode(contents []byte) (string, error)

ZipAndEncode Compresses the given contents using gzip and encodes it in base64

Types

type PlaceStatVar

type PlaceStatVar struct {
	Place   string
	StatVar string
}

PlaceStatVar holds a place and a stat var dcid.

type SamplingStrategy

type SamplingStrategy struct {
	// Maximum number of samples.
	//
	// -1 means sample all the data. A positive integer indicates the maximum
	// number of samples.
	MaxSample int
	// Sampling strategy for the child fields.
	Children map[string]*SamplingStrategy
	// Proto fields or map keys that are not sampled at all.
	Exclude []string
}

SamplingStrategy represents the strategy to sample a JSON object.

Sampling is performed uniform acroos the items for list, or the keys for map.For example, when MaxSample=4, sampling of [1,2,3,4,5,6,7,8,9] would give [3,5,7,9]

type TypePair

type TypePair struct {
	Child  string
	Parent string
}

TypePair represents two types that are related.