util

Documentation

Index

• Constants
• func CamelToSnake(str string) string
• func CheckValidDCIDs(dcids []string) bool
• func GetContainedIn(typeRelationJSONFilePath string) (map[TypePair][]string, error)
• func KeyToDcid(key string) (string, error)
• func MergeDedupe(s1 []string, s2 []string) []string
• func PrintMemUsage()
• func RandomString() string
• func RemoveKeyPrefix(key string) (string, error)
• func Sample(m proto.Message, strategy *SamplingStrategy) proto.Message
• func SnakeToCamel(s string) string
• func StringContainedIn(target string, strs []string) bool
• func StringList(strs []string) string
• func UnzipAndDecode(contents string) ([]byte, error)
• func ZipAndEncode(contents []byte) (string, error)
• type SamplingStrategy
• type TypePair

Constants

const (
	// BtPlaceStatsVarPrefix for place to statsvar list cache.
	BtPlaceStatsVarPrefix = "d/0/"
	// BtStatVarGroup is the key for statvar group cache.
	BtStatVarGroup = "d/1"
	// BtSVAndSVGExistence is the key for stat var and stat var group existence cache.
	BtSVAndSVGExistence = "d/2/"
	// BtPlaceObsPrefix for internal place obs cache.
	BtPlaceObsPrefix = "d/3/"
	// BtPlacePagePrefix for place page cache.
	BtPlacePagePrefix = "d/4/"
	// BtTriplesPrefix for internal GetTriples cache.
	BtTriplesPrefix = "d/7/"
	// BtArcsPrefix for internal arcs cache.
	BtArcsPrefix = "d/9/"
	// BtStatVarSummary for stat var summary cache.
	BtStatVarSummary = "d/a/"
	// BtPlacesInPrefix for GetPlacesIn cache.
	BtPlacesInPrefix = "d/c/"
	// BtChartDataPrefix for chart data.
	BtChartDataPrefix = "d/f/"
	// BtInPropValPrefix for in-arc prop value.
	BtInPropValPrefix = "d/l/"
	// BtOutPropValPrefix for out-arc prop value.
	BtOutPropValPrefix = "d/m/"
	// BtRelatedLocationsSameTypePrefix for related places with same type.
	BtRelatedLocationsSameTypePrefix = "d/o/"
	// BtRelatedLocationsSameTypeAndAncestorPrefix for related places with same type and ancestor.
	BtRelatedLocationsSameTypeAndAncestorPrefix = "d/q/"
	// BtRelatedLocationsSameTypePCPrefix for related places with same type, per capita.
	BtRelatedLocationsSameTypePCPrefix = "d/o0/"
	// BtRelatedLocationsSameTypeAndAncestorPCPrefix for related places with same type and ancestor,
	// per capita.
	BtRelatedLocationsSameTypeAndAncestorPCPrefix = "d/q0/"

	// BtFamily is the key for the row.
	BtFamily = "csv"
	// BtCacheLimit is the cache limit. The limit is per predicate and neighbor type.
	BtCacheLimit = 500
	// BtBatchQuerySize is the size of BigTable batch query.
	BtBatchQuerySize = 1000
	// LimitFactor is the amount to multiply the limit by to make sure certain
	// triples are returned by the BQ query.
	LimitFactor = 1
	// TextType represents text type.
	TextType = "Text"
)

Functions

func CamelToSnake

func CamelToSnake(str string) string

CamelToSnake converts a camel case string to snake case string.

func CheckValidDCIDs

func CheckValidDCIDs(dcids []string) bool

CheckValidDCIDs checks if DCIDs are valid. More criteria will be added as being discovered.

func GetContainedIn

func GetContainedIn(typeRelationJSONFilePath string) (map[TypePair][]string, error)

GetContainedIn returns the contained in relation change given two types.

func KeyToDcid

func KeyToDcid(key string) (string, error)

KeyToDcid ... The Bigtable key is in the form of "x/y/dcid^prop1^prop2^..."

func MergeDedupe

func MergeDedupe(s1 []string, s2 []string) []string

MergeDedupe merges two string slices and remove duplicate elements.

func PrintMemUsage

func PrintMemUsage()

PrintMemUsage outputs the current, total and OS memory being used. As well as the number of garage collection cycles completed.

func RandomString

func RandomString() string

RandomString creates a random string with 16 runes.

func RemoveKeyPrefix

func RemoveKeyPrefix(key string) (string, error)

RemoveKeyPrefix removes the prefix of a big query key

func Sample

func Sample(m proto.Message, strategy *SamplingStrategy) proto.Message

Sample constructs a sampled protobuf message based on the sampling strategy. The output is deterministic given the same strategy.

func SnakeToCamel

func SnakeToCamel(s string) string

SnakeToCamel converts a snake case string to camel case string.

func StringContainedIn

func StringContainedIn(target string, strs []string) bool

StringContainedIn returns true if TARGET is contained in STRS

func StringList

func StringList(strs []string) string

StringList formats a list of strings into a comma-separated list with each surrounded with quotes.

func UnzipAndDecode

func UnzipAndDecode(contents string) ([]byte, error)

UnzipAndDecode decompresses the given contents using gzip and decodes it from base64

func ZipAndEncode

func ZipAndEncode(contents []byte) (string, error)

ZipAndEncode Compresses the given contents using gzip and encodes it in base64

Types

type SamplingStrategy

type SamplingStrategy struct {
	// Maximum number of samples.
	//
	// -1 means sample all the data. A positive integer indicates the maximum
	// number of samples.
	MaxSample int
	// Sampling strategy for the child fields.
	Children map[string]*SamplingStrategy
	// Proto fields or map keys that are not sampled at all.
	Exclude []string
}

SamplingStrategy represents the strategy to sample a JSON object.

Sampling is performed uniform acroos the items for list, or the keys for map.For example, when MaxSample=4, sampling of [1,2,3,4,5,6,7,8,9] would give [3,5,7,9]

type TypePair

type TypePair struct {
	Child  string
	Parent string
}

TypePair represents two types that are related.