keyvaluelist

Documentation

Overview

Package keyvaluelist provides KeyValue string tuples and a sorted list of KeyValues as well as utilities to transalte these into hadoop like key/path-based-partitions.

Index

• Variables
• func MaybePartitions(record interface{}) string
• type KeyValue
  • func FromHadoopPartition(in string) (KeyValue, error)
  • func (kv KeyValue) ToHadoopPartition() string
• type KeyValues
  • func NewKeyValuesFromPath(path string) KeyValues
  • func (keyvals KeyValues) AsMap() map[string]string
  • func (keyvals KeyValues) ToFilter() func(string) bool
  • func (keyvals KeyValues) ToPartitionKey() string
• type PartitionGetter

Variables

var ErrInvalidFormat = fmt.Errorf("invalid format for hadoop partitions; must be QueryEscape({key})=QueryEscape({val})")

ErrInvalidFormat is returned when FromHadoopPartition fails to parse the input

Functions

func MaybePartitions

func MaybePartitions(record interface{}) string

MaybePartitions returns GetPartitions().ToPartitionKey() if the record is of type PartitionGetter; Otherwise it returns ""

Types

type KeyValue

type KeyValue struct {
	Key, Value string
}

KeyValue is a string tuple used to represent an order list of KeyValue pairs (unlike a map which is un-ordered)

func FromHadoopPartition

func FromHadoopPartition(in string) (KeyValue, error)

FromHadoopPartition encodes the KeyValue pair into a hadoop style partition string with the key and value QueryEncoded

func (KeyValue) ToHadoopPartition

func (kv KeyValue) ToHadoopPartition() string

ToHadoopPartition encodes the KeyValue pair into a hadoop style partition string with the key and value QueryEncoded

type KeyValues

type KeyValues []KeyValue

KeyValues is an ordered list of KeyValue elements.

func NewKeyValuesFromPath

func NewKeyValuesFromPath(path string) KeyValues

NewKeyValuesFromPath parses a hadoop/hive style path and returns any valid (if any) partition definitions as KeyValue pairs

func (KeyValues) AsMap

func (keyvals KeyValues) AsMap() map[string]string

AsMap returns the ordered KeyValue list as an unordered map

func (KeyValues) ToFilter

func (keyvals KeyValues) ToFilter() func(string) bool

ToFilter turns the key values into AND:ed filename-predicates during ReadFilteredByPrefix() scans It will return a (filter) function(string) bool which will answers the question "Does all the key value pairs exist as hadoop encoded partition keys in the provided string". Only supports exact matches.

func (KeyValues) ToPartitionKey

func (keyvals KeyValues) ToPartitionKey() string

ToPartitionKey formats KeyValues as key1=va1/key2=val2/.... as is common in hadoop file storage. keys and values are query escaped

type PartitionGetter

type PartitionGetter interface {
	GetPartitions() (KeyValues, error)
}

PartitionGetter is implemented by records that know how they want to be saved / clustered together.