dedup

Documentation

Index

• func KeyFromCSV(b []byte, fields []int, sep string) string
• func KeyFromJSON(b []byte, fields []string) string
• type Dedup
  • func New() *Dedup
  • func (w *Dedup) Add(key string, b []byte)
  • func (w *Dedup) Lines() [][]byte

Functions

func KeyFromCSV

func KeyFromCSV(b []byte, fields []int, sep string) string

KeyFromCSV generates a 'key' of fields values by concatenating the field values in the order received in fields.

If returned string is empty then the fields were not found.

If a fields index value is out of range then that field is ignored and not included in the key.

func KeyFromJSON

func KeyFromJSON(b []byte, fields []string) string

KeyFromJSON generates a 'key' of fields values by concatenating the field values in the order received in fields.

If returned string is empty then the fields were not found.

Types

type Dedup

type Dedup struct {
	// contains filtered or unexported fields
}

Dedup will dedup lines added to AddLine by the provided key. Newer lines replace older lines. If order is important then consider sorting all the lines first.

func New

func New() *Dedup

func (*Dedup) Add

func (w *Dedup) Add(key string, b []byte)

AddLine will add the bytes record b to the pool of deduped linesMp. As a help, basic csv and json key generators are provided as standalone functions in this package.

func (*Dedup) Lines

func (w *Dedup) Lines() [][]byte

Lines returns the deduped lines.