dedupe

Documentation

Index

• Variables
• type Dedupe
  • func NewDedupe(ch <-chan string, byteLen int) *Dedupe
  • func (d *Dedupe) Drain(opts ...Option)
  • func (d *Dedupe) GetResults() <-chan string
• type DedupeBackend
• type LevelDBBackend
  • func NewLevelDBBackend() *LevelDBBackend
  • func (l *LevelDBBackend) Cleanup()
  • func (l *LevelDBBackend) IterCallback(callback func(elem string))
  • func (l *LevelDBBackend) Upsert(elem string) bool
• type MapBackend
  • func NewMapBackend() *MapBackend
  • func (m *MapBackend) Cleanup()
  • func (m *MapBackend) IterCallback(callback func(elem string))
  • func (m *MapBackend) Upsert(elem string) bool
• type Option
  • func WithUnique(ch chan<- string) Option

Variables

var MaxInMemoryDedupeSize = 100 * 1024 * 1024

MaxInMemoryDedupeSize (default : 100 MB)

Types

type Dedupe

type Dedupe struct {
	// contains filtered or unexported fields
}

Dedupe is string deduplication type which removes all duplicates if

func NewDedupe

func NewDedupe(ch <-chan string, byteLen int) *Dedupe

NewDedupe returns a dedupe instance which removes all duplicates Note: If byteLen is not correct/specified alterx may consume lot of memory

func (*Dedupe) Drain

func (d *Dedupe) Drain(opts ...Option)

Drains channel and tries to dedupe it

func (*Dedupe) GetResults

func (d *Dedupe) GetResults() <-chan string

GetResults iterates over dedupe storage and returns results

type DedupeBackend

type DedupeBackend interface {
	// Upsert add/update key to backend/database
	Upsert(elem string) bool
	// Execute given callback on each element while iterating
	IterCallback(callback func(elem string))
	// Cleanup cleans any residuals after deduping
	Cleanup()
}

type LevelDBBackend

type LevelDBBackend struct {
	// contains filtered or unexported fields
}

func NewLevelDBBackend

func NewLevelDBBackend() *LevelDBBackend

func (*LevelDBBackend) Cleanup

func (l *LevelDBBackend) Cleanup()

func (*LevelDBBackend) IterCallback

func (l *LevelDBBackend) IterCallback(callback func(elem string))

func (*LevelDBBackend) Upsert

func (l *LevelDBBackend) Upsert(elem string) bool

type MapBackend

type MapBackend struct {
	// contains filtered or unexported fields
}

func NewMapBackend

func NewMapBackend() *MapBackend

func (*MapBackend) Cleanup

func (m *MapBackend) Cleanup()

func (*MapBackend) IterCallback

func (m *MapBackend) IterCallback(callback func(elem string))

func (*MapBackend) Upsert

func (m *MapBackend) Upsert(elem string) bool

type Option

type Option func(val string)

Option is a type for variadic options in Drain

func WithUnique

func WithUnique(ch chan<- string) Option

WithUnique is an option to send unique values to the provided channel