dedup

package
v1.8.0-rc.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 4, 2023 License: AGPL-3.0 Imports: 11 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DefaultRudderPath

func DefaultRudderPath() string

func Init

func Init()

Types

type DedupHandleT

type DedupHandleT struct {
	// contains filtered or unexported fields
}

func New

func New(path string, fns ...OptFn) *DedupHandleT

func (*DedupHandleT) Close

func (d *DedupHandleT) Close()

func (*DedupHandleT) FindDuplicates

func (d *DedupHandleT) FindDuplicates(messageIDs []string, allMessageIDsSet map[string]struct{}) (duplicateIndexes []int)

func (*DedupHandleT) MarkProcessed

func (d *DedupHandleT) MarkProcessed(messageIDs []string) error

MarkProcessed persist messageIDs in Disk, with expiry time of dedupWindow Any message mark here will appear in FindDuplicates() if queried inside the dedupWindow

func (*DedupHandleT) PrintHistogram

func (d *DedupHandleT) PrintHistogram()

type DedupI

type DedupI interface {
	FindDuplicates(messageIDs []string, allMessageIDsSet map[string]struct{}) (duplicateIndexes []int)
	MarkProcessed(messageIDs []string) error
	PrintHistogram()
	Close()
}

func GetInstance

func GetInstance(clearDB *bool) DedupI

GetInstance returns an instance of DedupI

type OptFn

type OptFn func(*DedupHandleT)

func FromConfig

func FromConfig() OptFn

func WithClearDB

func WithClearDB() OptFn

func WithWindow

func WithWindow(d time.Duration) OptFn

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL