dedupe

Documentation

Overview

Package dedupe provides an interface to dedupe anything. It was originally written for deduping the firehose. Given a prefix and a size it will create a unique hash based on the content of the message to be deduped. It will test if that hash has been seen before in whatever storage mechanism its been given.

DeduperStorage can be any type of storage that implements the given interface. See the memcache package for an example.

Index

• Variables
• type Deduper
  • func (f *Deduper) Contains(content []byte) (bool, error)
  • func (f *Deduper) Remove(content []byte) error
• type DeduperStorage

Variables

var (
	ErrorKeyExistsOnWrite = errors.New("Dedupe: key exists while attempting write")
	ErrorKeyExistsOnRead  = errors.New("Dedupe: key exists while attempting read")
	ErrorUnknown          = errors.New("Dedupe: unknown error occured")
	ErrorUnknownType      = errors.New("Dedupe: unknown filtering algorithm")
)

Error codes

Types

type Deduper

type Deduper struct {
	Type    string
	Size    int
	Prefix  string
	Storage DeduperStorage
}

Type - the type of filtering algorithm to use Size - the of the filter, the bigger the number the more unique (for lossy hashmaps only) Prefix - some kind of namespace that relates to your implementation Storage - storage mechanism for searching for duplicates, see memcache package for example

func (*Deduper) Contains

func (f *Deduper) Contains(content []byte) (bool, error)

After implementing your storage interface and instantiating a Deduper struct, call this method to determine if the message is a duplicate (true) or is not a duplicate (false). Will return an error if there is a failure in the storage mechanism.

d := &Deduper{Type: "hashmap", Size: 1000000, Prefix: "dedupe:firehose:memcache", Storage: &MemcacheDeduper{}}
if d.Contans(j.Message) == false {
	// process message
}

func (*Deduper) Remove

func (f *Deduper) Remove(content []byte) error

type DeduperStorage

type DeduperStorage interface {
	Exists(key string) (bool, error)
	Add(key string, value string) error
	Remove(key string) error
}