deduplicator

Documentation

Overview

Package deduplicator provides URL deduplication.

Index

• type Deduplicator
  • func (d *Deduplicator) Filter(urls []*url.URL) (filtered []*url.URL)

Types

type Deduplicator

type Deduplicator struct {
	// contains filtered or unexported fields
}

A Deduplicator manages the deduplication of URLs, allowing you to filter out those which you've already visited. The zero value is valid.

func (*Deduplicator) Filter

func (d *Deduplicator) Filter(urls []*url.URL) (filtered []*url.URL)

Filter implementation.