filtering

package
v0.107.37 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 7, 2023 License: GPL-3.0 Imports: 34 Imported by: 0

README

AdGuard Home's DNS filtering go library

Example use:

[ -z "$GOPATH" ] && export GOPATH=$HOME/go
go get -d github.com/AdguardTeam/AdGuardHome/filtering

Create file filter.go

package main

import (
    "github.com/AdguardTeam/AdGuardHome/filtering"
    "log"
)

func main() {
    filter := filtering.New()
    filter.AddRule("||dou*ck.net^")
    host := "www.doubleclick.net"
    res, err := filter.CheckHost(host)
    if err != nil {
        // temporary failure
        log.Fatalf("Failed to check host %q: %s", host, err)
    }
    if res.IsFiltered {
        log.Printf("Host %s is filtered, reason - %q, matched rule: %q", host, res.Reason, res.Rule)
    } else {
        log.Printf("Host %s is not filtered, reason - %q", host, res.Reason)
    }
}

And then run it:

go run filter.go

You will get:

2000/01/01 00:00:00 Host www.doubleclick.net is filtered, reason - 'FilteredBlackList', matched rule: '||dou*ck.net^'

You can also enable checking against AdGuard's SafeBrowsing:

package main

import (
    "github.com/AdguardTeam/AdGuardHome/filtering"
    "log"
)

func main() {
    filter := filtering.New()
    filter.EnableSafeBrowsing()
    host := "wmconvirus.narod.ru" // hostname for testing safebrowsing
    res, err := filter.CheckHost(host)
    if err != nil {
        // temporary failure
        log.Fatalf("Failed to check host %q: %s", host, err)
    }
    if res.IsFiltered {
        log.Printf("Host %s is filtered, reason - %q, matched rule: %q", host, res.Reason, res.Rule)
    } else {
        log.Printf("Host %s is not filtered, reason - %q", host, res.Reason)
    }
}

Documentation

Overview

Package filtering implements a DNS request and response filter.

Index

Constants

View Source 
const (
	CustomListID = -iota
	SysHostsListID
	BlockedSvcsListID
	ParentalListID
	SafeBrowsingListID
	SafeSearchListID
)

The IDs of built-in filter lists.

Keep in sync with client/src/helpers/constants.js. TODO(d.kolyshev): Add RewritesListID and don't forget to keep in sync.

Variables

This section is empty.

Functions

func InitModule 

func InitModule()

InitModule manually initializes blocked services map.

func ValidateUpdateIvl added in v0.107.14 

func ValidateUpdateIvl(i uint32) bool

ValidateUpdateIvl returns false if i is not a valid filters update interval.

Types

type BlockedServices added in v0.107.33 

type BlockedServices struct {
	// Schedule is blocked services schedule for every day of the week.
	Schedule *schedule.Weekly `json:"schedule" yaml:"schedule"`

	// IDs is the names of blocked services.
	IDs []string `json:"ids" yaml:"ids"`
}

BlockedServices is the configuration of blocked services.

func (*BlockedServices) Clone added in v0.107.33 

func (s *BlockedServices) Clone() (c *BlockedServices)

Clone returns a deep copy of blocked services.

func (*BlockedServices) Validate added in v0.107.33 

func (s *BlockedServices) Validate() (err error)

Validate returns an error if blocked services contain unknown service ID. s must not be nil.

type BlockingMode added in v0.107.37 

type BlockingMode string

BlockingMode is an enum of all allowed blocking modes. 

const (
	// BlockingModeCustomIP means respond with a custom IP address.
	BlockingModeCustomIP BlockingMode = "custom_ip"

	// BlockingModeDefault is the same as BlockingModeNullIP for
	// Adblock-style rules, but responds with the IP address specified in
	// the rule when blocked by an `/etc/hosts`-style rule.
	BlockingModeDefault BlockingMode = "default"

	// BlockingModeNullIP means respond with a zero IP address: "0.0.0.0"
	// for A requests and "::" for AAAA ones.
	BlockingModeNullIP BlockingMode = "null_ip"

	// BlockingModeNXDOMAIN means respond with the NXDOMAIN code.
	BlockingModeNXDOMAIN BlockingMode = "nxdomain"

	// BlockingModeREFUSED means respond with the REFUSED code.
	BlockingModeREFUSED BlockingMode = "refused"
)

Allowed blocking modes.

type Checker added in v0.107.30 

type Checker interface {
	// Check returns true if request for the host should be blocked.
	Check(host string) (block bool, err error)
}

Checker is used for safe browsing or parental control hash-prefix filtering.

type Config 

type Config struct {
	// BlockingIPv4 is the IP address to be returned for a blocked A request.
	BlockingIPv4 netip.Addr `yaml:"blocking_ipv4"`

	// BlockingIPv6 is the IP address to be returned for a blocked AAAA request.
	BlockingIPv6 netip.Addr `yaml:"blocking_ipv6"`

	// SafeBrowsingChecker is the safe browsing hash-prefix checker.
	SafeBrowsingChecker Checker `yaml:"-"`

	// ParentControl is the parental control hash-prefix checker.
	ParentalControlChecker Checker `yaml:"-"`

	SafeSearch SafeSearch `yaml:"-"`

	// BlockedServices is the configuration of blocked services.
	// Per-client settings can override this configuration.
	BlockedServices *BlockedServices `yaml:"blocked_services"`

	// EtcHosts is a container of IP-hostname pairs taken from the operating
	// system configuration files (e.g. /etc/hosts).
	EtcHosts *aghnet.HostsContainer `yaml:"-"`

	// Called when the configuration is changed by HTTP request
	ConfigModified func() `yaml:"-"`

	// Register an HTTP handler
	HTTPRegister aghhttp.RegisterFunc `yaml:"-"`

	// HTTPClient is the client to use for updating the remote filters.
	HTTPClient *http.Client `yaml:"-"`

	// ProtectionDisabledUntil is the timestamp until when the protection is
	// disabled.
	ProtectionDisabledUntil *time.Time `yaml:"protection_disabled_until"`

	SafeSearchConf SafeSearchConfig `yaml:"safe_search"`

	// DataDir is used to store filters' contents.
	DataDir string `yaml:"-"`

	// BlockingMode defines the way how blocked responses are constructed.
	BlockingMode BlockingMode `yaml:"blocking_mode"`

	// ParentalBlockHost is the IP (or domain name) which is used to respond to
	// DNS requests blocked by parental control.
	ParentalBlockHost string `yaml:"parental_block_host"`

	// SafeBrowsingBlockHost is the IP (or domain name) which is used to respond
	// to DNS requests blocked by safe-browsing.
	SafeBrowsingBlockHost string `yaml:"safebrowsing_block_host"`

	Rewrites []*LegacyRewrite `yaml:"rewrites"`

	// Filters are the blocking filter lists.
	Filters []FilterYAML `yaml:"-"`

	// WhitelistFilters are the allowing filter lists.
	WhitelistFilters []FilterYAML `yaml:"-"`

	// UserRules is the global list of custom rules.
	UserRules []string `yaml:"-"`

	SafeBrowsingCacheSize uint `yaml:"safebrowsing_cache_size"` // (in bytes)
	SafeSearchCacheSize   uint `yaml:"safesearch_cache_size"`   // (in bytes)
	ParentalCacheSize     uint `yaml:"parental_cache_size"`     // (in bytes)
	// TODO(a.garipov): Use timeutil.Duration
	CacheTime uint `yaml:"cache_time"` // Element's TTL (in minutes)

	// FiltersUpdateIntervalHours is the time period to update filters
	// (in hours).
	FiltersUpdateIntervalHours uint32 `yaml:"filters_update_interval"`

	// BlockedResponseTTL is the time-to-live value for blocked responses.  If
	// 0, then default value is used (3600).
	BlockedResponseTTL uint32 `yaml:"blocked_response_ttl"`

	// FilteringEnabled indicates whether or not use filter lists.
	FilteringEnabled bool `yaml:"filtering_enabled"`

	ParentalEnabled     bool `yaml:"parental_enabled"`
	SafeBrowsingEnabled bool `yaml:"safebrowsing_enabled"`

	// ProtectionEnabled defines whether or not use any of filtering features.
	ProtectionEnabled bool `yaml:"protection_enabled"`
	// contains filtered or unexported fields
}

Config allows you to configure DNS filtering with New() or just change variables directly.

type DNSFilter 

type DNSFilter struct {
	// contains filtered or unexported fields
}

DNSFilter matches hostnames and DNS requests against filtering rules.

func New 

func New(c *Config, blockFilters []Filter) (d *DNSFilter, err error)

New creates properly initialized DNS Filter that is ready to be used. c must be non-nil.

func (*DNSFilter) ApplyBlockedServices 

func (d *DNSFilter) ApplyBlockedServices(setts *Settings)

ApplyBlockedServices - set blocked services settings for this DNS request

func (*DNSFilter) ApplyBlockedServicesList added in v0.107.33 

func (d *DNSFilter) ApplyBlockedServicesList(setts *Settings, list []string)

ApplyBlockedServicesList appends filtering rules to the settings.

func (*DNSFilter) BlockedResponseTTL added in v0.107.37 

func (d *DNSFilter) BlockedResponseTTL() (ttl uint32)

BlockedResponseTTL returns TTL for blocked responses.

func (*DNSFilter) BlockingMode added in v0.107.37 

func (d *DNSFilter) BlockingMode() (mode BlockingMode, bIPv4, bIPv6 netip.Addr)

BlockingMode returns blocking mode properties.

func (*DNSFilter) CheckHost 

func (d *DNSFilter) CheckHost(
	host string,
	qtype uint16,
	setts *Settings,
) (res Result, err error)

CheckHost tries to match the host against filtering rules, then safebrowsing and parental control rules, if they are enabled.

func (*DNSFilter) CheckHostRules 

func (d *DNSFilter) CheckHostRules(host string, rrtype uint16, setts *Settings) (Result, error)

CheckHostRules tries to match the host against filtering rules only.

func (*DNSFilter) Close 

func (d *DNSFilter) Close()

Close - close the object

func (*DNSFilter) EnableFilters added in v0.107.14 

func (d *DNSFilter) EnableFilters(async bool)

func (*DNSFilter) EtcHostsRecords added in v0.107.37 

func (d *DNSFilter) EtcHostsRecords(hostname string) (recs []*hostsfile.Record)

EtcHostsRecords returns the hosts records for the hostname.

func (*DNSFilter) ParentalBlockHost added in v0.107.37 

func (d *DNSFilter) ParentalBlockHost() (host string)

ParentalBlockHost returns a host for parental protection blocked responses.

func (*DNSFilter) ProtectionStatus added in v0.107.37 

func (d *DNSFilter) ProtectionStatus() (status bool, disabledUntil *time.Time)

ProtectionStatus returns the status of protection and time until it's disabled if so.

func (*DNSFilter) RegisterFilteringHandlers added in v0.107.14 

func (d *DNSFilter) RegisterFilteringHandlers()

RegisterFilteringHandlers - register handlers

func (*DNSFilter) SafeBrowsingBlockHost added in v0.107.37 

func (d *DNSFilter) SafeBrowsingBlockHost() (host string)

SafeBrowsingBlockHost returns a host for safe browsing blocked responses.

func (*DNSFilter) SetBlockingMode added in v0.107.37 

func (d *DNSFilter) SetBlockingMode(mode BlockingMode, bIPv4, bIPv6 netip.Addr)

SetBlockingMode sets blocking mode properties.

func (*DNSFilter) SetEnabled 

func (d *DNSFilter) SetEnabled(enabled bool)

SetEnabled sets the status of the *DNSFilter.

func (*DNSFilter) SetProtectionEnabled added in v0.107.37 

func (d *DNSFilter) SetProtectionEnabled(status bool)

SetProtectionEnabled updates the status of protection.

func (*DNSFilter) SetProtectionStatus added in v0.107.37 

func (d *DNSFilter) SetProtectionStatus(status bool, disabledUntil *time.Time)

SetProtectionStatus updates the status of protection and time until it's disabled.

func (*DNSFilter) Settings added in v0.107.33 

func (d *DNSFilter) Settings() (s *Settings)

Settings returns filtering settings.

func (*DNSFilter) Start 

func (d *DNSFilter) Start()

Start - start the module: . start async filtering initializer goroutine . register web handlers

func (*DNSFilter) WriteDiskConfig 

func (d *DNSFilter) WriteDiskConfig(c *Config)

WriteDiskConfig - write configuration

type DNSRewriteResult 

type DNSRewriteResult struct {
	Response DNSRewriteResultResponse `json:",omitempty"`
	RCode    rules.RCode              `json:",omitempty"`
}

DNSRewriteResult is the result of application of $dnsrewrite rules.

type DNSRewriteResultResponse 

type DNSRewriteResultResponse map[rules.RRType][]rules.RRValue

DNSRewriteResultResponse is the collection of DNS response records the server returns.

type Filter 

type Filter struct {
	// FilePath is the path to a filtering rules list file.
	FilePath string `yaml:"-"`

	// Data is the content of the file.
	Data []byte `yaml:"-"`

	// ID is automatically assigned when filter is added using nextFilterID.
	ID int64 `yaml:"id"`
}

Filter represents a filter list

type FilterYAML added in v0.107.14 

type FilterYAML struct {
	Enabled     bool
	URL         string    // URL or a file path
	Name        string    `yaml:"name"`
	RulesCount  int       `yaml:"-"`
	LastUpdated time.Time `yaml:"-"`

	Filter `yaml:",inline"`
	// contains filtered or unexported fields
}

FilterYAML represents a filter list in the configuration file.

TODO(e.burkov): Investigate if the field ordering is important.

func (*FilterYAML) Path added in v0.107.14 

func (filter *FilterYAML) Path(dataDir string) string

Path to the filter contents

type LegacyRewrite added in v0.107.1 

type LegacyRewrite struct {
	// Domain is the domain pattern for which this rewrite should work.
	Domain string `yaml:"domain"`

	// Answer is the IP address, canonical name, or one of the special
	// values: "A" or "AAAA".
	Answer string `yaml:"answer"`

	// IP is the IP address that should be used in the response if Type is
	// dns.TypeA or dns.TypeAAAA.
	IP netip.Addr `yaml:"-"`

	// Type is the DNS record type: A, AAAA, or CNAME.
	Type uint16 `yaml:"-"`
}

LegacyRewrite is a single legacy DNS rewrite record.

Instances of *LegacyRewrite must never be nil.

func (*LegacyRewrite) Compare added in v0.107.37 

func (rw *LegacyRewrite) Compare(b *LegacyRewrite) (res int)

Compare is used to sort rewrites according to the following priority:

  1. A and AAAA > CNAME;
  2. wildcard > exact;
  3. lower level wildcard > higher level wildcard;

type LookupStats 

type LookupStats struct {
	Requests   uint64 // number of HTTP requests that were sent
	CacheHits  uint64 // number of lookups that didn't need HTTP requests
	Pending    int64  // number of currently pending HTTP requests
	PendingMax int64  // maximum number of pending HTTP requests
}

LookupStats store stats collected during safebrowsing or parental checks

type Reason 

type Reason int

Reason holds an enum detailing why it was filtered or not filtered 

const (

	// NotFilteredNotFound - host was not find in any checks, default value for result
	NotFilteredNotFound Reason = iota
	// NotFilteredAllowList - the host is explicitly allowed
	NotFilteredAllowList
	// NotFilteredError is returned when there was an error during
	// checking.  Reserved, currently unused.
	NotFilteredError

	// FilteredBlockList - the host was matched to be advertising host
	FilteredBlockList
	// FilteredSafeBrowsing - the host was matched to be malicious/phishing
	FilteredSafeBrowsing
	// FilteredParental - the host was matched to be outside of parental control settings
	FilteredParental
	// FilteredInvalid - the request was invalid and was not processed
	FilteredInvalid
	// FilteredSafeSearch - the host was replaced with safesearch variant
	FilteredSafeSearch
	// FilteredBlockedService - the host is blocked by "blocked services" settings
	FilteredBlockedService

	// Rewritten is returned when there was a rewrite by a legacy DNS rewrite
	// rule.
	Rewritten

	// RewrittenAutoHosts is returned when there was a rewrite by autohosts
	// rules (/etc/hosts and so on).
	RewrittenAutoHosts

	// RewrittenRule is returned when a $dnsrewrite filter rule was applied.
	//
	// TODO(a.garipov): Remove Rewritten and RewrittenAutoHosts by merging their
	// functionality into RewrittenRule.
	//
	// See https://github.com/AdguardTeam/AdGuardHome/issues/2499.
	RewrittenRule
)

func (Reason) In 

func (r Reason) In(reasons ...Reason) (ok bool)

In returns true if reasons include r.

func (Reason) Matched 

func (r Reason) Matched() bool

Matched returns true if any match at all was found regardless of whether it was filtered or not.

func (Reason) String 

func (r Reason) String() string

type Resolver 

type Resolver interface {
	LookupIP(ctx context.Context, network, host string) (ips []net.IP, err error)
}

Resolver is the interface for net.Resolver to simplify testing.

type Result 

type Result struct {
	// DNSRewriteResult is the $dnsrewrite filter rule result.
	DNSRewriteResult *DNSRewriteResult `json:",omitempty"`

	// CanonName is the CNAME value from the lookup rewrite result.  It is empty
	// unless Reason is set to Rewritten or RewrittenRule.
	CanonName string `json:",omitempty"`

	// ServiceName is the name of the blocked service.  It is empty unless
	// Reason is set to FilteredBlockedService.
	ServiceName string `json:",omitempty"`

	// IPList is the lookup rewrite result.  It is empty unless Reason is set to
	// Rewritten.
	IPList []netip.Addr `json:",omitempty"`

	// Rules are applied rules.  If Rules are not empty, each rule is not nil.
	Rules []*ResultRule `json:",omitempty"`

	// Reason is the reason for blocking or unblocking the request.
	Reason Reason `json:",omitempty"`

	// IsFiltered is true if the request is filtered.
	IsFiltered bool `json:",omitempty"`
}

Result contains the result of a request check.

All fields transitively have omitempty tags so that the query log doesn't become too large.

TODO(a.garipov): Clarify relationships between fields. Perhaps replace with a sum type or an interface?

type ResultRule 

type ResultRule struct {
	// Text is the text of the rule.
	Text string `json:",omitempty"`
	// IP is the host IP.  It is nil unless the rule uses the
	// /etc/hosts syntax or the reason is FilteredSafeSearch.
	IP netip.Addr `json:",omitempty"`
	// FilterListID is the ID of the rule's filter list.
	FilterListID int64 `json:",omitempty"`
}

ResultRule contains information about applied rules.

type SafeSearch added in v0.107.26 

type SafeSearch interface {
	// CheckHost checks host with safe search filter.  CheckHost must be safe
	// for concurrent use.  qtype must be either [dns.TypeA] or [dns.TypeAAAA].
	CheckHost(host string, qtype uint16) (res Result, err error)

	// Update updates the configuration of the safe search filter.  Update must
	// be safe for concurrent use.  An implementation of Update may ignore some
	// fields, but it must document which.
	Update(conf SafeSearchConfig) (err error)
}

SafeSearch interface describes a service for search engines hosts rewrites.

type SafeSearchConfig added in v0.107.26 

type SafeSearchConfig struct {
	// CustomResolver is the resolver used by safe search.
	CustomResolver Resolver `yaml:"-" json:"-"`

	// Enabled indicates if safe search is enabled entirely.
	Enabled bool `yaml:"enabled" json:"enabled"`

	Bing       bool `yaml:"bing" json:"bing"`
	DuckDuckGo bool `yaml:"duckduckgo" json:"duckduckgo"`
	Google     bool `yaml:"google" json:"google"`
	Pixabay    bool `yaml:"pixabay" json:"pixabay"`
	Yandex     bool `yaml:"yandex" json:"yandex"`
	YouTube    bool `yaml:"youtube" json:"youtube"`
}

SafeSearchConfig is a struct with safe search related settings.

type ServiceEntry 

type ServiceEntry struct {
	Name  string
	Rules []*rules.NetworkRule
}

ServiceEntry - blocked service array element

type Settings 

type Settings struct {
	ClientName string
	ClientIP   netip.Addr
	ClientTags []string

	ServicesRules []ServiceEntry

	ProtectionEnabled   bool
	FilteringEnabled    bool
	SafeSearchEnabled   bool
	SafeBrowsingEnabled bool
	ParentalEnabled     bool

	// ClientSafeSearch is a client configured safe search.
	ClientSafeSearch SafeSearch
}

Settings are custom filtering settings for a client.

type Stats 

type Stats struct {
	Safebrowsing LookupStats
	Parental     LookupStats
	Safesearch   LookupStats
}

Stats store LookupStats for safebrowsing, parental and safesearch

Source Files

View all Source files

Directories

Path Synopsis
Package hashprefix used for safe browsing and parent control.
 Package hashprefix used for safe browsing and parent control.
Package rewrite implements DNS Rewrites storage and request matching.
 Package rewrite implements DNS Rewrites storage and request matching.
Package rulelist contains the implementation of the standard rule-list filter that wraps an urlfilter filtering-engine.
 Package rulelist contains the implementation of the standard rule-list filter that wraps an urlfilter filtering-engine.
Package safesearch implements safesearch host matching.
 Package safesearch implements safesearch host matching.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.