urlfilter

README

AdGuard content blocking library

Pure GO library that implements AdGuard filtering rules syntax.

You can learn more about AdGuard filtering rules syntax from this article.

TODO:

• Basic filtering rules
  • Core blocking syntax
  • Basic engine
  • Basic rules validation (don't match everything, unexpected modifiers, etc)
  • Domain modifier semantics: https://github.com/AdguardTeam/AdguardBrowserExtension/issues/1474
  • TLD support: https://kb.adguard.com/en/general/how-to-create-your-own-ad-filters#wildcard-for-tld
• Benchmark basic rules matching
• Hosts matching rules
  • /etc/hosts matching
  • $badfilter support for host-blocking network rules
• Memory optimization
• Tech document
• Cosmetic rules
  • Basic element hiding and CSS rules
    • Proper CSS rules validation
  • ExtCSS rules
    • ExtCSS rules validation
  • Scriptlet rules
  • JS rules
• Proxy implementation
  • Simple MITM proxy example
  • Add cosmetic filters to the proxy example
  • Handling cosmetic modifiers $elemhide, $generichide, $jsinject
  • (!) Server certificate verification - it should pass badssl.com/dashboard/
  • Use fetch metadata to detect the content type: https://www.w3.org/TR/fetch-metadata/
  • Unit tests coverage
  • Fix TODOs
  • Proxy - handle CSP (including tags with CSP)
  • Proxy - proper blocking page code
  • Proxy - unblocking via a temporary cookie
  • Proxy - content script caching
  • Proxy - content script compression
  • Proxy - brotli support (see here)
  • Content script - babel plugin
  • Content script - apply ExtCSS rules
  • Content script - styles protection
  • Content script - JS unit tests
  • Content script - GO unit tests
• HTML filtering rules
• Advanced modifiers
  • $important
  • $replace
  • $csp
  • $cookie
  • $redirect
  • $badfilter
  • $badfilter (https://github.com/AdguardTeam/CoreLibs/issues/1241)
  • $ping modifier (https://github.com/AdguardTeam/CoreLibs/issues/1258)

How to use

TODO

Documentation

Overview

Package urlfilter contains implementation of AdGuard content blocking engine

Index

• type CosmeticEngine
  • func NewCosmeticEngine(s *filterlist.RuleStorage) *CosmeticEngine
  • func (e *CosmeticEngine) Match(hostname string, includeCSS, includeJS, includeGenericCSS bool) CosmeticResult
• type CosmeticResult
• type DNSEngine
  • func NewDNSEngine(s *filterlist.RuleStorage) *DNSEngine
  • func (d *DNSEngine) Match(hostname string) (res *DNSResult, matched bool)
  • func (d *DNSEngine) MatchRequest(dReq *DNSRequest) (res *DNSResult, matched bool)
• type DNSRequest
• type DNSResult
  • func (res *DNSResult) DNSRewrites() (nrules []*rules.NetworkRule)
  • func (res *DNSResult) DNSRewritesAll() (nrules []*rules.NetworkRule)
• type Engine
  • func NewEngine(s *filterlist.RuleStorage) *Engine
  • func (e *Engine) GetCosmeticResult(hostname string, option rules.CosmeticOption) CosmeticResult
  • func (e *Engine) MatchRequest(r *rules.Request) (res *rules.MatchingResult)
• type NetworkEngine
  • func NewNetworkEngine(s *filterlist.RuleStorage) (engine *NetworkEngine)
  • func NewNetworkEngineSkipStorageScan(s *filterlist.RuleStorage) (engine *NetworkEngine)
  • func (n *NetworkEngine) AddRule(f *rules.NetworkRule, storageIdx int64)
  • func (n *NetworkEngine) Match(r *rules.Request) (*rules.NetworkRule, bool)
  • func (n *NetworkEngine) MatchAll(r *rules.Request) (result []*rules.NetworkRule)
• type ScriptsResult
• type StylesResult

Types

type CosmeticEngine

type CosmeticEngine struct {
	// contains filtered or unexported fields
}

CosmeticEngine combines all the cosmetic rules and allows to quickly find all rules matching this or that hostname

func NewCosmeticEngine

func NewCosmeticEngine(s *filterlist.RuleStorage) *CosmeticEngine

NewCosmeticEngine builds a new cosmetic engine from the specified rule storage

func (*CosmeticEngine) Match

func (e *CosmeticEngine) Match(hostname string, includeCSS, includeJS, includeGenericCSS bool) CosmeticResult

Match builds scripts and styles that needs to be injected into the specified page hostname is the page hostname includeCSS defines if we should inject any CSS and element hiding rules (see $elemhide) includeJS defines if we should inject JS into the page (see $jsinject) includeGenericCSS defines if we should inject generic CSS and element hiding rules (see $generichide) TODO: Additionally, we should provide a method that writes result to an io.Writer

type CosmeticResult

type CosmeticResult struct {
	ElementHiding StylesResult
	CSS           StylesResult
	JS            ScriptsResult
}

CosmeticResult represents all scripts and styles that needs to be injected into the page

type DNSEngine

type DNSEngine struct {

	// RulesCount is the count of rules loaded to the engine.
	RulesCount int
	// contains filtered or unexported fields
}

DNSEngine combines host rules and network rules and is supposed to quickly find matching rules for hostnames. First, it looks over network rules and returns first rule found. Then, if nothing found, it looks up the host rules.

func NewDNSEngine

func NewDNSEngine(s *filterlist.RuleStorage) *DNSEngine

NewDNSEngine parses the specified filter lists and returns a DNSEngine built from them. key of the map is the filter list ID, value is the raw content of the filter list.

func (*DNSEngine) Match

func (d *DNSEngine) Match(hostname string) (res *DNSResult, matched bool)

Match finds a matching rule for the specified hostname. It returns true and the list of rules found or false and nil. A list of rules is returned when there are multiple host rules matching the same domain, for example:

192.168.0.1 example.local
2000::1 example.local

func (*DNSEngine) MatchRequest

func (d *DNSEngine) MatchRequest(dReq *DNSRequest) (res *DNSResult, matched bool)

MatchRequest matches the specified DNS request. The return parameter matched is true if the result has a basic network rule or some host rules.

For compatibility reasons, it is also false when there are DNS rewrite and other kinds of special network rules, so users who need those will need to ignore the matched return parameter and instead inspect the results of the corresponding DNSResult getters.

TODO(ameshkov): return nil when there's no match. Currently, the logic is flawed because it analyzes the DNSResult even when matched is false and looks for $dnsrewrite rules.

type DNSRequest

type DNSRequest struct {
	// ClientIP is the IP address to match against $client modifiers.  The
	// default zero value won't be considered.
	ClientIP netip.Addr

	// ClientName is the name to match against $client modifiers.  The default
	// empty value won't be considered.
	ClientName string

	// Hostname is the hostname to filter.
	Hostname string

	// SortedClientTags is the list of tags to match against $ctag modifiers.
	SortedClientTags []string

	// DNSType is the type of the resource record (RR) of a DNS request, for
	// example "A" or "AAAA".  See [rules.RRValue] for all acceptable constants
	// and their corresponding values.
	DNSType rules.RRType

	// Answer if the filtering request is for filtering a DNS response.
	Answer bool
}

DNSRequest represents a DNS query with associated metadata.

type DNSResult

type DNSResult struct {
	// NetworkRule is the matched network rule, if any.  If it is nil,
	// HostRulesV4 and HostRulesV6 may still contain matched hosts-file style
	// rules.
	NetworkRule *rules.NetworkRule

	// HostRulesV4 are the host rules with IPv4 addresses.
	HostRulesV4 []*rules.HostRule

	// HostRulesV6 are the host rules with IPv6 addresses.
	HostRulesV6 []*rules.HostRule

	// NetworkRules are all matched network rules.  These include unprocessed
	// DNS rewrites, exception rules, and so on.
	NetworkRules []*rules.NetworkRule
}

DNSResult is the result of matching a DNS filtering request.

func (*DNSResult) DNSRewrites

func (res *DNSResult) DNSRewrites() (nrules []*rules.NetworkRule)

DNSRewrites returns $dnsrewrite network rules applying exception logic. For example, rules like:

||example.com^$dnsrewrite=127.0.0.1
||example.com^$dnsrewrite=127.0.0.2
@@||example.com^$dnsrewrite=127.0.0.1

Will result in example.com being rewritten to only return 127.0.0.2.

To get all DNS rewrite rules without applying any exception logic, use (*DNSResult).DNSRewritesAll.

func (*DNSResult) DNSRewritesAll

func (res *DNSResult) DNSRewritesAll() (nrules []*rules.NetworkRule)

DNSRewritesAll returns all $dnsrewrite network rules. To get the rules with exception logic applied, use (*DNSResult).DNSRewrites.

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine represents the filtering engine with all the loaded rules

func NewEngine

func NewEngine(s *filterlist.RuleStorage) *Engine

NewEngine parses the filtering rules and creates a filtering engine of them

func (*Engine) GetCosmeticResult

func (e *Engine) GetCosmeticResult(hostname string, option rules.CosmeticOption) CosmeticResult

GetCosmeticResult gets cosmetic result for the specified hostname and cosmetic options

func (*Engine) MatchRequest

func (e *Engine) MatchRequest(r *rules.Request) (res *rules.MatchingResult)

MatchRequest - matches the specified request against the filtering engine and returns the matching result.

type NetworkEngine

type NetworkEngine struct {

	// RulesCount is the count of rules added to the engine.
	RulesCount int
	// contains filtered or unexported fields
}

NetworkEngine is the engine that supports quick search over network rules.

func NewNetworkEngine

func NewNetworkEngine(s *filterlist.RuleStorage) (engine *NetworkEngine)

NewNetworkEngine builds an instance of the network engine. This method scans the specified rule storage and adds all rules.NetworkRule found there to the internal lookup tables.

func NewNetworkEngineSkipStorageScan

func NewNetworkEngineSkipStorageScan(s *filterlist.RuleStorage) (engine *NetworkEngine)

NewNetworkEngineSkipStorageScan creates a new instance of *NetworkEngine, but unlike NewNetworkEngine it does not scans the storage.

func (*NetworkEngine) AddRule

func (n *NetworkEngine) AddRule(f *rules.NetworkRule, storageIdx int64)

AddRule adds rule to the network engine.

func (*NetworkEngine) Match

func (n *NetworkEngine) Match(r *rules.Request) (*rules.NetworkRule, bool)

Match searches over all filtering rules loaded to the engine. It returns true if a match was found alongside the matching rule.

func (*NetworkEngine) MatchAll

func (n *NetworkEngine) MatchAll(r *rules.Request) (result []*rules.NetworkRule)

MatchAll finds all rules matching the specified request regardless of the rule types. It will find both allowlist and blocklist rules.

type ScriptsResult

type ScriptsResult struct {
	Generic  []string
	Specific []string
}

ScriptsResult contains scripts to be executed on a page

type StylesResult

type StylesResult struct {
	Generic        []string `json:"generic"`
	Specific       []string `json:"specific"`
	GenericExtCSS  []string `json:"genericExtCss"`
	SpecificExtCSS []string `json:"specificExtCss"`
}

StylesResult contains either element hiding or CSS rules