urlfilter

README

AdGuard content blocking library

Pure GO library that implements AdGuard filtering rules syntax.

You can learn more about AdGuard filtering rules syntax from this article.

TODO:

• Basic filtering rules
  • Core blocking syntax
  • Basic engine
  • Basic rules validation (don't match everything, unexpected modifiers, etc)
  • Domain modifier semantics: https://github.com/AdguardTeam/AdguardBrowserExtension/issues/1474
  • TLD support: https://kb.adguard.com/en/general/how-to-create-your-own-ad-filters#wildcard-for-tld
• Benchmark basic rules matching
• Hosts matching rules
  • /etc/hosts matching
  • $badfilter support for host-blocking network rules
• Memory optimization
• Tech document
• Cosmetic rules
  • Basic element hiding and CSS rules
    • Proper CSS rules validation
  • ExtCSS rules
    • ExtCSS rules validation
  • Scriptlet rules
  • JS rules
• Proxy implementation
  • Simple MITM proxy example
  • Add cosmetic filters to the proxy example
  • Handling cosmetic modifiers $elemhide, $generichide, $jsinject
  • (!) Server certificate verification - it should pass badssl.com/dashboard/
  • Use fetch metadata to detect the content type: https://www.w3.org/TR/fetch-metadata/
  • Unit tests coverage
  • Fix TODOs
  • Proxy - handle CSP (including tags with CSP)
  • Proxy - proper blocking page code
  • Proxy - unblocking via a temporary cookie
  • Proxy - content script caching
  • Proxy - content script compression
  • Proxy - brotli support (see here)
  • Content script - babel plugin
  • Content script - apply ExtCSS rules
  • Content script - styles protection
  • Content script - JS unit tests
  • Content script - GO unit tests
• HTML filtering rules
• Advanced modifiers
  • $important
  • $replace
  • $csp
  • $cookie
  • $redirect
  • $badfilter
  • $badfilter (https://github.com/AdguardTeam/CoreLibs/issues/1241)
  • $ping modifier (https://github.com/AdguardTeam/CoreLibs/issues/1258)

How to use

TODO

Documentation

Overview

Package urlfilter contains implementation of AdGuard content blocking engine

Index

• type CosmeticEngine
  • func NewCosmeticEngine(s *filterlist.RuleStorage) *CosmeticEngine
  • func (e *CosmeticEngine) Match(hostname string, includeCSS bool, includeJS bool, includeGenericCSS bool) CosmeticResult
• type CosmeticResult
• type DNSEngine
  • func NewDNSEngine(s *filterlist.RuleStorage) *DNSEngine
  • func (d *DNSEngine) Match(hostname string) (DNSResult, bool)
  • func (d *DNSEngine) MatchRequest(dReq DNSRequest) (DNSResult, bool)
• type DNSRequest
• type DNSResult
• type Engine
  • func NewEngine(s *filterlist.RuleStorage) *Engine
  • func (e *Engine) GetCosmeticResult(hostname string, option rules.CosmeticOption) CosmeticResult
  • func (e *Engine) MatchRequest(r *rules.Request) rules.MatchingResult
• type NetworkEngine
  • func NewNetworkEngine(s *filterlist.RuleStorage) *NetworkEngine
  • func (n *NetworkEngine) Match(r *rules.Request) (*rules.NetworkRule, bool)
  • func (n *NetworkEngine) MatchAll(r *rules.Request) []*rules.NetworkRule
• type ScriptsResult
• type StylesResult

Types

type CosmeticEngine

type CosmeticEngine struct {
	// contains filtered or unexported fields
}

CosmeticEngine combines all the cosmetic rules and allows to quickly find all rules matching this or that hostname

func NewCosmeticEngine

func NewCosmeticEngine(s *filterlist.RuleStorage) *CosmeticEngine

NewCosmeticEngine builds a new cosmetic engine from the specified rule storage

func (*CosmeticEngine) Match

func (e *CosmeticEngine) Match(hostname string, includeCSS bool, includeJS bool, includeGenericCSS bool) CosmeticResult

Match builds scripts and styles that needs to be injected into the specified page hostname is the page hostname includeCSS defines if we should inject any CSS and element hiding rules (see $elemhide) includeJS defines if we should inject JS into the page (see $jsinject) includeGenericCSS defines if we should inject generic CSS and element hiding rules (see $generichide) TODO: Additionally, we should provide a method that writes result to an io.Writer

type CosmeticResult

type CosmeticResult struct {
	ElementHiding StylesResult
	CSS           StylesResult
	JS            ScriptsResult
}

CosmeticResult represents all scripts and styles that needs to be injected into the page

type DNSEngine

type DNSEngine struct {
	RulesCount int // count of rules loaded to the engine

	IPEngine *rules.IPEngine
	IPRules  *rules.IPRule
	// contains filtered or unexported fields
}

DNSEngine combines host rules and network rules and is supposed to quickly find matching rules for hostnames. First, it looks over network rules and returns first rule found. Then, if nothing found, it looks up the host rules.

func NewDNSEngine

func NewDNSEngine(s *filterlist.RuleStorage) *DNSEngine

NewDNSEngine parses the specified filter lists and returns a DNSEngine built from them. key of the map is the filter list ID, value is the raw content of the filter list.

func (*DNSEngine) Match

func (d *DNSEngine) Match(hostname string) (DNSResult, bool)

Match finds a matching rule for the specified hostname.

It returns true and the list of rules found or false and nil. The list of rules can be found when there're multiple host rules matching the same domain. For instance: 192.168.0.1 example.local 2000::1 example.local

func (*DNSEngine) MatchRequest

func (d *DNSEngine) MatchRequest(dReq DNSRequest) (DNSResult, bool)

MatchRequest - matches the specified DNS request

It returns true and the list of rules found or false and nil. The list of rules can be found when there're multiple host rules matching the same domain. For instance: 192.168.0.1 example.local 2000::1 example.local

type DNSRequest

type DNSRequest struct {
	Hostname string // Hostname (or IP address)

	Answer           bool     // If true - this hostname or IP is from a DNS response
	SortedClientTags []string // Sorted list of client tags ($ctag)
	ClientIP         string   // Client IP address
	ClientName       string   // Client name
}

DNSRequest represents a DNS query with associated metadata.

type DNSResult

type DNSResult struct {
	NetworkRule *rules.NetworkRule // a network rule or nil
	HostRulesV4 []*rules.HostRule  // host rules for IPv4 or nil
	HostRulesV6 []*rules.HostRule  // host rules for IPv6 or nil
	IPRule      *rules.IPRule
}

DNSResult - the return value of Match() function

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine represents the filtering engine with all the loaded rules

func NewEngine

func NewEngine(s *filterlist.RuleStorage) *Engine

NewEngine parses the filtering rules and creates a filtering engine of them

func (*Engine) GetCosmeticResult

func (e *Engine) GetCosmeticResult(hostname string, option rules.CosmeticOption) CosmeticResult

GetCosmeticResult gets cosmetic result for the specified hostname and cosmetic options

func (*Engine) MatchRequest

func (e *Engine) MatchRequest(r *rules.Request) rules.MatchingResult

MatchRequest - matches the specified request against the filtering engine and returns the matching result.

type NetworkEngine

type NetworkEngine struct {
	RulesCount int // RulesCount -- count of rules added to the engine
	// contains filtered or unexported fields
}

NetworkEngine is the engine that supports quick search over network rules

func NewNetworkEngine

func NewNetworkEngine(s *filterlist.RuleStorage) *NetworkEngine

NewNetworkEngine builds an instance of the network engine

func (*NetworkEngine) Match

func (n *NetworkEngine) Match(r *rules.Request) (*rules.NetworkRule, bool)

Match searches over all filtering rules loaded to the engine It returns true if a match was found alongside the matching rule

func (*NetworkEngine) MatchAll

func (n *NetworkEngine) MatchAll(r *rules.Request) []*rules.NetworkRule

MatchAll finds all rules matching the specified request regardless of the rule types It will find both whitelist and blacklist rules

type ScriptsResult

type ScriptsResult struct {
	Generic  []string
	Specific []string
}

ScriptsResult contains scripts to be executed on a page

type StylesResult

type StylesResult struct {
	Generic        []string `json:"generic"`
	Specific       []string `json:"specific"`
	GenericExtCSS  []string `json:"genericExtCss"`
	SpecificExtCSS []string `json:"specificExtCss"`
}

StylesResult contains either element hiding or CSS rules