https

README

Go HTTPS Everywhere

HTTPS Everywhere rewrite engine implementation in Golang.

Contains exports for both compressed ruleset construction, client-side read-only use, and the reconstruction into memory of compressed rules, and finally the actual intended URL rewrite logic. Matching and rewrite operations use a regex interface bridge package, which can be implemented in the target environment.

Currently missing the cookie secure flag feature which will be in a future iteration.

Contact: developer@tenta.io

Installation

1. go get github.com/tenta-browser/go-https-everywhere

API

• Parse(): Reads, and constructs the rulesets into memory
• Encode()/Decode()/EncodeToPath(): Handles encode and decode operations
• TryRewrite(): Searches and (if applicable) rewrites the input url according to the rewrite rules
• ShowStats(): Prints a line of encoding statistics

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

For any questions, please contact developer@tenta.io

Contributing

We welcome contributions, feedback and plain old complaining. Feel free to open an issue or shoot us a message to developer@tenta.io. If you'd like to contribute, please open a pull request and send us an email to sign a contributor agreement.

About the EFF

HTTPS Everywhere is a project of the Electronic Frontier Foundation.

The Electronic Frontier Foundation is the leading nonprofit organization defending civil liberties in the digital world. Founded in 1990, EFF champions user privacy, free expression, and innovation through impact litigation, policy analysis, grassroots activism, and technology development.

Support the EFF and HTTPS Everywhere

About Tenta

This HTTPS Everywhere Library is brought to you by Team Tenta. Tenta is your private, encrypted browser that protects your data instead of selling. We're building a next-generation browser that combines all the privacy tools you need, including built-in OpenVPN. Everything is encrypted by default. That means your bookmarks, saved tabs, web history, web traffic, downloaded files, IP address and DNS. A truly incognito browser that's fast and easy.

Documentation

Overview

A golang implementation of the EFF's HTTPS Anywhere This source contains exports for both _server_ and client-side use, as in, the construction of the ruleset in proprietary format and the reconstruction into memory, and actual intended URL rewrite logic The general approach is to teach a filter about the simple target hosts (no wildcard, or one wildcard at one of the endings of the pattern), this will block the vast majority of sites (with testing time well below the millisecond), complicated wildcards will be saved in a map, and transformed into a pcre regex pattern (the * token will be expanded into [^\.]+) and precompiled for speed (right now there are 22 such cases), and tried by matching the input versus the compiled regex; lastly there's a hashmap (which has indices the 32bit hash of the target string representation, values, the associated rulesets) which will load the available rules to apply, this is a filtering and retrieval logic. Upon generating the structure hash collisions are handled by evacuating the colliding entry into the forward structure. A flow of rewrite is as follows: 1. try to match input to forward map regexes 1.1 if match occurs, apply the first rule that fits (return if url is excluded), return the resulting url 2. try url in filter, if is not found (not, there is 0% false negatives) return 3. find the asociated rulesets with combinations {url, url_first_subdomain_wildcarded, url_tld_wildcarded} (Example: input = somesubdomain.example.com -> {somesubdomain.example.com, *.example.com, somesubdomain.example.*}) 3.1 if there's a match, apply the first rule that fits (return if url is excluded), and return the new url Encoding takes the structures and serializes them in a space optimized format, cuckoofilter has already an encode implemented, slice is encoded in a straightforward manner, regularMap (aka map[uint32][]int, aka hash(url)->array(of_applicable_rulesets)) needs an extra step, since the unique values are around 5K (the `int`s from all the `[]int`s), the implementation is to flip the map and try to encode a [][]uint32 where the index of the first dimension is the value from the map, and the second index is the order of occurence of the hash, and finally the uint32 values are the hashes Exported functions: Parse -- reads the rules from the given path, and constructs the appropriate structures resulting in a HtEvSt or an error Encode/EncodeToPath/Decode -- as their name suggests handles encoding and decoding of the structure, EncodeToPath flushes the compressed format to a specified file. TryRewrite -- searches for and applies the appropriate rewrite rules, returns the new url (or old one if no match occured) or an error ShowStats -- prints the statistics of the structure in memory

Index

• Variables
• type ExclusionSt
• type HtEvSt
  • func Decode(b []byte) (h *HtEvSt, e error)
  • func Parse(RulePath string) (*HtEvSt, error)
  • func (h *HtEvSt) Encode() (ret []byte, e error)
  • func (h *HtEvSt) EncodeToPath(outFile string) (b []byte, e error)
  • func (h *HtEvSt) ShowStats()
  • func (h *HtEvSt) TryRewrite(in string) (out string, e error)
• type RuleSt
• type RulesetSt
  • func (r *RulesetSt) String() (s string)
• type SimplifiedRulesetSt
  • func (s *SimplifiedRulesetSt) String() (z string)
• type TargetSt
• type TestSt

Variables

var LogEnabled = true

LogEnabled controls whether a log is written

var RuleIndex int

RuleIndex - used for indexing the RulesetSt's as they are generated

Types

type ExclusionSt

type ExclusionSt struct {
	Pattern string `xml:"pattern,attr"`
}

ExclusionSt - structure used to import exclusion data from xml

type HtEvSt

type HtEvSt struct {
	// contains filtered or unexported fields
}

HtEvSt - internal representation of the input rulesets

func Decode

func Decode(b []byte) (h *HtEvSt, e error)

Decode - reconstructs the structure from a byte slice

func Parse

func Parse(RulePath string) (*HtEvSt, error)

Parse - reads rule xml files and constructs their in-memory representation

func (*HtEvSt) Encode

func (h *HtEvSt) Encode() (ret []byte, e error)

Encode - encodes internal structure

func (*HtEvSt) EncodeToPath

func (h *HtEvSt) EncodeToPath(outFile string) (b []byte, e error)

EncodeToPath - encodes internal structure into a byte slice, and flushes it to disk

func (*HtEvSt) ShowStats

func (h *HtEvSt) ShowStats()

ShowStats - prints a statistics line about the internal structure

func (*HtEvSt) TryRewrite

func (h *HtEvSt) TryRewrite(in string) (out string, e error)

TryRewrite - exported function which finds rule struct (if applicable), and applies the (most) appropriate rewrite rule okay, so, tri-state return: problem -> e != nil, no match -> out == "" && e == nil, match -> out != "" && e != nil

type RuleSt

type RuleSt struct {
	From string `xml:"from,attr"`
	To   string `xml:"to,attr"`
}

RuleSt - structure used to import rewrite data from xml

type RulesetSt

type RulesetSt struct {
	Index     int
	Name      string        `xml:"name,attr"`
	Disabled  string        `xml:"default_off,attr"`
	Platform  string        `xml:"platform,attr"`
	Target    []TargetSt    `xml:"target"`
	Rule      []RuleSt      `xml:"rule"`
	Exclusion []ExclusionSt `xml:"exclusion"`
	Test      []TestSt      `xml:"test"`
}

RulesetSt - represents an xml rule file

func (*RulesetSt) String

func (r *RulesetSt) String() (s string)

type SimplifiedRulesetSt

type SimplifiedRulesetSt struct {
	// contains filtered or unexported fields
}

SimplifiedRulesetSt - a rule file, holding only necessary data

func (*SimplifiedRulesetSt) String

func (s *SimplifiedRulesetSt) String() (z string)

type TargetSt

type TargetSt struct {
	Host string `xml:"host,attr"`
}

TargetSt - structure used to import target data from xml

type TestSt

type TestSt struct {
	URL string `xml:"url,attr"`
}

TestSt - structure used to import testing data from xml