utils

Documentation

Index

• func CompileRegex(patterns []string) ([]*regexp.Regexp, error)
• func CreateMapFromValues(data map[string][]string) models.PropertyMap
• func ExtractTitleFromBody(body string) (string, error)
• func SetupNewTorClient(torProxy string) (*http.Client, error)
• type TorCheckResult
  • func CheckIfTorConnectionExists(torProxy string) (*TorCheckResult, error)

Functions

func CompileRegex

func CompileRegex(patterns []string) ([]*regexp.Regexp, error)

CompileRegex - compiles various Collly filters such as URLFIlters, DomainFilters, etc.

func CreateMapFromValues

func CreateMapFromValues(data map[string][]string) models.PropertyMap

CreateMapFromValues is used to convert a map into a PropertyMap which is JSONB in postgres

func ExtractTitleFromBody

func ExtractTitleFromBody(body string) (string, error)

ExtractTitleFromBody is there because I could not parse it with Colly so There's a nice regexp, it works.

func SetupNewTorClient

func SetupNewTorClient(torProxy string) (*http.Client, error)

SetupNewTorCLient allows us to pass custom transport, used for checking If our tor connection actually works.

Types

type TorCheckResult

type TorCheckResult struct {
	IsTor  bool
	Client *http.Client // Include the HTTP client
}

func CheckIfTorConnectionExists

func CheckIfTorConnectionExists(torProxy string) (*TorCheckResult, error)