tinytown

Documentation

Overview

Package tinytown processes URLTeam's second generation Terror of Tiny Town releases.

Index

• Variables
• func DownloadTorrents(dir string) error
• func DownloadTransmission(c *trpc.Client, dir string) error
• func GetReleaseIDs() ([]string, error)
• func ProcessProject(filename string, fn ProcessFunc) error
• func ProcessReleases(root string, fn ProcessFunc) error
• func SearchReleases(root, shortener string, shortcodes []string) ([]*beacon.Link, error)
• type Health
  • func GetHealth() (*Health, error)
• type Meta
• type ProcessFunc
• type ProjectStats

Variables

var Tracker = "https://tracker.archiveteam.org:1338"

Tracker is the base URL of the Terror of Tiny Town tracker instance. This can be changed to use an alternate tracker.

Functions

func DownloadTorrents

func DownloadTorrents(dir string) error

DownloadTorrents downloads all terroroftinytown releases via torrent.

func DownloadTransmission

func DownloadTransmission(c *trpc.Client, dir string) error

func GetReleaseIDs

func GetReleaseIDs() ([]string, error)

GetReleaseIDs queries the Internet Archive for the identifiers of all incremental terroroftinytown releases.

func ProcessProject

func ProcessProject(filename string, fn ProcessFunc) error

ProcessProject processes every link dump in a project release by calling fn on every link.

func ProcessReleases

func ProcessReleases(root string, fn ProcessFunc) error

ProcessReleases processes every release in a directory by calling fn on every link.

func SearchReleases

func SearchReleases(root, shortener string, shortcodes []string) ([]*beacon.Link, error)

Types

type Health

type Health struct {
	HTTPStatusCode    int                     // e.g. 200
	HTTPStatusMessage string                  // e.g. "OK"
	GitHash           []byte                  // e.g. 80ffc526a8b3fd188e6f73fab7b425af61f45d28
	Projects          []string                // IDs of all projects, including disabled ones, e.g. "bitly_6"
	ProjectStats      map[string]ProjectStats // key: project ID
}

func GetHealth

func GetHealth() (*Health, error)

type Meta

type Meta struct {
	Name              string  `json:"name"`
	MinVersion        int     `json:"min_version"`        // minimum library version
	MinClientVersion  int     `json:"min_client_version"` // minimum pipeline version
	Alphabet          string  `json:"alphabet"`
	URLTemplate       string  `json:"url_template"`
	RequestDelay      float64 `json:"request_delay"`     // e.g. 0.5
	RedirectCodes     []int   `json:"redirect_codes"`    // HTTP codes
	NoRedirectCodes   []int   `json:"no_redirect_codes"` // HTTP codes
	UnavailableCodes  []int   `json:"unavailable_codes"` // HTTP codes
	BannedCodes       []int   `json:"banned_codes"`      // HTTP codes
	BodyRegex         string  `json:"body_regex"`
	LocationAntiRegex string  `json:"location_anti_regex"`
	Method            string  `json:"method"` // HTTP method, e.g. "head"
	Enabled           bool    `json:"enabled"`
	Autoqueue         bool    `json:"autoqueue"`
	NumCountPerItem   int     `json:"num_count_per_item"`
	MaxNumItems       int     `json:"max_num_items"`
	LowerSequenceNum  int64   `json:"lower_sequence_num"`
	AutoreleaseTime   int     `json:"autorelease_time"`
}

Meta contains link dump metadata from a *.meta.json.xz file.

type ProcessFunc

type ProcessFunc func(l *beacon.Link, m *Meta, shortcodeLen int, releaseFilename, dumpFilename string) error

ProcessFunc is the type of function that is called for each link visited.

type ProjectStats

type ProjectStats struct {
	Found   int64
	Scanned int64
}