goctftime

README

In progress...

Documentation

Overview

Package goctftime parses and stores data from ctftime.org into a Firestore database so that it can be easily indexed and queried by an Android application that displays CTF Time data.

Handlers

The various handlers listen on the server for a GET request to their respective path. This triggers the execution of the handler logic. Each handler is responsible for parsing and storing some portion of ctftime.org. Handler logic is broken up into two phases.

The first phase triggers multiple goroutines to parse and store data concurrently. By default, the maximum number of goroutines running at once is 10. To change the maximum number of goroutines running at once, modify the maxRoutines variable. This concurrent phase only requests pages that we have scraped before.

The second phase operates on a single thread and checks to see if new content exists. If new content exists, it is parsed and stored in Firestore. Finally, we update the value used in phase one to delineate the range of known content.

GetLast and UpdateLast

Each scraper operates concurrently as it requests known content. The definition of "known" is based on a Firestore value for the respective piece of content (teams, events, etc.). This value is retrieved and possibly updated on each scraping iteration.

Index

• func CalculateHash(data interface{}) string
• func CompareCtfHash(id int, ctf Ctf, fbc FirebaseContext) (bool, error)
• func CompareTeamHash(id int, team Team, fbc FirebaseContext) (bool, error)
• func Connect(token option.ClientOption) (*firestore.Client, error)
• func DefaultHandler(_ http.ResponseWriter, _ *http.Request)
• func Fetch(url string) (*http.Response, error)
• func GenerateToken() (option.ClientOption, error)
• func GetLastCtfId(fbc FirebaseContext) int
• func GetLastTeamId(fbc FirebaseContext) int
• func ParseAndStoreCtf(ctfId int, resp *http.Response, fbc FirebaseContext) error
• func ParseAndStoreTeam(teamId int, resp *http.Response, fbc FirebaseContext) error
• func StoreCtf(ctfId int, ctf Ctf, fbc FirebaseContext) error
• func StoreTeam(teamId int, team Team, fbc FirebaseContext) error
• func UpdateCtfsHandler(w http.ResponseWriter, r *http.Request)
• func UpdateLastCtfId(fbc FirebaseContext, newCtfId int)
• func UpdateLastTeamId(fbc FirebaseContext, newPageNumber int)
• func UpdateTeamsHandler(w http.ResponseWriter, r *http.Request)
• type Ctf
• type FirebaseContext
  • func NewFirebaseContext(ctx context.Context, token option.ClientOption) (FirebaseContext, error)
• type Member
• type Score
• type Team

Functions

func CalculateHash

func CalculateHash(data interface{}) string

CalculateHash calculates the sha256 hash of a data structure.

func CompareCtfHash

func CompareCtfHash(id int, ctf Ctf, fbc FirebaseContext) (bool, error)

func CompareTeamHash

func CompareTeamHash(id int, team Team, fbc FirebaseContext) (bool, error)

func Connect

func Connect(token option.ClientOption) (*firestore.Client, error)

Connect connects to Firestore and returns an authenticated client that can read to/write from the database.

func DefaultHandler

func DefaultHandler(_ http.ResponseWriter, _ *http.Request)

DefaultHandler handles any request not defined by an existing handler. This function simply drops spurious requests and should not be modified.

func Fetch

func Fetch(url string) (*http.Response, error)

Fetch executes a GET request to a URL. If the response status code is not 200, an error is returned specifying which URL failed and the status code of the request.

func GenerateToken

func GenerateToken() (option.ClientOption, error)

GenerateToken reads an API key and returns an option to be used by Connect.

func GetLastCtfId

func GetLastCtfId(fbc FirebaseContext) int

func GetLastTeamId

func GetLastTeamId(fbc FirebaseContext) int

func ParseAndStoreCtf

func ParseAndStoreCtf(ctfId int, resp *http.Response, fbc FirebaseContext) error

func ParseAndStoreTeam

func ParseAndStoreTeam(teamId int, resp *http.Response, fbc FirebaseContext) error

func StoreCtf

func StoreCtf(ctfId int, ctf Ctf, fbc FirebaseContext) error

func StoreTeam

func StoreTeam(teamId int, team Team, fbc FirebaseContext) error

func UpdateCtfsHandler

func UpdateCtfsHandler(w http.ResponseWriter, r *http.Request)

func UpdateLastCtfId

func UpdateLastCtfId(fbc FirebaseContext, newCtfId int)

func UpdateLastTeamId

func UpdateLastTeamId(fbc FirebaseContext, newPageNumber int)

func UpdateTeamsHandler

func UpdateTeamsHandler(w http.ResponseWriter, r *http.Request)

Types

type Ctf

type Ctf struct {
	Hash  string
	Image string // relative Url to image
	Name  string
	Url   string
}

type FirebaseContext

type FirebaseContext struct {
	Ctx context.Context  // context used in connection to Firestore
	Fb  firestore.Client // client used in connection to Firestore
}

FirebaseContext contains the necessary variables to get/set Firestore data

func NewFirebaseContext

func NewFirebaseContext(ctx context.Context, token option.ClientOption) (FirebaseContext, error)

NewFirebaseContext creates a new FirebaseContext object for a Firestore request.

type Member

type Member struct {
	Id   int
	Name string
}

type Score

type Score struct {
	Points float64
	Rank   int
}

type Team

type Team struct {
	Hash string
	// General
	Aliases             []string
	Academic            string
	CountryCode         string
	Description         string
	Logo                string // relative URL
	Members             []Member
	Name                string
	NameCaseInsensitive string
	Scores              map[string]Score
	// Social
	Email      string
	ICQ        string
	Jabber     string
	LinkedIn   string
	OtherLinks []string
	Skype      string
	Telegram   string
	Twitter    string
	Website    string
}