tf

Documentation

Overview

Package tf - transferrer - has two uses.

1. As standalone client (see ./cmd), fetching completed questionnaires from [remote-host]/transferrer-endpoint as gzipped JSON via http(s) request; then extracting CSV data from the downloaded JSON files. JSON questionnaires and CSVs are stored to ./app-bucket/responses/downloaded/[survey_id]; multiple configs are required for this mode; they reside in ./app-bucket/transferrer; config-autogen.json contains some seed values for a valid config for the main server app; config-[survey].json contains the full blown settings for a main server app logins-remote-salt.json is needed for form tokens; remote-fmt.json or remote-fmt-localhost.json contain

data to make the https POST
destination survey
remote login

2. For direct download; as extension to [remote-host]/transferrer-endpoint; directly returning the CSV extract; we simply have to login as an admin

who has a role [survey_id]-downloader != "";

then we have to append format=CSV to the usual URL parameters; for instance https://survey2.zew.de:443/transferrer-endpoint?fetch_all=1&survey_id=fmt&wave_id=2022-04&format=CSV

Consider to get rid of the standalone mode - since the download of JSON files has become uninteresting

Index

• func ConfigsMainApp()
• func LogAndRespond(w http.ResponseWriter, r *http.Request, s string, err error)
• func PipeQStoResponse(w http.ResponseWriter, r *http.Request, qs []*qst.QuestionnaireT)
• func ProcessQs(cfgRem *RemoteConnConfigT, qs []*qst.QuestionnaireT, ...) (string, error)
• func RetrieveFromLocal(pth string, fetchAll string) ([]*qst.QuestionnaireT, error)
• func RetrieveFromRemote(cfgRem *RemoteConnConfigT) ([]*qst.QuestionnaireT, error)
• func Superset(keys [][]string) (superset []string)
• type RemoteConnConfigT
  • func ConfigTransferrer() *RemoteConnConfigT
  • func Example1() RemoteConnConfigT
  • func Example2() RemoteConnConfigT
  • func LoadRemote(r io.Reader) *RemoteConnConfigT

Functions

func ConfigsMainApp

func ConfigsMainApp()

ConfigsMainApp loads configs and logins *similar* to main app

func LogAndRespond

func LogAndRespond(w http.ResponseWriter, r *http.Request, s string, err error)

func PipeQStoResponse

func PipeQStoResponse(
	w http.ResponseWriter,
	r *http.Request,
	qs []*qst.QuestionnaireT,
)

PipeQStoResponse takes a slice of questionnaires and writess them into the http response

func ProcessQs

func ProcessQs(cfgRem *RemoteConnConfigT, qs []*qst.QuestionnaireT, saveQSFilesToDownloadDir bool) (string, error)

ProcessQs iterates over qs and extracts columns and values; it is independent of the structure of the questionaires in qs

func RetrieveFromLocal

func RetrieveFromLocal(
	pth string,
	fetchAll string,

) ([]*qst.QuestionnaireT, error)

RetrieveFromLocal reads the JSONified questionnaires from app-bucket on the survey server

func RetrieveFromRemote

func RetrieveFromRemote(cfgRem *RemoteConnConfigT) (
	[]*qst.QuestionnaireT,
	error,
)

RetrieveFromRemote requests the JSONified questionnaires from the survey server endpoint; decompresses the GZIPPed response and parses the bytes into a slice of questionnaires

func Superset

func Superset(keys [][]string) (superset []string)

Superset returns the union of all keys; the sort order is kept; new or sparse keys are inserted according to their last known predecessor; otherwise at the end. Purpose: Create CSV columns in intuitive order.

Types

type RemoteConnConfigT

type RemoteConnConfigT struct {
	RemoteHost    string
	BindSocket    string
	URLPathPrefix string

	AdminLogin string // Some admin account of the remote machine
	Pass       string

	SurveyType string
	WaveID     string // special value "current" is evaluated to current year

	DownloadDir string
	MinUserID   int // constrain range of UserIDs being processed, exclude test user data entry
	MaxUserID   int // see MinUserID
}

RemoteConnConfigT is on top of of the ordinary config

func ConfigTransferrer

func ConfigTransferrer() *RemoteConnConfigT

ConfigTransferrer loads the transferrer config; using loadRemote(); TransferrerEndpointH() in direct download mode uses another method with far fewer values

func Example1

func Example1() RemoteConnConfigT

Example1 returns a minimal configuration for usage as client

func Example2

func Example2() RemoteConnConfigT

Example2 returns a minimal configuration for usage on remote server

func LoadRemote

func LoadRemote(r io.Reader) *RemoteConnConfigT

LoadRemote reads from an io.Reader to avoid cyclical deps.