config

Documentation

Index

• func BindFlags(flagSet *pflag.FlagSet)
• func InitConfig() error
• type Config
  • func GetConfig() *Config

Functions

func BindFlags

func BindFlags(flagSet *pflag.FlagSet)

BindFlags binds the flags to the viper configuration This is needed because viper doesn't support same flag name accross multiple commands Details here: https://github.com/spf13/viper/issues/375#issuecomment-794668149

func InitConfig

func InitConfig() error

InitConfig initializes the configuration Flags -> Env -> Config file -> Consul config Latest has precedence over the rest

Types

type Config

type Config struct {
	LogLevel                       string   `mapstructure:"log-level"`
	UserAgent                      string   `mapstructure:"user-agent"`
	Job                            string   `mapstructure:"job"`
	Cookies                        string   `mapstructure:"cookies"`
	APIPort                        string   `mapstructure:"api-port"`
	PrometheusPrefix               string   `mapstructure:"prometheus-prefix"`
	Proxy                          string   `mapstructure:"proxy"`
	WARCPrefix                     string   `mapstructure:"warc-prefix"`
	WARCOperator                   string   `mapstructure:"warc-operator"`
	CDXDedupeServer                string   `mapstructure:"warc-cdx-dedupe-server"`
	WARCTempDir                    string   `mapstructure:"warc-temp-dir"`
	CDXCookie                      string   `mapstructure:"cdx-cookie"`
	HQAddress                      string   `mapstructure:"hq-address"`
	HQKey                          string   `mapstructure:"hq-key"`
	HQSecret                       string   `mapstructure:"hq-secret"`
	HQProject                      string   `mapstructure:"hq-project"`
	HQStrategy                     string   `mapstructure:"hq-strategy"`
	LogFileOutputDir               string   `mapstructure:"log-file-output-dir"`
	ElasticSearchUsername          string   `mapstructure:"es-user"`
	ElasticSearchPassword          string   `mapstructure:"es-password"`
	ElasticSearchIndexPrefix       string   `mapstructure:"es-index-prefix"`
	PyroscopeAddress               string   `mapstructure:"pyroscope-address"`
	DisableHTMLTag                 []string `mapstructure:"disable-html-tag"`
	ExcludeHosts                   []string `mapstructure:"exclude-host"`
	IncludeHosts                   []string `mapstructure:"include-host"`
	ExcludeString                  []string `mapstructure:"exclude-string"`
	DomainsBypassProxy             []string `mapstructure:"bypass-proxy"`
	ElasticSearchURLs              []string `mapstructure:"es-url"`
	WorkersCount                   int      `mapstructure:"workers"`
	MaxConcurrentAssets            int      `mapstructure:"max-concurrent-assets"`
	MaxHops                        uint8    `mapstructure:"max-hops"`
	MaxRedirect                    uint8    `mapstructure:"max-redirect"`
	MaxRetry                       uint8    `mapstructure:"max-retry"`
	HTTPTimeout                    int      `mapstructure:"http-timeout"`
	MaxConcurrentRequestsPerDomain int      `mapstructure:"max-concurrent-per-domain"`
	ConcurrentSleepLength          int      `mapstructure:"concurrent-sleep-length"`
	CrawlTimeLimit                 int      `mapstructure:"crawl-time-limit"`
	CrawlMaxTimeLimit              int      `mapstructure:"crawl-max-time-limit"`
	MinSpaceRequired               int      `mapstructure:"min-space-required"`
	WARCPoolSize                   int      `mapstructure:"warc-pool-size"`
	WARCDedupeSize                 int      `mapstructure:"warc-dedupe-size"`
	HQBatchSize                    int64    `mapstructure:"hq-batch-size"`
	KeepCookies                    bool     `mapstructure:"keep-cookies"`
	Headless                       bool     `mapstructure:"headless"`
	DisableSeencheck               bool     `mapstructure:"disable-seencheck"`
	JSON                           bool     `mapstructure:"json"`
	Debug                          bool     `mapstructure:"debug"`
	LiveStats                      bool     `mapstructure:"live-stats"`
	API                            bool     `mapstructure:"api"`
	Prometheus                     bool     `mapstructure:"prometheus"`
	DomainsCrawl                   bool     `mapstructure:"domains-crawl"`
	CaptureAlternatePages          bool     `mapstructure:"capture-alternate-pages"`
	RandomLocalIP                  bool     `mapstructure:"random-local-ip"`
	WARCOnDisk                     bool     `mapstructure:"warc-on-disk"`
	DisableLocalDedupe             bool     `mapstructure:"disable-local-dedupe"`
	CertValidation                 bool     `mapstructure:"cert-validation"`
	DisableAssetsCapture           bool     `mapstructure:"disable-assets-capture"`
	HQ                             bool     // Special field to check if HQ is enabled depending on the command called
	HQContinuousPull               bool     `mapstructure:"hq-continuous-pull"`
	HQRateLimitSendBack            bool     `mapstructure:"hq-rate-limiting-send-back"`
	NoStdoutLogging                bool     `mapstructure:"no-stdout-log"`
	NoBatchWriteWAL                bool     `mapstructure:"ultrasafe-queue"`
	Handover                       bool     `mapstructure:"handover"`
}

Config holds all configuration for our program

func GetConfig

func GetConfig() *Config

GetConfig returns the config struct