utils

Documentation

Overview

Package utils contains various common utils separate by utility types

Package utils contains various common utils separate by utility types

Index

• func IsWebsocketURL(rawurl string) bool
• func SetupHTTPEthClient(ethURL string) (*ethclient.Client, error)
• func SetupWebsocketEthClient(ethURL string, killChan <-chan struct{}, pingDelay int) (*ethclient.Client, error)
• type CrawlerConfig
  • func (c *CrawlerConfig) Address() string
  • func (c *CrawlerConfig) DataVersion() string
  • func (c *CrawlerConfig) Dbname() string
  • func (c *CrawlerConfig) FetchListingAddresses() error
  • func (c *CrawlerConfig) FetchMultiSigAddresses() error
  • func (c *CrawlerConfig) OutputUsage()
  • func (c *CrawlerConfig) Password() string
  • func (c *CrawlerConfig) PersistType() cconfig.PersisterType
  • func (c *CrawlerConfig) PoolConnLifetimeSecs() *int
  • func (c *CrawlerConfig) PoolMaxConns() *int
  • func (c *CrawlerConfig) PoolMaxIdleConns() *int
  • func (c *CrawlerConfig) PopulateFromEnv() error
  • func (c *CrawlerConfig) Port() int
  • func (c *CrawlerConfig) User() string
• type Edge
• type PageInfo
• type RebuildConfig
  • func (r *RebuildConfig) OutputUsage()
  • func (r *RebuildConfig) PopulateFromEnv() error
• type WatcherSubscription
  • func NewWatcherSubscription(watcherName string, producer func(<-chan struct{}) error) WatcherSubscription

Functions

func IsWebsocketURL

func IsWebsocketURL(rawurl string) bool

IsWebsocketURL return true if the given URL is a websocket URL

func SetupHTTPEthClient

func SetupHTTPEthClient(ethURL string) (*ethclient.Client, error)

SetupHTTPEthClient returns an HTTP ethclient if URL is valid

func SetupWebsocketEthClient

func SetupWebsocketEthClient(ethURL string, killChan <-chan struct{}, pingDelay int) (*ethclient.Client, error)

SetupWebsocketEthClient returns an websocket ethclient if URL is valid. Sets up a "command" ping if killChan is not nil and pingDelay > 0. If message is sent to killChan, stops the ping.

Types

type CrawlerConfig

type CrawlerConfig struct {
	EthAPIURL     string `envconfig:"eth_api_url" required:"true" desc:"Ethereum HTTP API address"`
	EthWsAPIURL   string `envconfig:"eth_ws_api_url" desc:"Ethereum Websocket API address (optional, disables watchers if empty)"`
	EthStartBlock uint64 `envconfig:"eth_start_block" desc:"Sets the start Eth block (default 0)" default:"0"`

	// Enables polling mode, disables websockets
	PollingEnabled      bool `envconfig:"polling_enabled" desc:"Enable polling only mode (true disables listeners)"`
	PollingIntervalSecs int  `envconfig:"polling_int_secs" desc:"Sets the polling interval"`

	// CivilListingsGraphqlURL enables call to retrieve newsroom listings from Civil.
	// Should pass in the URL to the GraphQL endpoint to enable.
	CivilListingsGraphqlURL string `envconfig:"civil_listing_graphql_url" desc:"URL of the Civil Listings GraphQL endpoint"`

	// ContractAddresses map a contract type to a string of contract addresses.  If there are more than 1
	// contract to be tracked for a particular type, delimit the addresses with '|'.
	ContractAddresses   map[string]string           `` /* 131-byte string literal not displayed */
	ContractAddressObjs map[string][]common.Address `ignored:"true"`

	PersisterType             cconfig.PersisterType `ignored:"true"`
	PersisterTypeName         string                `split_words:"true" required:"true" desc:"Sets the persister type to use"`
	PersisterPostgresAddress  string                `split_words:"true" desc:"If persister type is Postgresql, sets the address"`
	PersisterPostgresPort     int                   `split_words:"true" desc:"If persister type is Postgresql, sets the port"`
	PersisterPostgresDbname   string                `split_words:"true" desc:"If persister type is Postgresql, sets the database name"`
	PersisterPostgresUser     string                `split_words:"true" desc:"If persister type is Postgresql, sets the database user"`
	PersisterPostgresPw       string                `split_words:"true" desc:"If persister type is Postgresql, sets the database password"`
	PersisterPostgresMaxConns *int                  `split_words:"true" desc:"If persister type is Postgresql, sets the max conns in pool"`
	PersisterPostgresMaxIdle  *int                  `split_words:"true" desc:"If persister type is Postgresql, sets the max idle conns in pool"`
	PersisterPostgresConnLife *int                  `split_words:"true" desc:"If persister type is Postgresql, sets the max conn lifetime in secs"`

	PubSubProjectID string `split_words:"true" desc:"Sets the Google Cloud PubSub Project ID"`
	PubSubTopicName string `split_words:"true" desc:"Sets the Google Cloud PubSub Topic name"`

	// VersionNumber is the version of DB for postgres persistence
	VersionNumber string `split_words:"true" desc:"Sets the version for table"`

	// Configuration for supported error reporting (Stackdriver, Sentry)
	StackdriverProjectID      string `split_words:"true" desc:"Sets the Stackdriver Google Cloud project ID. If empty, will disable logging"`
	StackdriverServiceName    string `split_words:"true" desc:"Sets the Stackdriver service name"`
	StackdriverServiceVersion string `split_words:"true" desc:"Sets the Stackdriver service version"`

	SentryDsn        string `split_words:"true" desc:"Sets the Sentry DSN"`
	SentryEnv        string `split_words:"true" desc:"Sets the Sentry environment"`
	SentryLoggerName string `split_words:"true" desc:"Sets the Sentry logger name"`
	SentryRelease    string `split_words:"true" desc:"Sets the Sentry release value"`

	// Configuration for pprof profiling
	PprofEnable                bool   `split_words:"true" desc:"Enables the local pprof endpoints for debugging and profiling"`
	CloudProfileProjectID      string `split_words:"true" desc:"Sets the cloud profiler Google Cloud project ID. If empty, will disable cloud profiler agent"`
	CloudProfileServiceName    string `split_words:"true" desc:"Sets the service name to with in the Google Cloud profiler"`
	CloudProfileServiceVersion string `split_words:"true" desc:"Sets the Google Cloud profiler service version"`

	PreemptSecs *int `split_words:"true" desc:"Sets the secs delay before listener prempt restart. 0 if no preempt"`
}

CrawlerConfig is the master config for the crawler derived from environment variables.

func (*CrawlerConfig) Address

func (c *CrawlerConfig) Address() string

PostgresAddress returns the postgres persister address, implements PersisterConfig

func (*CrawlerConfig) DataVersion

func (c *CrawlerConfig) DataVersion() string

DataVersion returns the version number from the config

func (*CrawlerConfig) Dbname

func (c *CrawlerConfig) Dbname() string

PostgresDbname returns the postgres persister db name, implements PersisterConfig

func (*CrawlerConfig) FetchListingAddresses

func (c *CrawlerConfig) FetchListingAddresses() error

FetchListingAddresses retrieves the list of Civil newsroom addresses if given the endpoint URL

func (*CrawlerConfig) FetchMultiSigAddresses

func (c *CrawlerConfig) FetchMultiSigAddresses() error

FetchMultiSigAddresses retrieves the list of Civil multi sig addresses if given the endpoint URL

func (*CrawlerConfig) OutputUsage

func (c *CrawlerConfig) OutputUsage()

OutputUsage prints the usage string to os.Stdout

func (*CrawlerConfig) Password

func (c *CrawlerConfig) Password() string

PostgresPw returns the postgres persister password, implements PersisterConfig

func (*CrawlerConfig) PersistType

func (c *CrawlerConfig) PersistType() cconfig.PersisterType

PersistType returns the persister type, implements PersisterConfig

func (*CrawlerConfig) PoolConnLifetimeSecs

func (c *CrawlerConfig) PoolConnLifetimeSecs() *int

PoolConnLifetimeSecs returns the conn lifetime for a pool, if configured, implements PersisterConfig

func (*CrawlerConfig) PoolMaxConns

func (c *CrawlerConfig) PoolMaxConns() *int

PoolMaxConns returns the max conns for a pool, if configured, implements PersisterConfig

func (*CrawlerConfig) PoolMaxIdleConns

func (c *CrawlerConfig) PoolMaxIdleConns() *int

PoolMaxIdleConns returns the max idleconns for a pool, if configured, implements PersisterConfig

func (*CrawlerConfig) PopulateFromEnv

func (c *CrawlerConfig) PopulateFromEnv() error

PopulateFromEnv processes the environment vars, populates CrawlerConfig with the respective values, and validates the values.

func (*CrawlerConfig) Port

func (c *CrawlerConfig) Port() int

PostgresPort returns the postgres persister port, implements PersisterConfig

func (*CrawlerConfig) User

func (c *CrawlerConfig) User() string

PostgresUser returns the postgres persister user, implements PersisterConfig

type Edge

type Edge struct {
	Node struct {
		Name            graphql.String
		ContractAddress graphql.String
		Whitelisted     graphql.Boolean
		ApprovalDate    graphql.Int
		LastGovState    graphql.String
	}
}

Edge represents an edge field in the query

type PageInfo

type PageInfo struct {
	EndCursor   graphql.String
	HasNextPage graphql.Boolean
}

PageInfo represents a pageinfo field in the query

type RebuildConfig

type RebuildConfig struct {
	PersisterType            cconfig.PersisterType `ignored:"true"`
	PersisterPostgresAddress string                `split_words:"true" required:"true" desc:"If persister type is Postgresql, sets the address"`
	PersisterPostgresPort    int                   `split_words:"true" required:"true" desc:"If persister type is Postgresql, sets the port"`
	PersisterPostgresDbname  string                `split_words:"true" required:"true" desc:"If persister type is Postgresql, sets the database name"`
	PersisterPostgresUser    string                `split_words:"true" required:"true" desc:"If persister type is Postgresql, sets the database user"`
	PersisterPostgresPw      string                `split_words:"true" required:"true" desc:"If persister type is Postgresql, sets the database password"`
}

RebuildConfig is the master config for the rebuild script derived from environment variables.

func (*RebuildConfig) OutputUsage

func (r *RebuildConfig) OutputUsage()

OutputUsage prints the usage string to os.Stdout

func (*RebuildConfig) PopulateFromEnv

func (r *RebuildConfig) PopulateFromEnv() error

PopulateFromEnv processes the environment vars, populates RebuildConfig with the respective values, and validates the values.

type WatcherSubscription

type WatcherSubscription interface {
	Err() <-chan error
	Unsubscribe()
}

WatcherSubscription is represents a subscription to a stream of watcher events. This is a modification/custom version of the Subscription struct in the go-ethereum package, so much credit to the authors there. https://github.com/ethereum/go-ethereum/blob/master/event/subscription.go

func NewWatcherSubscription

func NewWatcherSubscription(watcherName string, producer func(<-chan struct{}) error) WatcherSubscription

NewWatcherSubscription runs a producer function as a watcher subscription in a new goroutine. The channel given to the producer is closed when Unsubscribe is called. Returns error returned by producer to the subscription error channel. This is a modification/custom version of the NewSubscription struct in the go-ethereum package, so much credit to the authors there. https://github.com/ethereum/go-ethereum/blob/master/event/subscription.go Differs in how it handles completion of the producer function with separate channel to prevent deadlocks.