Documentation
¶
Index ¶
- func Delayrequest(v1 *viper.Viper) (int64, error)
- func GetDriveCredentials(authFilename string) (srv *drive.Service, err error)
- func GetFileFromGDrive(srv *drive.Service, fileId string) (*drive.File, string, error)
- func GetFileList(srv *drive.Service, parentFolder string, isSharedDrive bool, ...) ([]*drive.File, error)
- func GetFromGDrive(mc *minio.Client, v1 *viper.Viper) (string, error)
- func GetGraph(mc *minio.Client, v1 *viper.Viper) (string, error)
- func HeadlessNG(v1 *viper.Viper, mc *minio.Client, m map[string][]string, db *bolt.DB)
- func PageRender(v1 *viper.Viper, mc *minio.Client, logger *log.Logger, timeout time.Duration, ...) error
- func ResRetrieve(v1 *viper.Viper, mc *minio.Client, m map[string][]string, db *bolt.DB)
- func ResourceURLs(v1 *viper.Viper, mc *minio.Client, headless bool, db *bolt.DB) map[string][]string
- func StoreProv(v1 *viper.Viper, mc *minio.Client, k, sha, urlloc string) error
- func StoreProvNG(v1 *viper.Viper, mc *minio.Client, k, sha, urlloc, objprefix string) error
- func Threadcount(v1 *viper.Viper) (int64, error)
- func ThreadedHeadlessNG(v1 *viper.Viper, mc *minio.Client, m map[string][]string, db *bolt.DB)
- func Upload(v1 *viper.Viper, mc *minio.Client, logger *log.Logger, bucketName string, ...) (string, error)
- type ProvData
- type Sources
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func GetDriveCredentials ¶
func GetFileFromGDrive ¶
We get this: but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.
https://support.google.com/websearch/answer/86640
<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title> Sorry...</title><style> body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }</style></head><body><div><table><tr><td><b><font face=sans-serif size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font color=#fbbc05>o</font><font color=#4285f4>g</font><font color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td style="text-align: left; vertical-align: bottom; padding-bottom: 15px; width: 50%"><div style="border-bottom: 1px solid #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.</p></div><div style="margin-left: 4em;">See <a href="https://support.google.com/websearch/answer/86640">Google Help</a> for more information.<br/><br/></div><div style="text-align: center; border-top: 1px solid #dfdfdf;"><a href="https://www.google.com">Google Home</a></div></body></html>
func GetFileList ¶
func HeadlessNG ¶
HeadlessNG gets schema.org entries in sites that put the JSON-LD in dynamically with JS. It uses a chrome headless instance (which MUST BE RUNNING). TODO.. trap out error where headless is NOT running
func PageRender ¶
func ResRetrieve ¶
ResRetrieve is a function to pull down the data graphs at resources
func ResourceURLs ¶
func ResourceURLs(v1 *viper.Viper, mc *minio.Client, headless bool, db *bolt.DB) map[string][]string
ResourceURLs looks gets the resource URLs for a domain. The results is a map with domain name as key and []string of the URLs to process.
func StoreProv ¶
StoreProv creates and stores a prov record for each JSON-LD data graph k is the domain / provider sha is the sha of the JSON-LD file summoned urlloc is the URL for the resource (source URL)
func StoreProvNG ¶
func ThreadedHeadlessNG ¶
ThreadedHeadlessNG does not work.. ;)
Types ¶
type ProvData ¶
type ProvData struct {
RESID string
SHA256 string
PID string
SOURCE string
DATE string
RUNID string
URN string
PNAME string
DOMAIN string
}
ProvData is the struct holding the prov data for a summoned data graph
type Sources ¶
type Sources = configTypes.Sources
Source Files