store

Documentation

Overview

Package store contains functions for writing parsed feeds to cloud storage.

Index

• func GetSession(creds *AWSCredentials) (sesh *session.Session, err error)
• func MustOpen(path string) *leveldb.DB
• func Upload(s *session.Session, doc Document, bucket string) error
• func VerifyCredentials(creds *AWSCredentials) bool
• type AWSCredentials
• type Document
• type Feed

Functions

func GetSession

func GetSession(creds *AWSCredentials) (sesh *session.Session, err error)

GetSession is a helper function that returns an AWS session that can be reused for multiple writes. It takes in a pointer to an AWSCredentials struct and returns a pointer to an AWS connection and a nil, else a nil and the connection error.

func MustOpen

func MustOpen(path string) *leveldb.DB

MustOpen either initializes a new leveldb database at the path of the string provided or opens an existing leveldb database found at that path.

func Upload

func Upload(s *session.Session, doc Document, bucket string) error

Upload a single Document to the specified S3 bucket using the established AWS session, setting file information including name (the feedID, language code, year, month, day, and hash of the content), the content size and type, and the encryption on the uploaded file.

func VerifyCredentials

func VerifyCredentials(creds *AWSCredentials) bool

VerifyCredentials is a helper function that verifies the credentials are correct and the bucket exists, returning true if so else false.

Types

type AWSCredentials

type AWSCredentials struct {
	Region string
	Bucket string
}

AWSCredentials stores the region and bucket needed to access an S3 bucket.

type Document

type Document struct {
	FeedID       string
	LanguageCode string
	Year         int
	Month        string
	Day          int
	Title        string
	Description  string
	Content      []byte
	Encoding     string `json:",omitempty"`
	Link         string
}

A Document is the generic representation of an individual entry in a feed

type Feed

type Feed struct {
	URL    string `json:"url"`
	Active bool   `json:"active"`
	Error  string `json:"error"`
}