storage

Documentation

Overview

Package storage of the Dataflow kit describes Store interface for read/ write operations with downloaded data and parsed results.

Storage types like Diskv, MongoDB implement methods which satisfy Store interface.

Index

• Constants
• type Record
• type Store
  • func NewStore(sType string) Store

Constants

const (
	CACHE        = "Cache"
	COOKIES      = "Cookies"
	INTERMEDIATE = "Intermediate"
)

Storage types

Types

type Record

type Record struct {
	Type    string
	Key     string
	Value   []byte
	ExpTime int64
}

Record struct keeps Key/Value and expiration time of specified type

type Store

type Store interface {
	//Reads value from storage by specified key
	Read(rec Record) (value []byte, err error)
	//Writes specified pair key value to storage.
	//expTime value sets TTL for Redis storage.
	//expTime set Metadata Expires value for S3Storage
	Write(rec Record) error
	// IsExists checkes whether specified by key record exists
	IsExists(rec Record) bool
	//Is key expired ? It checks if parse results storage item is expired. Set up  Expiration as "ITEM_EXPIRE_IN" environment variable.
	//html pages cache stores this info in sResponse.Expires . It is not used for fetch endpoint.
	Expired(rec Record) bool
	//Delete deletes specified item from the store
	Delete(rec Record) error
	//DeleteAll erases all items from the store
	DeleteAll() error
	// Close storage connection
	Close()
}

Store is the key interface of storage. All other structs implement methods wchich satisfy that interface.

func NewStore

func NewStore(sType string) Store

NewStore creates New initialized Store instance with predefined parameters Storage Types: S3, Spaces, Redis, Diskv