storage

Documentation

Overview

Package storage of the Dataflow kit describes Store interface for read/ write operations with downloaded data and parsed results.

Storage types like AWS S3, Digital Ocean Spaces, Diskv, Redis implement methods wchich satisfy Store interface.

Index

• Constants
• type DiskvConn
  • func (d DiskvConn) Close()
  • func (d DiskvConn) Delete(rec Record) error
  • func (d DiskvConn) DeleteAll() error
  • func (d DiskvConn) Expired(rec Record) bool
  • func (d DiskvConn) Read(rec Record) (value []byte, err error)
  • func (d DiskvConn) Write(rec Record) error
• type Record
• type Store
  • func NewStore(sType string) Store

Constants

const (
	//Storage types
	CACHE        = "Cache"
	COOKIES      = "Cookies"
	INTERMEDIATE = "Intermediate"
)

Types

type DiskvConn

type DiskvConn struct {
	// contains filtered or unexported fields
}

DiskvConn stores connection parameters for Diskv storage

func (DiskvConn) Close

func (d DiskvConn) Close()

Close storage connection

func (DiskvConn) Delete

func (d DiskvConn) Delete(rec Record) error

Delete deletes specified key from Diskv storage.

func (DiskvConn) DeleteAll

func (d DiskvConn) DeleteAll() error

DeleteAll deletes everything from Diskv storage.

func (DiskvConn) Expired

func (d DiskvConn) Expired(rec Record) bool

Expired returns Expired value of specified key from DiskV.

func (DiskvConn) Read

func (d DiskvConn) Read(rec Record) (value []byte, err error)

Read loads value according to the specified key from DiskV KV storage.

func (DiskvConn) Write

func (d DiskvConn) Write(rec Record) error

Write stores key/ value pair along with Expiration time to DiskV KV storage.

type Record

type Record struct {
	Type    string
	Key     string
	Value   []byte
	ExpTime int64
}

Record struct keeps Key/Value and expiration time of specified type

type Store

type Store interface {
	//Reads value from storage by specified key
	Read(rec Record) (value []byte, err error)
	//Writes specified pair key value to storage.
	//expTime value sets TTL for Redis storage.
	//expTime set Metadata Expires value for S3Storage
	Write(rec Record) error
	//Is key expired ? It checks if parse results storage item is expired. Set up  Expiration as "ITEM_EXPIRE_IN" environment variable.
	//html pages cache stores this info in sResponse.Expires . It is not used for fetch endpoint.
	Expired(rec Record) bool
	//Delete deletes specified item from the store
	Delete(rec Record) error
	//DeleteAll erases all items from the store
	DeleteAll() error
	// Close storage connection
	Close()
}

Store is the key interface of storage. All other structs implement methods wchich satisfy that interface.

func NewStore

func NewStore(sType string) Store

NewStore creates New initialized Store instance with predefined parameters Storage Types: S3, Spaces, Redis, Diskv, Cassandra