collyzstandard

README

collyzstandard

collyzstandard is a Go package providing an experimental work-in-progress prototype of a cache compressor for Colly.

Installation

$ go get github.com/jimsmart/collyzstandard

import "github.com/jimsmart/collyzstandard"

Dependencies

• Zstd — Go wrapper for Facebook's fast real-time compression algorithm.
• Standard library.
• Ginkgo and Gomega if you wish to run the tests.

Example

import "github.com/jimsmart/collysqlite"
import "github.com/jimsmart/collyzstandard"

cache := collysqlite.NewCache("./cache")
compcache := collyzstandard.NewCompressor(cache)

// TODO SetCache is a proposed method on colly.Collector that is currently unimplemented.
// c := colly.NewCollector()
// c.SetCache(compcache)
// ...

Documentation

GoDocs https://godoc.org/github.com/jimsmart/collyzstandard

Testing

To run the tests execute go test inside the project folder.

For a full coverage report, try:

$ go test -coverprofile=coverage.out && go tool cover -html=coverage.out

License

Package collyzstandard is copyright 2018 by Jim Smart and released under the MIT License

Documentation

Index

• Constants
• type CollyCache
• type Compressor
  • func NewCompressor(cache CollyCache) *Compressor
  • func (c *Compressor) Get(url string) ([]byte, error)
  • func (c *Compressor) Init() error
  • func (c *Compressor) Put(url string, data []byte) error
  • func (c *Compressor) Remove(url string) error

Constants

const DefaultCompressionLevel = zstd.BestCompression

DefaultCompressionLevel used, where fastest = 1, best = 20, default = 5.

Types

type CollyCache

type CollyCache interface {
	Init() error
	Get(url string) ([]byte, error)
	Put(url string, data []byte) error
	Remove(url string) error
}

CollyCache is the proposed interface for pluggable cache implementations in Colly.

type Compressor

type Compressor struct {
	Level   int
	Cache   CollyCache
	Logging bool
}

Compressor wraps a CollyCache to provide data compression using the Zstandard algorithm.

func NewCompressor

func NewCompressor(cache CollyCache) *Compressor

NewCompressor creates wraps the given CollyCache with a Compressor, configured to use the default (maximum) compression level.

func (*Compressor) Get

func (c *Compressor) Get(url string) ([]byte, error)

func (*Compressor) Init

func (c *Compressor) Init() error

func (*Compressor) Put

func (c *Compressor) Put(url string, data []byte) error

func (*Compressor) Remove

func (c *Compressor) Remove(url string) error