indexer

package
v0.3.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 25, 2020 License: GPL-3.0 Imports: 52 Imported by: 3

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GetAllPagesFromIndex added in v0.2.1 

func GetAllPagesFromIndex(facade *Facade, query *torznab.Query) <-chan search.ExternalResultItem

IteratePages goes over all the pages in an index and returns the results through a channel.

func ListBuiltInIndexes added in v0.2.1 

func ListBuiltInIndexes() ([]string, error)

ListBuiltInIndexes returns a list of all the embedded indexes that are supported.

func Watch 

func Watch(facade *Facade, initialQuery *torznab.Query, intervalSec int) <-chan search.ExternalResultItem

Watch tracks an index for any new items, through all search pages(or max pages). Whenever old results are found, or we've exhausted the number of pages, the search restarts from the start. The interval is in seconds, it's used to sleep after each search for new results.

Types

type Aggregate 

type Aggregate struct {
	Indexers []Indexer
}

func NewAggregate 

func NewAggregate(indexers []Indexer) *Aggregate

CreateWithId a new aggregate from the given indexers

func (*Aggregate) Capabilities 

func (ag *Aggregate) Capabilities() torznab.Capabilities

func (*Aggregate) Check 

func (ag *Aggregate) Check() error

Check checks all indexers, if they can be searched.

func (*Aggregate) Download 

func (ag *Aggregate) Download(u string) (io.ReadCloser, error)

func (*Aggregate) GetEncoding 

func (ag *Aggregate) GetEncoding() string

func (*Aggregate) Info 

func (ag *Aggregate) Info() Info

func (*Aggregate) MaxSearchPages 

func (ag *Aggregate) MaxSearchPages() uint

MaxSearchPages returns the maximum number of pages that this aggregate can search, this is using the maximum paged index in the aggregate.

func (*Aggregate) Open 

func (ag *Aggregate) Open(s *search.ExternalResultItem) (io.ReadCloser, error)

func (*Aggregate) ProcessRequest 

func (ag *Aggregate) ProcessRequest(req *http.Request) (*http.Response, error)

func (*Aggregate) Search 

func (ag *Aggregate) Search(query *torznab.Query, srch search.Instance) (search.Instance, error)

func (*Aggregate) SearchIsSinglePaged 

func (ag *Aggregate) SearchIsSinglePaged() bool

SearchIsSinglePaged this is true only if all indexes inside the aggregate are single paged.

type AggregateInfo 

type AggregateInfo struct{}

func (*AggregateInfo) GetId 

func (a *AggregateInfo) GetId() string

func (*AggregateInfo) GetLanguage 

func (a *AggregateInfo) GetLanguage() string
func (a *AggregateInfo) GetLink() string

func (*AggregateInfo) GetTitle 

func (a *AggregateInfo) GetTitle() string

type AssetLoader 

type AssetLoader struct {
	Names    []string
	Resolver DefinitionDataLoader
}

func (*AssetLoader) List 

func (l *AssetLoader) List() ([]string, error)

List all the names of the embedded definitions

func (*AssetLoader) Load 

func (l *AssetLoader) Load(key string) (*IndexerDefinition, error)

Load a definition with a given name

type CachedScope 

type CachedScope struct {
	// contains filtered or unexported fields
}

func (*CachedScope) CreateAggregate 

func (c *CachedScope) CreateAggregate(config config.Config) (Indexer, error)

CreateAggregate gets you an aggregate of all the valid configured indexers this includes indexers that don't need a login.

func (*CachedScope) CreateAggregateForCategories 

func (c *CachedScope) CreateAggregateForCategories(config config.Config, cats []categories.Category) (Indexer, error)

CreateAggregateForCategories creates a new aggregate with the indexers that match a set of categories

func (*CachedScope) Lookup 

func (c *CachedScope) Lookup(config config.Config, key string) (Indexer, error)

Lookup finds the matching Indexer.

type DefinitionDataLoader 

type DefinitionDataLoader func(key string) ([]byte, error)

An indexer definition loader that uses embedded definitions.

type DefinitionLoader 

type DefinitionLoader interface {
	//List - Lists available trackers.
	List() ([]string, error)
	//Load - Load a definition of an Indexer from it's name
	Load(key string) (*IndexerDefinition, error)
}

DefinitionLoader loads an index definition by name or lists the names of the supported indexes. 

var (
	ErrUnknownIndexer       = errors.New("unknown indexer")
	DefaultDefinitionLoader DefinitionLoader
	Loader                  DefinitionLoader
)

func CreateEmbeddedDefinitionSource 

func CreateEmbeddedDefinitionSource(definitionNames []string, loader DefinitionDataLoader) DefinitionLoader

CreateEmbeddedDefinitionSource creates a new definition loader from a set of resource names and a loader.

type Facade 

type Facade struct {
	//The indexer that we're using
	Indexer Indexer
	//Configuration for the indexer.
	Config config.Config
	Scope  Scope
}

Facade for an indexer/aggregate, helps manage the scope of the index, it's configuration and the index itself.

func NewAggregateFacadeWithCategories added in v0.2.1 

func NewAggregateFacadeWithCategories(config config.Config, cats ...categories.Category) *Facade

NewAggregateFacadeWithCategories Finds an indexer from the config, that matches the given categories.

func NewEmptyFacade 

func NewEmptyFacade(config config.Config) *Facade

NewEmptyFacade creates a new indexer facade with it's own scope and config.

func NewFacade 

func NewFacade(indexerName string, config config.Config, cats ...categories.Category) (*Facade, error)

NewFacade Creates a new facade for an indexer with the given name and config. If any categories are given, the facade must be for an indexer that supports these categories. If you don't provide a name or name is `all`, an aggregate is used.

func NewFacadeFromConfiguration 

func NewFacadeFromConfiguration(config config.Config) *Facade

NewFacadeFromConfiguration Creates a new facade using the configuration

func (*Facade) GetDefaultOptions 

func (th *Facade) GetDefaultOptions() *GenericSearchOptions

GetDefaultOptions gets the default search options

func (*Facade) Search 

func (th *Facade) Search(searchContext search.Instance, query *torznab.Query) (search.Instance, error)

Search using a given query. The search covers only 1 page.

func (*Facade) SearchKeywords 

func (th *Facade) SearchKeywords(searchContext search.Instance, query string, page uint) (search.Instance, error)

SearchKeywords performs a search for a given page

func (*Facade) SearchKeywordsWithCategory 

func (th *Facade) SearchKeywordsWithCategory(searchContext search.Instance, query string, cat categories.Category, page uint) (search.Instance, error)

SearchKeywordsWithCategory Search for *keywords* matching the needed category.

type FileIndexLoader 

type FileIndexLoader struct {
	Directories []string
}

func NewFsLoader 

func NewFsLoader(appName string) *FileIndexLoader

NewFsLoader creates a new file index loader which looks for definitions in ~/.#{appName}/indexes and ./indexes

func (*FileIndexLoader) List 

func (fs *FileIndexLoader) List() ([]string, error)

func (*FileIndexLoader) Load 

func (fs *FileIndexLoader) Load(key string) (*IndexerDefinition, error)

Load - Load a definition of an Indexer from it's name

func (*FileIndexLoader) String 

func (fs *FileIndexLoader) String() string

type GenericSearchOptions 

type GenericSearchOptions struct {
	//The count of pages we can fetch
	PageCount uint
	//The initial search page
	StartingPage         uint
	MaxRequestsPerSecond uint
	StopOnStaleResults   bool
}

GenericSearchOptions options for the search.

type Indexer 

type Indexer interface {
	Info() Info
	Search(query *torznab.Query, srch search.Instance) (search.Instance, error)
	Download(urlStr string) (io.ReadCloser, error)
	Capabilities() torznab.Capabilities
	GetEncoding() string
	ProcessRequest(req *http.Request) (*http.Response, error)
	Open(s *search.ExternalResultItem) (io.ReadCloser, error)
	//Check if the Indexer works ok.
	//This might be needed to validate the search result extraction.
	Check() error
	//The maximum number of pages we can search
	MaxSearchPages() uint
	SearchIsSinglePaged() bool
}

func CreateIndexer 

func CreateIndexer(config config.Config, indexerName string) (Indexer, error)

CreateIndexer creates a new Indexer or aggregate Indexer with the given configuration.

type IndexerDefinition 

type IndexerDefinition struct {
	Site         string            `yaml:"site"`
	Settings     []settingsField   `yaml:"settings"`
	Name         string            `yaml:"name"`
	Description  string            `yaml:"description"`
	Language     string            `yaml:"language"`
	Links        stringorslice     `yaml:"links"`
	Capabilities capabilitiesBlock `yaml:"caps"`
	Login        loginBlock        `yaml:"login"`
	Ratio        ratioBlock        `yaml:"ratio"`
	Search       searchBlock       `yaml:"search"`

	Encoding string `yaml:"encoding"`
	//Entities that the index contains
	Entities []entityBlock `yaml:"entities"`
	//The ms to wait between each request.
	RateLimit int `yaml:"ratelimit"`
	// contains filtered or unexported fields
}

func LoadEnabledDefinitions 

func LoadEnabledDefinitions(conf interface{}) ([]*IndexerDefinition, error)

LoadEnabledDefinitions loads all of the definitions that are covered by the current definition loader (`Loader`).

func ParseDefinition 

func ParseDefinition(src []byte) (*IndexerDefinition, error)

func ParseDefinitionFile 

func ParseDefinitionFile(f *os.File) (*IndexerDefinition, error)

ParseDefinitionFile loads an Indexer's definition from a file

func (*IndexerDefinition) Stats 

func (id *IndexerDefinition) Stats() IndexerDefinitionStats

type IndexerDefinitionStats 

type IndexerDefinitionStats struct {
	Size    int64
	ModTime time.Time
	Hash    string
	Source  string
}

type IndexerInfo 

type IndexerInfo struct {
	ID       string
	Title    string
	Language string
	Link     string
}

func (IndexerInfo) GetId 

func (i IndexerInfo) GetId() string

func (IndexerInfo) GetLanguage 

func (i IndexerInfo) GetLanguage() string
func (i IndexerInfo) GetLink() string

func (IndexerInfo) GetTitle 

func (i IndexerInfo) GetTitle() string

type IndexerState 

type IndexerState struct {
	// contains filtered or unexported fields
}

func (*IndexerState) GetBool 

func (is *IndexerState) GetBool(key string) bool

func (*IndexerState) Has 

func (is *IndexerState) Has(key string) bool

func (*IndexerState) Set 

func (is *IndexerState) Set(key string, val interface{})

type Info 

type Info interface {
	GetId() string
	GetTitle() string
	GetLanguage() string
	GetLink() string
}

type LoginError 

type LoginError struct {
	// contains filtered or unexported fields
}

func (*LoginError) Error 

func (e *LoginError) Error() string

type MultipleDefinitionLoader 

type MultipleDefinitionLoader []DefinitionLoader

func (MultipleDefinitionLoader) List 

func (ml MultipleDefinitionLoader) List() ([]string, error)

func (MultipleDefinitionLoader) Load 

func (ml MultipleDefinitionLoader) Load(key string) (*IndexerDefinition, error)

Load an indexer with the matching name

type RunContext 

type RunContext struct {
	Search *search.Search
}

type Runner 

type Runner struct {
	Storage storage.ItemStorage
	// contains filtered or unexported fields
}

Runner works index definitions in order to extract data.

func NewRunner 

func NewRunner(def *IndexerDefinition, opts RunnerOpts) *Runner

NewRunner Start a runner for a given indexer.

func (*Runner) CachePage added in v0.2.2 

func (r *Runner) CachePage(browsable browser.Browsable) error

If caching is enabled, we cache the page's contents in our pagecache the current browser page is cached

func (*Runner) Capabilities 

func (r *Runner) Capabilities() torznab.Capabilities

Capabilities gets the torznab formatted capabilities of this Indexer.

func (*Runner) Check 

func (r *Runner) Check() error

Check sees if the index can be searched

func (*Runner) Download 

func (r *Runner) Download(u string) (io.ReadCloser, error)

func (*Runner) GetEncoding 

func (r *Runner) GetEncoding() string

GetEncoding returns the encoding that's set to be used in this index. This can be changed in the index's definition.

func (*Runner) Info 

func (r *Runner) Info() Info

func (*Runner) MaxSearchPages 

func (r *Runner) MaxSearchPages() uint

func (*Runner) Open 

func (r *Runner) Open(s *search.ExternalResultItem) (io.ReadCloser, error)

func (*Runner) ProcessRequest 

func (r *Runner) ProcessRequest(req *http.Request) (*http.Response, error)

func (*Runner) Ratio 

func (r *Runner) Ratio() (string, error)

func (*Runner) Search 

func (r *Runner) Search(query *torznab.Query, srch search.Instance) (search.Instance, error)

SearchKeywords for a given torrent

func (*Runner) SearchIsSinglePaged 

func (r *Runner) SearchIsSinglePaged() bool

type RunnerOpts 

type RunnerOpts struct {
	Config     config.Config
	CachePages bool
	Transport  http.RoundTripper
}

type RunnerPatternData 

type RunnerPatternData struct {
	Query      *torznab.Query
	Keywords   string
	Categories []string
	Context    RunContext
}

type Scope 

type Scope interface {
	Lookup(config config.Config, key string) (Indexer, error)
	CreateAggregateForCategories(config config.Config, cats []categories.Category) (Indexer, error)
}

func NewScope 

func NewScope() Scope

NewScope creates a new scope for indexer runners

Source Files

View all Source files

Directories

Path Synopsis
mocks
Package mocks is a generated GoMock package.
 Package mocks is a generated GoMock package.
Package mocks is a generated GoMock package.
 Package mocks is a generated GoMock package.
mocks
Package mocks is a generated GoMock package.
 Package mocks is a generated GoMock package.
web

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.