preprocess

Documentation

Overview

Package preprocess handles preprocessing and transformation of queries.

Index

• func AlphaNum(text string) string
• func Lowercase(text string) string
• func ProcessQuery(query cqr.CommonQueryRepresentation, processor QueryProcessor) cqr.CommonQueryRepresentation
• func StripNumbers(text string) string
• type BooleanTransformation
  • func DateRestrictions(pubDatesFile string) BooleanTransformation
• type ElasticsearchTransformation
• type QueryProcessor
• type QueryTransformations
• type Transformation
  • func Analyse(query cqr.CommonQueryRepresentation, ...) Transformation
  • func AndSimplify(query cqr.CommonQueryRepresentation, topic string) Transformation
  • func OrSimplify(query cqr.CommonQueryRepresentation, topic string) Transformation
  • func RCTFilter(query cqr.CommonQueryRepresentation, _ string) Transformation
  • func RelaxPhrases(query cqr.CommonQueryRepresentation, topic string) Transformation
  • func RemoveExplosionMeSH(query cqr.CommonQueryRepresentation, topic string) Transformation
  • func SetAnalyseField(query cqr.CommonQueryRepresentation, ...) Transformation
  • func Simplify(query cqr.CommonQueryRepresentation, topic string) Transformation
• type Transformations
• type TransformedQueries

Functions

func AlphaNum

func AlphaNum(text string) string

AlphaNum removes all non-alphanumeric characters from a query.

func Lowercase

func Lowercase(text string) string

Lowercase transforms all capital letters to lowercase.

func ProcessQuery

func ProcessQuery(query cqr.CommonQueryRepresentation, processor QueryProcessor) cqr.CommonQueryRepresentation

ProcessQuery applies a query processor to a query.

func StripNumbers

func StripNumbers(text string) string

StripNumbers removes numbers from a query.

Types

type BooleanTransformation

type BooleanTransformation func(q cqr.CommonQueryRepresentation, topic string) Transformation

BooleanTransformation is a transformation that can be made to a Boolean query.

func DateRestrictions

func DateRestrictions(pubDatesFile string) BooleanTransformation

DateRestrictions loads a file in the format:

CD008122	19400101	20100114
CD008587	19920101	20151130
CD008759	19460101	20160630
CD008892	19660101	20160301
CD009175	19500101	20120626
CD009263	19400101	20120921
CD009694	19920101	20120831
CD010213	19460101	20160719
CD010296	19460101	20160701
CD010502	19800101	20130201
CD010657	19400101	20160331

Where the fist column is the topic of the query, and the other two columns are the start and end dates of the restriction.

type ElasticsearchTransformation

type ElasticsearchTransformation func(query cqr.CommonQueryRepresentation, source *stats.ElasticsearchStatisticsSource) Transformation

ElasticsearchTransformation is a specific transformation that uses an Elasticsearch statistics source.

type QueryProcessor

type QueryProcessor func(text string) string

QueryProcessor is applied to queries before analysing/measuring.

type QueryTransformations

type QueryTransformations struct {
	BooleanTransformations       []BooleanTransformation
	ElasticsearchTransformations []ElasticsearchTransformation
	Output                       string
}

QueryTransformations is the information needed to perform query transformations.

type Transformation

type Transformation func() cqr.CommonQueryRepresentation

Transformation is an major modification to a query.

func Analyse

func Analyse(query cqr.CommonQueryRepresentation, source *stats.ElasticsearchStatisticsSource) Transformation

Analyse runs the specified Elasticsearch analyser on a query and returns a new, analysed query.

func AndSimplify

func AndSimplify(query cqr.CommonQueryRepresentation, topic string) Transformation

AndSimplify replaces all operators with `and`.

func OrSimplify

func OrSimplify(query cqr.CommonQueryRepresentation, topic string) Transformation

OrSimplify replaces all operators with `or`.

func RCTFilter

func RCTFilter(query cqr.CommonQueryRepresentation, _ string) Transformation

RCTFilter adds a randomised controlled trials filter to queries. randomized controlled trial.pt. controlled clinical trial.pt. randomized.ab. placebo.ab. clinical trials as topic.sh. randomly.ab. trial.ti.

func RelaxPhrases

func RelaxPhrases(query cqr.CommonQueryRepresentation, topic string) Transformation

RelaxPhrases replaces phrase queries with OR clauses.

func RemoveExplosionMeSH

func RemoveExplosionMeSH(query cqr.CommonQueryRepresentation, topic string) Transformation

RemoveExplosionMeSH removes the explosion from MeSH terms.

func SetAnalyseField

func SetAnalyseField(query cqr.CommonQueryRepresentation, source *stats.ElasticsearchStatisticsSource) Transformation

SetAnalyseField sets the text and title fields to be analysed by the specified analyser.

func Simplify

func Simplify(query cqr.CommonQueryRepresentation, topic string) Transformation

Simplify replaces anything that is not an `and` operator with and `or` operator.

type Transformations

type Transformations []Transformation

Transformations is a collection of transformation operations.

type TransformedQueries

type TransformedQueries []cqr.CommonQueryRepresentation

TransformedQueries is a collection of transformed queries.