minquery

README

minquery

MongoDB / mgo query that supports efficient pagination (cursors to continue listing documents where we left off).

Note: Only MongoDB 3.2 and newer versions support the feature used by this package.

Note #2: minquery v1.0.0 uses the gopkg.in/mgo.v2 mgo driver which has gone unmaintained for a long time now. minquery v2.0.0 (tip of master) uses the new, community supported fork github.com/globalsign/mgo. It is highly recommended to switch over to globalsign/mgo. If you can't or don't want to, you may continue to use the v1.0.0 release with gopkg.in/mgo.v2.

Introduction

Let's say we have a users collection in MongoDB modeled with this Go struct:

type User struct {
    ID      bson.ObjectId `bson:"_id"`
    Name    string        `bson:"name"`
    Country string        `bson:"country"`
}

To achieve paging of the results of some query, MongoDB and the mgo driver package has built-in support in the form of Query.Skip() and Query.Limit(), e.g.:

session, err := mgo.Dial(url) // Acquire Mongo session, handle error!

c := session.DB("").C("users")
q := c.Find(bson.M{"country" : "USA"}).Sort("name", "_id").Limit(10)

// To get the nth page:
q = q.Skip((n-1)*10)

var users []*User
err = q.All(&users)

This however becomes slow if the page number increases, as MongoDB can't just "magically" jump to the x^th document in the result, it has to iterate over all the result documents and omit (not return) the first x that need to be skipped.

MongoDB provides the right solution: If the query operates on an index (it has to work on an index), cursor.min() can be used to specify the first index entry to start listing results from.

This Stack Overflow answer shows how it can be done using a mongo client: How to do pagination using range queries in MongoDB?

Note: the required index for the above query would be:

db.users.createIndex(
    {
        country: 1,
        name: 1,
        _id: 1
    }
)

There is one problem though: the mgo package has no support specifying this min().

Introducing minquery

Unfortunately the mgo driver does not provide API calls to specify cursor.min().

But there is a solution. The mgo.Database type provides a Database.Run() method to run any MongoDB commands. The available commands and their documentation can be found here: Database commands

Starting with MongoDB 3.2, a new find command is available which can be used to execute queries, and it supports specifying the min argument that denotes the first index entry to start listing results from.

Good. What we need to do is after each batch (documents of a page) generate the min document from the last document of the query result, which must contain the values of the index entry that was used to execute the query, and then the next batch (the documents of the next page) can be acquired by setting this min index entry prior to executing the query.

This index entry –let's call it cursor from now on– may be encoded to a string and sent to the client along with the results, and when the client wants the next page, he sends back the cursor saying he wants results starting after this cursor.

And this is where minquery comes into the picture. It provides a wrapper to configure and execute a MongoDB find command, allowing you to specify a cursor, and after executing the query, it gives you back the new cursor to be used to query the next batch of results. The wrapper is the MinQuery type which is very similar to mgo.Query but it supports specifying MongoDB's min via the MinQuery.Cursor() method.

The above solution using minquery looks like this:

q := minquery.New(session.DB(""), "users", bson.M{"country" : "USA"}).
    Sort("name", "_id").Limit(10)
// If this is not the first page, set cursor:
// getLastCursor() represents your logic how you acquire the last cursor.
if cursor := getLastCursor(); cursor != "" {
    q = q.Cursor(cursor)
}

var users []*User
newCursor, err := q.All(&users, "country", "name", "_id")

And that's all. newCursor is the cursor to be used to fetch the next batch.

Note #1: When calling MinQuery.All(), you have to provide the names of the cursor fields, this will be used to build the cursor data (and ultimately the cursor string) from.

Note #2: If you're retrieving partial results (by using MinQuery.Select()), you have to include all the fields that are part of the cursor (the index entry) even if you don't intend to use them directly, else MinQuery.All() will not have all the values of the cursor fields, and so it will not be able to create the proper cursor value.

Documentation

Overview

Package minquery provides a mgo-like Query type called MinQuery, which supports efficient query pagination (cursors to continue listing documents where we left off).

Example using MinQuery

Let's say we have a users collection in MongoDB modeled with this Go struct:

type User struct {
    ID      bson.ObjectId `bson:"_id"`
    Name    string        `bson:"name"`
    Country string        `bson:"country"`
}

To query users having country=USA, sorted by Name and ID:

q := minquery.New(session.DB(""), "users", bson.M{"country" : "USA"}).
    Sort("name", "_id").Limit(10)
// If this is not the first page, set cursor:
// getLastCursor() represents your logic how you acquire the last cursor.
if cursor := getLastCursor(); cursor != "" {
    q = q.Cursor(cursor)
}

var users []*User
newCursor, err := q.All(&users, "country", "name", "_id")

And that's all. newCursor is the cursor to be used to fetch the next batch.

Note #1: When calling MinQuery.All(), you have to provide the names of the cursor fields, this will be used to build the cursor data (and ultimately the cursor string) from.

Note #2: If you're retrieving partial results (by using MinQuery.Select()), you have to include all the fields that are part of the cursor (the index entry) even if you don't intend to use them directly, else MinQuery.All() will not have all the values of the cursor fields, and so it will not be able to create the proper cursor value.

Index

• Variables
• type CursorCodec
• type CursorCodecMongo
• type MinQuery
  • func New(db *mgo.Database, coll string, query interface{}, hint map[string]int) MinQuery

Variables

var DefaultCursorCodec cursorCodec

DefaultCursorCodec is the default CursorCodec value that is used if none is specified. The default implementation produces web-safe cursor strings.

var DefaultCursorCodecMongo cursorCodecMongo

var UseMongoDriver = false

Types

type CursorCodec

type CursorCodec interface {
	// CreateCursor returns a cursor string from the specified fields.
	CreateCursor(cursorData bson.D) (string, error)

	// ParseCursor parses the cursor string and returns the cursor data.
	ParseCursor(c string) (cursorData bson.D, err error)
}

CursorCodec represents a symmetric pair of functions that can be used to convert cursor data of type bson.D to a string and vice versa.

type CursorCodecMongo

type CursorCodecMongo interface {
	CreateCursorMongo(cursorData mongoDriverBson.D) (string, error)
	ParseCursorMongo(c string) (cursorData mongoDriverBson.D, err error)
}

type MinQuery

type MinQuery interface {
	// Sort asks the database to order returned documents according to
	// the provided field names.
	Sort(fields ...string) MinQuery

	// Select enables selecting which fields should be retrieved for
	// the results found.
	Select(selector interface{}) MinQuery

	// Limit restricts the maximum number of documents retrieved to n,
	// and also changes the batch size to the same value.
	Limit(n int) MinQuery

	// Cursor sets the cursor, which specifies the last index entry
	// that was already returned, and result documents will be listed after this.
	// Parsing a cursor may fail which is not returned. If an invalid cursor
	// is specified, All() will fail and return the error.
	Cursor(c string) MinQuery

	// CursorCoded sets the CursorCodec to be used to parse and to create cursors.
	// This gives you the possibility to implement your own logic to create cursors,
	// including encryption should you need it.
	CursorCodec(cc CursorCodec) MinQuery

	// All retrieves all documents from the result set into the provided slice.
	// cursorFields lists the fields (in order) to be used to generate
	// the returned cursor.
	All(result interface{}, cursorFields ...string) (cursor string, err error)
}

MinQuery is an mgo-like Query that supports cursors to continue listing documents where we left off. If a cursor is set, it specifies the last index entry that was already returned, and result documents will be listed after this.

func New

func New(db *mgo.Database, coll string, query interface{}, hint map[string]int) MinQuery

New returns a new MinQuery.