gateway

README

cassandra-http-gateway

Conveniently assemble read-only HTTP gateway services for Apache Cassandra.

This framework acts as a query builder for CQL SELECT statements. The resulting query objects bind request parameters to attributes in the database, process HTTP requests, and return JSON-serialized results in a response. It can be used to publish read-only dataset access to internal clients.

It is -by design- very minimalist in nature. No validation of queries are performed, and JSON serialization is handled by Cassandra (ala SELECT JSON ...). Results are returned as an array of zero or more objects (rows).

Rationale

It has become increasingly common to have datasets that are materialized -either via batch analytics, and/or streaming events- and persisted for low-latency access. At the WMF, we typically persist these datasets to a multi-tenant Cassandra cluster (though use of other clusters & stores is possible). What is for us a half-dozen or so use-cases (at the time of writing), is being projected to grow to many 10s or 100s in the near future. At this scale, and with datasets from disparate stakeholders, this will likely become unmanageable.

An HTTP data gateway is one way of decoupling client applications from the underlying database. There are many potential benefits from this -for example- preserving the future ability to transparently migrate clusters or redistribute datasets, and to better manage resource utilization (rate limiting etc). Even something as mundane as a fleet-wide upgrade of native drivers could be prohibitively expensive when code owners are many, varied in resources and priorities, or worse when code becomes orphaned entirely.

Issues

• There is (currently) no support for streaming; Results are buffered in memory before being returned. Make sure you are not expecting a result set large enough for this to be an issue.
• Because there is no validation of input, obvious client errors will manifest as query failures. For example, if a client were to pass an alphanumeric value in a parameter bound to an integer field in the database, the framework will happily pass this on to Cassandra, resulting in a status 500 (instead of the expected 400).

See also: T294468

Usage

With a dataset like...

CREATE KEYSPACE music WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
CREATE TABLE music.tracks (artist text, album text, track text, release_date timestamp, PRIMARY KEY(artist, album, track));

INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Good Enough', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Why Can''t This Be Love', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Get Up', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Dreams', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Summer Nights', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Best of Both Worlds', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Love Walks In', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', '"5150"', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Inside', '1986-03-24');

And code that looks like...

package main

import (
    "fmt"
    "os"

    "github.com/gocql/gocql"
    "github.com/julienschmidt/httprouter"
)

func main() {
    var cluster *gocql.ClusterConfig
    var err error
    var session *gocql.Session

    cluster = gocql.NewCluster("localhost")
    cluster.Consistency = gocql.One

    if session, err = cluster.CreateSession(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }

    var router = new httprouter.New()
    router.GET("/music/:artist/:album", func(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
        SelectBuilder.
            From("music", "tracks").
            Session(session).
            Bind(ps).
            Build().
            Handle(w, r)
    })
}

You get a response like...

$ http http://localhost:8080/music/Van%20Halen/5150
HTTP/1.1 200 OK
Content-Length: 987
Content-Type: application/json
Date: Sat, 02 Apr 2022 01:25:04 GMT

{
    "rows": [
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "\"5150\""
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Best of Both Worlds"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Dreams"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Get Up"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Good Enough"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Inside"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Love Walks In"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Summer Nights"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Why Can't This Be Love"
        }
    ]
}


$

Documentation

Index

• Variables
• type Field
• type FieldBuilderType
  • func Column(name string) FieldBuilderType
  • func (b FieldBuilderType) Alias(label string) FieldBuilderType
  • func (b FieldBuilderType) CastTo(kind string) FieldBuilderType
  • func (b FieldBuilderType) Column(name string) FieldBuilderType
  • func (b FieldBuilderType) Get() Field
• type Predicate
• type Relation
• type SelectBuilderType
  • func (b SelectBuilderType) Bind(ps httprouter.Params) SelectBuilderType
  • func (b SelectBuilderType) Build() *SelectStatement
  • func (b SelectBuilderType) CounterVec(counter *prometheus.CounterVec) SelectBuilderType
  • func (b SelectBuilderType) From(keyspace, table string) SelectBuilderType
  • func (b SelectBuilderType) HistogramVec(histogram *prometheus.HistogramVec) SelectBuilderType
  • func (b SelectBuilderType) Logger(logger *log.Logger) SelectBuilderType
  • func (b SelectBuilderType) Select(fields ...Field) SelectBuilderType
  • func (b SelectBuilderType) Session(session *gocql.Session) SelectBuilderType
  • func (b SelectBuilderType) Where(column string, relation Relation, value interface{}) SelectBuilderType
• type SelectStatement
  • func (s *SelectStatement) Handle(w http.ResponseWriter, request *http.Request)
  • func (s *SelectStatement) String() string

Variables

var (
	SelectBuilder = builder.Register(SelectBuilderType{}, SelectStatement{}).(SelectBuilderType)
	FieldBuilder  = builder.Register(FieldBuilderType{}, Field{}).(FieldBuilderType)
)

Types

type Field

type Field struct {
	Column string
	CastTo string
	Alias  string
}

Field correponds to a field in the projection of a CQL SELECT statement.

type FieldBuilderType

type FieldBuilderType builder.Builder

FieldBuilderType is a fluent DSL for constructing Fields

func Column

func Column(name string) FieldBuilderType

Column is convience function that returns a new Field builder with the supplied column name assigned.

func (FieldBuilderType) Alias

func (b FieldBuilderType) Alias(label string) FieldBuilderType

Alias assigns an alias for the Field, and returns the builder.

func (FieldBuilderType) CastTo

func (b FieldBuilderType) CastTo(kind string) FieldBuilderType

CastTo assigns the type to cast a Field value to, and returns the builder.

func (FieldBuilderType) Column

func (b FieldBuilderType) Column(name string) FieldBuilderType

Column assigns the column name attribute of a Field, and returns the builder.

func (FieldBuilderType) Get

func (b FieldBuilderType) Get() Field

Get constructs and returns a Field object

type Predicate

type Predicate struct {
	Column   string
	Relation Relation
	Value    interface{}
}

Predicate represents a predicate in the WHERE clause of a CQL SELECT.

type Relation

type Relation string

Relation types correspond to CQL statement relations.

const (
	EQ  Relation = "="
	GT  Relation = ">"
	LT  Relation = "<"
	GTE Relation = ">="
	LTE Relation = "<="
)

type SelectBuilderType

type SelectBuilderType builder.Builder

SelectBuilderType is a fluent DSL for constructing SelectStatements

func (SelectBuilderType) Bind

func (b SelectBuilderType) Bind(ps httprouter.Params) SelectBuilderType

Bind creates predicates with an EQ ('=') relation for each of the parameters in the supplied httprouter.Params.

func (SelectBuilderType) Build

func (b SelectBuilderType) Build() *SelectStatement

Build constructs and returns a SelectStatement.

func (SelectBuilderType) CounterVec

func (b SelectBuilderType) CounterVec(counter *prometheus.CounterVec) SelectBuilderType

CounterVec assigns a Prometheus CounterVec object to the statement under construction. The object supplied will be used to count HTTP requests proccessed by Handle().

func (SelectBuilderType) From

func (b SelectBuilderType) From(keyspace, table string) SelectBuilderType

From assigns the keyspace and table name of the SELECT statement under construction.

func (SelectBuilderType) HistogramVec

func (b SelectBuilderType) HistogramVec(histogram *prometheus.HistogramVec) SelectBuilderType

HistogramVec assigns a Prometheus HistogramVec object to the statement under construction. The object supplied will be used to create a latency histogram of HTTP requests processed by Handle().

func (SelectBuilderType) Logger

func (b SelectBuilderType) Logger(logger *log.Logger) SelectBuilderType

Logger assigns a logger object to the statement under construction. The supplied object will be used to output logging information from HTTP requests processed by Handle().

func (SelectBuilderType) Select

func (b SelectBuilderType) Select(fields ...Field) SelectBuilderType

Select adds zero or more fields to the projection of the SELECT statement being constructed. If no fields are added (or if Select() is never invoked), then the projection will contain '*'.

func (SelectBuilderType) Session

func (b SelectBuilderType) Session(session *gocql.Session) SelectBuilderType

Session assigns a GoCQL Session object to the statement under construction. The Session supplied will be used to execute CQL queries against.

func (SelectBuilderType) Where

func (b SelectBuilderType) Where(column string, relation Relation, value interface{}) SelectBuilderType

Where adds a predicate to the SELECT statement under construction.

type SelectStatement

type SelectStatement struct {
	Keyspace   string
	Table      string
	Fields     []Field
	Predicates []Predicate

	Session      *gocql.Session
	CounterVec   *prometheus.CounterVec
	HistogramVec *prometheus.HistogramVec
	Logger       *log.Logger
	// contains filtered or unexported fields
}

SelectStatement represents a CQL SELECT statement

func (*SelectStatement) Handle

func (s *SelectStatement) Handle(w http.ResponseWriter, request *http.Request)

Handle processes an HTTP request, querying the database for results, and returning them as a JSON-formatted response.

func (*SelectStatement) String

func (s *SelectStatement) String() string

String returns the CQL query string for a SelectStatement