acceptable

README

Acceptable

This is a library that handles Accept headers, which form the basis of content negotiation in HTTP server applications written in Go. It provides an implementation of the proactive server-driven content negotiation algorithm specified in RFC-7231 section 5.3.

There is also support for conditional requests (RFC-7232) using entity tags and last-modified timestamps.

Bring your favourite router and framework - this library can be used with Gin, Echo, etc.

Please see the documentation for more info.

Status

This API is well-tested and known to work but not yet fully released because it may yet require breaking API changes.

Documentation

Overview

Package acceptable is a library that handles headers for content negotiation and conditional requests in web applications written in Go. Content negotiation is specified by RFC (http://tools.ietf.org/html/rfc7231) and, less formally, by Ajax (https://en.wikipedia.org/wiki/XMLHttpRequest).

Subpackages

* data - for holding response data & metadata prior to rendering the response

* header - for parsing and representing certain HTTP headers

* offer - for enumerating offers to be matched against requests

* templates - for rendering Go templates

Easy content negotiation

Server-based content negotiation is essentially simple: the user agent sends a request including some preferences (accept headers), then the server selects one of several possible ways of sending the response. Finding the best match depends on you listing your available response representations. This is all rolled up into a simple-to-use function `acceptable.RenderBestMatch`. What this does is described in detail in [RFC-7231](https://tools.ietf.org/html/rfc7231#section-5.3), but it's easy to use in practice.

en := ... obtain some content in English
fr := ... obtain some content in French

// long-hand construction of an offer for indented JSON
offer1 := offer.Of(acceptable.JSON("  "), contenttype.ApplicationJSON).With(en, "en").With(fr, "fr")

// short-hand construction of an XML offer
offer2 := acceptable.DefaultXMLOffer.With(en, "en").With(fr, "fr")
// equivalent to
//offer2 := offer.Of(acceptable.XML("xml"), contenttype.ApplicationXML).With(en, "en").With(fr, "fr")

// a catch-all offer is optional
catchAll := offer.Of(acceptable.TXT(), contenttype.Any).With(en, "en").With(fr, "fr")

err := acceptable.RenderBestMatch(request, offer1, offer2, catchAll)

The RenderBestMatch function searches for the offer that best matches the request headers. If none match, the response will be 406-Not Acceptable. If you need to have a catch-all case, include offer.Of(contenttype.Any) last in the list (contenttype.Any is "*/*").

Each offer will (usually) have a suitable rendering function. Several are provided (for JSON, XML etc), but you can also provide your own. Also, the templates sub-package provides Go template support.

The offers can also be restricted by language matching. This is done via the `With` method. The language(s) is matched against the Accept-Language header using the basic prefix algorithm. This means for example that if you specify "en" it will match "en", "en-GB" and everything else beginning with "en-", but if you specify "en-GB", it only matches "en-GB" and "en-GB-*", but won't match "en-US" or even "en". (This implements the basic filtering language matching algorithm defined in https://tools.ietf.org/html/rfc4647.)

Sometimes, the With method might not care about language, so simply use the "*" wildcard instead. For example, offer.With("*", data) attaches data to the offer and doesn't restrict the offer to any particular language. This could also be used as a catch-all case if it comes after one or more With with a specified language. However, the standard (RFC-7231) advises that a response should be returned even when language matching has failed; RenderBestMatch will do this by picking the first language listed as a fallback, so the catch-all case is only necessary if its data is different to that of the first case.

Providing response data

The response data (en and fr above) can be structs, slices, maps, or other values. Alternatively they can be data.Data values. These allow for lazy evaluation of the content and also support conditional requests. This comes into its own when there are several offers each with their own data model - if these were all to be read from the database before selection of the best match, all but one would be wasted. Lazy evaluation of the selected data easily overcomes this problem.

en := data.Lazy(func(template, language string, dataRequired bool) (data interface{}, meta *data.Metadata, err error) {
    return ...
})

Besides the data and error returned values, some metadata can optionally be returned. This is the basis for easy support for conditional requests (see [RFC-7232](https://tools.ietf.org/html/rfc7232)).

If the metadata is nil, it is simply ignored. However, if it contains a hash of the data (e.g. via MD5) known as the entity tag or etag, then the response will have an ETag header. User agents that recognise this will later repeat the request along with an If-None-Match header. If present, If-None-Match is recognised before rendering starts and a successful match will avoid the need for any rendering. Due to the lazy content fetching, it prevents unnecessary database traffic etc.

The metadata can also carry the last-modified timestamp of the data, if this is known. When present, this becomes the Last-Modified header and is checked on subsequent requests using the If-Modified-Since.

The template and language parameters are used for templated/web content data; otherwise they are ignored. The dataRequired parameter is used for a two-pass approach: the first call is to get the etag; the data itself can also be returned but *is optional*. The second call is made if the first call didn't return data - this time it *is required*.

The two-pass lazy evualation is intended to avoid fetching large data items when they will actually not be needed, i.e. in conditional requests that yield 304-Not Modified.

Otherwise, the selected response processor will render the actual response using the data provided, for example a struct will become JSON text if the JSON() processor renders it.

Character set transcoding

Most responses will be UTF-8, sometimes UTF-16. All other character sets (e.g. Windows-1252) are now strongly deprecated.

However, legacy support for other character sets is provided. Transcoding is implemented by Match.ApplyHeaders so that the Accept-Charset content negotiation can be implemented. This depends on finding an encoder in golang.org/x/text/encoding/htmlindex (this has an extensive list but no other encoders are supported).

Whenever possible, responses will be UTF-8. Not only is this strongly recommended, it also avoids any transcoding processing overhead. It means for example that "Accept-Charset: iso-8859-1, utf-8" will ignore the iso-8859-1 preference because it can use UTF-8. Conversely, "Accept-Charset: iso-8859-1" will always have to transcode into ISO-8859-1 because there is no UTF-8 option.

Example

package main

import (
	"fmt"
	"log"
	"net/http"
	"net/http/httptest"
	"sort"

	"github.com/rickb777/acceptable"
	"github.com/rickb777/acceptable/data"
	"github.com/rickb777/acceptable/offer"
)

func main() {
	// In this example, the same content is available in three languages. Three different
	// approaches can be used.

	// 1. simple values can be used
	en := "Hello!" // get English content

	// 2. values can be wrapped in a data.Data
	fr := data.Of("Bonjour!").ETag("hash1") // get French content and some metadata

	// 3. this uses a lazy evaluation function, wrapped in a data.Data
	es := data.Lazy(func(template string, language string) (interface{}, error) {
		return "Hola!", nil // get Spanish content - eg from database
	}).ETagUsing(func(template, language string) (string, error) {
		// allows us to obtain the etag lazily, should we need to
		return "hash2", nil
	})

	// We're implementing an HTTP handler, so we are given a request and a response.

	req1, _ := http.NewRequest("GET", "/request1", nil) // some incoming request
	req1.Header.Set("Accept", "text/plain, text/html")
	req1.Header.Set("Accept-Language", "es, fr;q=0.8, en;q=0.6")

	req2, _ := http.NewRequest("GET", "/request2", nil) // some incoming request
	req2.Header.Set("Accept", "application/json")
	req2.Header.Set("Accept-Language", "fr")

	req3, _ := http.NewRequest("GET", "/request3", nil) // some incoming request
	req3.Header.Set("Accept", "text/html")
	req3.Header.Set("Accept-Language", "fr")
	req3.Header.Set("If-None-Match", `"hash1"`)

	requests := []*http.Request{req1, req2, req3}

	for _, req := range requests {
		res := httptest.NewRecorder() // replace with the server's http.ResponseWriter

		// Now do the content negotiation. This example has six supported content types, all of them
		// able to serve any of the three example languages.
		//
		// The first offer is for JSON - this is often the most widely used because it also supports
		// Ajax requests.

		err := acceptable.RenderBestMatch(res, req, "home.html",
			offer.Of(acceptable.JSON("  "), "application/json").
				With(en, "en").With(fr, "fr").With(es, "es"),

			offer.Of(acceptable.XML("xml", "  "), "application/xml").
				With(en, "en").With(fr, "fr").With(es, "es"),

			offer.Of(acceptable.CSV(), "text/csv").
				With(en, "en").With(fr, "fr").With(es, "es"),

			offer.Of(acceptable.TXT(), "text/plain").
				With(en, "en").With(fr, "fr").With(es, "es"),

			acceptable.TextHtmlOffer("example/templates/en", ".html", nil).
				With(en, "en").With(fr, "fr").With(es, "es"),

			acceptable.ApplicationXhtmlOffer("example/templates/en", ".html", nil).
				With(en, "en").With(fr, "fr").With(es, "es"),
		)

		if err != nil {
			log.Fatal(err) // replace with suitable error handling
		}

		// ----- ignore the following, which is needed only for the example test to run -----
		fmt.Printf("%s %s %d\n", req.Method, req.URL, res.Code)
		fmt.Printf("%d headers\n", len(res.Header()))
		var hdrs []string
		for h, _ := range res.Header() {
			hdrs = append(hdrs, h)
		}
		sort.Strings(hdrs)
		for _, h := range hdrs {
			fmt.Printf("%s: %s\n", h, res.Header().Get(h))
		}
		fmt.Println()
		fmt.Println(res.Body.String())
	}

}

Output:

GET /request1 200
4 headers
Content-Language: es
Content-Type: text/plain;charset=utf-8
Etag: "hash2"
Vary: Accept, Accept-Language

Hola!

GET /request2 200
4 headers
Content-Language: fr
Content-Type: application/json;charset=utf-8
Etag: "hash1"
Vary: Accept, Accept-Language

"Bonjour!"

GET /request3 304
4 headers
Content-Language: fr
Content-Type: text/html;charset=utf-8
Etag: "hash1"
Vary: Accept, Accept-Language

Index

• Variables
• func ApplicationXhtmlOffer(dir, suffix string, funcMap template.FuncMap) offer.Offer
• func BestRequestMatch(req *http.Request, available ...offer.Offer) *offer.Match
• func Binary() offer.Processor
• func CSV(comma ...rune) offer.Processor
• func IsAjax(req *http.Request) bool
• func JSON(indent ...string) offer.Processor
• func RenderBestMatch(w http.ResponseWriter, req *http.Request, template string, ...) error
• func TXT() offer.Processor
• func TextHtmlOffer(dir, suffix string, funcMap template.FuncMap) offer.Offer
• func XML(root string, indent ...string) offer.Processor

Examples

• Package

Variables

var (
	// DefaultImageOffer is an Offer for image/* content using the Binary() processor.
	DefaultImageOffer = offer.Of(Binary(), contenttype.ImageAny)

	// DefaultCSVOffer is an Offer for text/plain content using the CSV() processor.
	DefaultCSVOffer = offer.Of(CSV(), contenttype.TextCSV)

	// DefaultJSONOffer is an Offer for application/json content using the JSON() processor without indentation.
	DefaultJSONOffer = offer.Of(JSON(), contenttype.ApplicationJSON)

	// DefaultTXTOffer is an Offer for text/plain content using the TXT() processor.
	DefaultTXTOffer = offer.Of(TXT(), contenttype.TextPlain)

	// DefaultXMLOffer is an Offer for application/xml content using the XML("") processor without indentation.
	DefaultXMLOffer = offer.Of(XML("xml"), contenttype.ApplicationXML)
)

var Debug = func(string, ...interface{}) {}

Debug can be used for observing decisions made by the negotiation algorithm. By default it is no-op.

Functions

func ApplicationXhtmlOffer

func ApplicationXhtmlOffer(dir, suffix string, funcMap template.FuncMap) offer.Offer

ApplicationXhtmlOffer is an Offer for application/xhtml+xml content using the Template() processor.

func BestRequestMatch

func BestRequestMatch(req *http.Request, available ...offer.Offer) *offer.Match

BestRequestMatch finds the content type and language that best matches the accepted media ranges and languages contained in request headers. The result contains the best match, based on the rules of RFC-7231. On exit, the result will contain the preferred language and charset, if these are known.

Whenever the result is nil, the response should be 406-Not Acceptable.

For all Ajax requests, the available offers are filtered so that only those capable of providing an Ajax response are considered by the content negotiation algorithm. The other offers are discarded.

The order of offers is important. It determines the order they are compared against the request headers, and it determines what defaults will be used when exact matching is not possible.

If no available offers are provided, the response will always be nil. Note too that Ajax requests will result in nil being returned if no offer is capable of handling them, even if other offers are provided.

func Binary

func Binary() offer.Processor

Binary creates an output processor that outputs binary data in a form suitable for image/* and similar responses. Model values should be one of the following:

* []byte * io.Reader * io.WriterTo * nil

func CSV

func CSV(comma ...rune) offer.Processor

CSV creates an output processor that serialises a dataModel in CSV form. With no arguments, the default format is comma-separated; you can supply any rune to be used as an alternative separator.

Model values should be one of the following:

* string or []string, or [][]string

* fmt.Stringer or []fmt.Stringer, or [][]fmt.Stringer

* []int or similar (bool, int8, int16, int32, int64, uint8, uint16, uint32, uint63, float32, float64, complex)

* [][]int or similar (bool, int8, int16, int32, int64, uint8, uint16, uint32, uint63, float32, float64, complex)

* struct for some struct in which all the fields are exported and of simple types (as above).

* []struct for some struct in which all the fields are exported and of simple types (as above).

func IsAjax

func IsAjax(req *http.Request) bool

IsAjax tests whether a request has the Ajax header sent by browsers for XHR requests.

func JSON

func JSON(indent ...string) offer.Processor

JSON creates a new processor for JSON with a specified indentation.

func RenderBestMatch

func RenderBestMatch(w http.ResponseWriter, req *http.Request, template string, available ...offer.Offer) error

RenderBestMatch uses BestRequestMatch to find the best matching offer for the request, and then renders the response. The returned error, if any, will have arisen from either the content provider (see data.Content) or the response processor (see offer.Processor).

func TXT

func TXT() offer.Processor

TXT creates an output processor that serialises strings in a form suitable for text/plain responses. Model values should be one of the following:

* string

* fmt.Stringer

* encoding.TextMarshaler

func TextHtmlOffer

func TextHtmlOffer(dir, suffix string, funcMap template.FuncMap) offer.Offer

TextHtmlOffer is an Offer for text/html content using the Template() processor.

func XML

func XML(root string, indent ...string) offer.Processor

XML creates a new processor for XML with root element and optional indentation. The root element is used only when processing content that is a sequence of data items.