lb

README

Low Bandwidth Matrix

This repository implements MSC3079 in Go. It also provides several command line tools to get up to speed with existing low bandwidth enabled servers.

Mobile implementations

See mobile for Android/iOS bindings.

Command Line Tools

• jc: This tool can be used to convert JSON <--> CBOR.
• coap: This tool can be used to send a single CoAP request/response, similar to curl.
• proxy: This tool can be used to add low bandwidth support to any Matrix homeserver.

These can be tied together to interact with low-bandwidth enabled Matrix servers. For example:

# convert inline JSON to CBOR then pipe into coap to localhost:8008 then convert the CBOR response back to JSON and print to stdout
./jc -out "-" '{"auth":{"type":"m.login.dummy"},"username":"foo","password":"barbarbar"}' \
| ./coap -X POST -d '-' -H "Content-Type: application/cbor" -k  https://localhost:8008/_matrix/client/r0/register \
| ./jc -c2j -out '-' '-'

Documentation

Overview

Package lb provides an implementation of MSC3079: Low Bandwidth CS API

Index

• Variables
• func CBORToJSONHandler(next http.Handler, codec *CBORCodec, logger Logger) http.Handler
• type CBORCodec
  • func NewCBORCodec(keys map[string]int, canonical bool) (*CBORCodec, error)
  • func NewCBORCodecV1(canonical bool) *CBORCodec
  • func (c *CBORCodec) CBORToJSON(input io.Reader) ([]byte, error)
  • func (c *CBORCodec) JSONToCBOR(input io.Reader) ([]byte, error)
• type CoAPHTTP
  • func NewCoAPHTTP(paths *CoAPPath) *CoAPHTTP
  • func (co *CoAPHTTP) CoAPHTTPHandler(next http.Handler, ob *Observations) coapmux.Handler
  • func (co *CoAPHTTP) CoAPToHTTPRequest(r *message.Message) *http.Request
  • func (co *CoAPHTTP) CoAPToHTTPResponse(r *pool.Message) *http.Response
  • func (co *CoAPHTTP) HTTPRequestToCoAP(req *http.Request, doFn func(*pool.Message) error) error
• type CoAPPath
  • func NewCoAPPath(pathMappings map[string]string) (*CoAPPath, error)
  • func NewCoAPPathV1() *CoAPPath
  • func (c *CoAPPath) CoAPPathToHTTPPath(p string) string
  • func (c *CoAPPath) HTTPPathToCoapPath(p string) string
• type HasUpdatedFn
• type Logger
• type Observations
  • func NewObservations(next http.Handler, codec *CBORCodec, hasUpdatedFn HasUpdatedFn, ...) *Observations
  • func NewSyncObservations(next http.Handler, c *CoAPPath, codec *CBORCodec) *Observations
  • func (o *Observations) HandleBlockwise(w coapmux.ResponseWriter, r *coapmux.Message)
  • func (o *Observations) HandleRegistration(req *http.Request, w coapmux.ResponseWriter, r *coapmux.Message, register bool)
• type ObserveUpdateFn

Examples

• CBORCodec.CBORToJSON
• CBORCodec.JSONToCBOR
• CBORToJSONHandler

Variables

var OptionIDAccessToken = message.OptionID(256)

The CoAP Option ID corresponding to the access_token for Matrix requests

Functions

func CBORToJSONHandler

func CBORToJSONHandler(next http.Handler, codec *CBORCodec, logger Logger) http.Handler

CBORToJSONHandler transparently wraps JSON http handlers to accept and produce CBOR. It wraps the provided `next` handler and modifies it in two ways:

• If the request header is 'application/cbor' then convert the request body to JSON and overwrite the request body with the JSON, then invoke the `next` handler.
• Supply a wrapped http.ResponseWriter to the `next` handler which will convert JSON written via Write() into CBOR, if and only if the header 'application/json' is written first (before WriteHeader() is called).

This is the main function users of this library should use if they wish to transparently handle CBOR. This needs to be combined with CoAP handling to handle all of MSC3079.

Example

package main

import (
	"bytes"
	"encoding/hex"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"net/http/httptest"

	"github.com/matrix-org/lb"
)

func main() {
	// This handler accepts JSON and returns JSON: it knows nothing about CBOR.
	// We will give it the test case CBOR and it will return the same test case CBOR
	// by wrapping this handler in CBORToJSONHandler
	handler := http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
		event := struct {
			Sender  string                 `json:"sender"`
			Type    string                 `json:"type"`
			Content map[string]interface{} `json:"content"`
		}{}
		// The request body was transparently converted from CBOR to JSON for us
		if err := json.NewDecoder(req.Body).Decode(&event); err != nil {
			panic(err)
		}
		// Check the fields
		if event.Sender != "@alice:localhost" ||
			event.Type != "m.room.message" ||
			event.Content["msgtype"] != "m.text" ||
			event.Content["body"] != "Hello World" {

			w.WriteHeader(400)
			return
		}
		// we MUST tell it that we are sending back JSON before WriteHeader is called
		w.Header().Set("Content-Type", "application/json")
		w.WriteHeader(200)
		// This response body will transparently be converted into CBOR
		w.Write([]byte(`
		{
			"type": "m.room.message",
			"content": {
			  "msgtype": "m.text",
			  "body": "Hello World"
			},
			"sender": "@alice:localhost",
			"room_id": "!foo:localhost",
			"unsigned": {
			  "bool_value": true,
			  "null_value": null
			}
		  }`))
	})

	// This is where we call into the library, the rest of this is just setup/verification code
	// ----------------------------------------------------------------------------------
	// wrap the JSON handler and set it on the default serv mux
	http.Handle("/json_endpoint", lb.CBORToJSONHandler(handler, lb.NewCBORCodecV1(true), nil))
	// ----------------------------------------------------------------------------------

	// Test case from MSC3079
	inputData, err := hex.DecodeString(`a5026e6d2e726f6f6d2e6d65737361676503a2181b6b48656c6c6f20576f726c64181c666d2e74657874056e21666f6f3a6c6f63616c686f7374067040616c6963653a6c6f63616c686f737409a26a626f6f6c5f76616c7565f56a6e756c6c5f76616c7565f6`)
	if err != nil {
		panic(err)
	}
	server := httptest.NewServer(http.DefaultServeMux)
	defer server.Close()
	res, err := http.Post(server.URL+"/json_endpoint", "application/cbor", bytes.NewBuffer(inputData))
	if err != nil {
		panic(err)
	}
	if res.StatusCode != 200 {
		panic(fmt.Sprintf("returned HTTP %d", res.StatusCode))
	}
	defer res.Body.Close()
	// cbor should have been returned
	respBody, err := ioutil.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}
	fmt.Println(hex.EncodeToString(respBody))
}

Output:

a5026e6d2e726f6f6d2e6d65737361676503a2181b6b48656c6c6f20576f726c64181c666d2e74657874056e21666f6f3a6c6f63616c686f7374067040616c6963653a6c6f63616c686f737409a26a626f6f6c5f76616c7565f56a6e756c6c5f76616c7565f6

Types

type CBORCodec

type CBORCodec struct {
	// contains filtered or unexported fields
}

CBORCodec allows the conversion between JSON and CBOR.

func NewCBORCodec

func NewCBORCodec(keys map[string]int, canonical bool) (*CBORCodec, error)

NewCBORCodec creates a CBOR codec which will map the enum keys given. If canonical is set, the output from this codec with be in canonical format for CBOR (RFC 7049 Section 3.9) and in Matrix Canonical JSON for JSON (https://matrix.org/docs/spec/appendices#canonical-json). Generally, you don't want to set canonical to true unless you are performing tests which need to produce a deterministic output (e.g sorted keys).

Users of this library should prefer NewCBORCodecV1 which sets up all the enum keys for you. This function is exposed for bleeding edge or custom enums.

func NewCBORCodecV1

func NewCBORCodecV1(canonical bool) *CBORCodec

NewCBORCodecV1 creates a v1 codec capable of converting JSON <--> CBOR. If canonical is set, the output from this codec with be in canonical format for CBOR (RFC 7049 Section 3.9) and in Matrix Canonical JSON for JSON (https://matrix.org/docs/spec/appendices#canonical-json). Generally, you don't want to set canonical to true unless you are performing tests which need to produce a deterministic output (e.g sorted keys) as it consumes extra CPU.

func (*CBORCodec) CBORToJSON

func (c *CBORCodec) CBORToJSON(input io.Reader) ([]byte, error)

CBORToJSON converts a single CBOR object into a single JSON object

Example

package main

import (
	"bytes"
	"encoding/hex"
	"fmt"

	"github.com/matrix-org/lb"
)

func main() {
	// Test case from MSC3079
	input := `a5026e6d2e726f6f6d2e6d65737361676503a2181b6b48656c6c6f20576f726c64181c666d2e74657874056e21666f6f3a6c6f63616c686f7374067040616c6963653a6c6f63616c686f737409a26a626f6f6c5f76616c7565f56a6e756c6c5f76616c7565f6`
	inputBytes, err := hex.DecodeString(input)
	if err != nil {
		panic(err)
	}
	v1 := lb.NewCBORCodecV1(true)
	output, err := v1.CBORToJSON(bytes.NewBuffer(inputBytes))
	if err != nil {
		panic(err)
	}
	fmt.Println(string(output))
}

Output:

{"content":{"body":"Hello World","msgtype":"m.text"},"room_id":"!foo:localhost","sender":"@alice:localhost","type":"m.room.message","unsigned":{"bool_value":true,"null_value":null}}

func (*CBORCodec) JSONToCBOR

func (c *CBORCodec) JSONToCBOR(input io.Reader) ([]byte, error)

JSONToCBOR converts a single JSON object into a single CBOR object

Example

package main

import (
	"bytes"
	"encoding/hex"
	"fmt"

	"github.com/matrix-org/lb"
)

func main() {
	// Test case from MSC3079
	input := `
	{
		"type": "m.room.message",
		"content": {
		  "msgtype": "m.text",
		  "body": "Hello World"
		},
		"sender": "@alice:localhost",
		"room_id": "!foo:localhost",
		"unsigned": {
		  "bool_value": true,
		  "null_value": null
		}
	  }`
	v1 := lb.NewCBORCodecV1(true)
	output, err := v1.JSONToCBOR(bytes.NewBufferString(input))
	if err != nil {
		panic(err)
	}
	fmt.Println(hex.EncodeToString(output))
}

Output:

a5026e6d2e726f6f6d2e6d65737361676503a2181b6b48656c6c6f20576f726c64181c666d2e74657874056e21666f6f3a6c6f63616c686f7374067040616c6963653a6c6f63616c686f737409a26a626f6f6c5f76616c7565f56a6e756c6c5f76616c7565f6

type CoAPHTTP

type CoAPHTTP struct {
	// Optional logger if you want to debug request/responses
	Log Logger
	// Which set of CoAP enum paths to use (e.g v1)
	Paths *CoAPPath
	// Custom generator for CoAP tokens. NewCoAPHTTP uses a monotonically increasing integer.
	NextToken func() message.Token
}

CoAPHTTP provides many ways to convert to and from HTTP/CoAP.

func NewCoAPHTTP

func NewCoAPHTTP(paths *CoAPPath) *CoAPHTTP

NewCoAPHTTP returns various mapping functions and a wrapped HTTP handler for transparently mapping to and from HTTP.

To aid debugging, you can set `CoAPHTTP.Log` after creation to log when things go wrong.

func (*CoAPHTTP) CoAPHTTPHandler

func (co *CoAPHTTP) CoAPHTTPHandler(next http.Handler, ob *Observations) coapmux.Handler

CoAPHTTPHandler transparently wraps an HTTP handler to accept and produce CoAP.

`Observations` is an optional and allows the HTTP request to be observed in accordance with the CoAP OBSERVE specification.

func (*CoAPHTTP) CoAPToHTTPRequest

func (co *CoAPHTTP) CoAPToHTTPRequest(r *message.Message) *http.Request

CoAPToHTTPRequest converts a coap message into an HTTP request for http.Handler (lossy) Conversion expects the following coap options: (message body is not modified)

Uri-Host = "example.net"
Uri-Path = "_matrix"
Uri-Path = "client"
Uri-Path = "versions"
Uri-Query = "access_token=foobar"
Uri-Query = "limit=5"
=> example.net/_matrix/client/versions?access_token=foobar&limit=5

func (*CoAPHTTP) CoAPToHTTPResponse

func (co *CoAPHTTP) CoAPToHTTPResponse(r *pool.Message) *http.Response

func (*CoAPHTTP) HTTPRequestToCoAP

func (co *CoAPHTTP) HTTPRequestToCoAP(req *http.Request, doFn func(*pool.Message) error) error

HTTPRequestToCoAP converts an HTTP request to a CoAP message then invokes doFn. This callback MUST immediately make the CoAP request and not hold a reference to the Message as it will be de-allocated back to a sync.Pool when the function ends. Returns an error if it wasn't possible to convert the HTTP request to CoAP, or if doFn returns an error.

type CoAPPath

type CoAPPath struct {
	// contains filtered or unexported fields
}

CoAPPath handles mapping to and from HTTP/CoAP paths The mapping function converts things like:

/_matrix/client/r0/sync  =>  /7
/_matrix/client/r0/user/@frank:localhost/account_data/im.vector.setting.breadcrumbs  =>  /r/@frank:localhost/im.vector.setting.breadcrumbs

All static path segments are folded down into a single URI friendly byte, then dynamic path segments are overlaid in the order they appear in the the full format.

func NewCoAPPath

func NewCoAPPath(pathMappings map[string]string) (*CoAPPath, error)

NewCoAPPath makes a CoAPPath with the path mappings given. `pathMappings` MUST be in the form:

  {
	    "9": "/_matrix/client/r0/rooms/{roomId}/send/{eventType}/{txnId}"
  }

Specifically, the keys are the path enums, and the values are the HTTP paths with `{placeholder}` variables. These variables are important to determine what the CoAP path output should be and MUST be enclosed in {} (you cannot use $).

Users of this library should prefer NewCoAPPathV1 which sets up all the enum paths for you. This function is exposed for bleeding edge or custom enums.

func NewCoAPPathV1

func NewCoAPPathV1() *CoAPPath

NewCoAPPathV1 creates CoAP enum path mappings for version 1. This allows conversion between HTTP paths and CoAP enum paths such as:

/_matrix/client/r0/user/@frank:localhost/account_data/im.vector.setting.breadcrumbs
/r/@frank:localhost/im.vector.setting.breadcrumbs

func (*CoAPPath) CoAPPathToHTTPPath

func (c *CoAPPath) CoAPPathToHTTPPath(p string) string

CoAPPathToHTTPPath converts a coap path to a full HTTP path e.g converts /7 into /_matrix/client/r0/sync Returns the input path if this is not a coap enum path

func (*CoAPPath) HTTPPathToCoapPath

func (c *CoAPPath) HTTPPathToCoapPath(p string) string

HTTPPathToCoapPath converts an HTTP path into a coap path e.g converts /_matrix/client/r0/sync into /7 Returns the input path if this path isn't mapped to a coap enum path

type HasUpdatedFn

type HasUpdatedFn func(path string, prevRespBody, respBody []byte) bool

HasUpdatedFn is a function which returns true if the responses indicate some form of change that the client should be notified about. `prevRespBody` will be <nil> on the first invocation.

type Logger

type Logger interface {
	Printf(format string, v ...interface{})
}

Logger is an interface which can be satisfied to print debug logging when things go wrong. It is entirely optional, in which cases errors are silent.

type Observations

type Observations struct {
	Codec *CBORCodec
	Log   Logger
	// contains filtered or unexported fields
}

Observations is capable of handling CoAP OBSERVE requests and handles long-polling http.Handler requests on the client's behalf. Tokens can be extracted and used in subsequent requests by setting an observation update function.

func NewObservations

func NewObservations(next http.Handler, codec *CBORCodec, hasUpdatedFn HasUpdatedFn, fns ...ObserveUpdateFn) *Observations

NewObservations makes a new observations struct. `next` must be the normal HTTP handlers which will be called on behalf of the client. `fns` are optional path-specific update functions which can update a long-poll e.g extracting `next_batch` from the /sync body and using it as ?since= in the next request. `hasUpdatedFn` is optional and returns whether the response is meaningful or not. If hasUpdatedFn is missing, all responses are treated as meaningful.

func NewSyncObservations

func NewSyncObservations(next http.Handler, c *CoAPPath, codec *CBORCodec) *Observations

NewSyncObservations returns an Observations capable of processing Matrix /sync requests

func (*Observations) HandleBlockwise

func (o *Observations) HandleBlockwise(w coapmux.ResponseWriter, r *coapmux.Message)

HandleBlockwise MAY send back an entire response, if it can be determined that the request is part of a blockwise request.

func (*Observations) HandleRegistration

func (o *Observations) HandleRegistration(req *http.Request, w coapmux.ResponseWriter, r *coapmux.Message, register bool)

HandleRegistration (de)registers an observation of a resource and performs HTTP requests on behalf of the client.

The response writer and message must be the OBSERVE request.

type ObserveUpdateFn

type ObserveUpdateFn func(path string, prevRespBody []byte, req *http.Request) *http.Request

ObserveUpdateFn is a function which can update the long-poll request between calls. prevRespBody will be <nil> if this is the first call