json

README

JSON Message Transformer

JSON Transformer provides Message Transformer for JSON messages. To transform SuperMQ Message successfully, the payload must be a JSON object.

For the messages that contain JSON array as the root element, JSON Transformer does normalization of the data: it creates a separate JSON message for each JSON object in the root. In order to be processed and stored properly, JSON messages need to contain message format information. For the sake of the simpler storing of the messages, nested JSON objects are flatten to a single JSON object, using composite keys with the default separator /. This implies that the separator character (/) is not allowed in the JSON object key. For example, the following JSON object:

{
    "name": "name",
    "id":8659456789564231564,
    "in": 3.145,
    "alarm": true,
    "ts": 1571259850000,
    "d": {
        "tmp": 2.564,
        "hmd": 87,
        "loc": {
            "x": 1,
            "y": 2
        }
    }
}

will be transformed to:

{
    "name": "name",
    "id":8659456789564231564,
    "in": 3.145,
    "alarm": true,
    "ts": 1571259850000,
    "d/tmp": 2.564,
    "d/hmd": 87,
    "d/loc/x": 1,
    "d/loc/y": 2
}

The message format is stored in the subtopic. It's the last part of the subtopic. In the example:

http://localhost:8008/channels/<channelID>/messages/home/temperature/myFormat

the message format is myFormat. It can be any valid subtopic name, JSON transformer is format-agnostic. The format is used by the JSON message consumers so that they can process the message properly. If the format is not present (i.e. message subtopic is empty), JSON Transformer will report an error. Since the Transformer is agnostic to the format, having format in the subtopic does not prevent the publisher to send the content of different formats to the same subtopic. It's up to the consumer to handle this kind of issue. Message writers, for example, will store the message(s) in the table/collection/measurement (depending on the underlying database) with the name of the format (which in the example is myFormat). SuperMQ writers will try to save any format received (whether it will be successful depends on the writer implementation and the underlying database), but it's recommended that the publisher takes care not to send different formats to the same subtopic.

Having a message format in the subtopic means that the subscriber has an option to subscribe to only one message format. This is a nice feature because message subscribers know what's the expected format of the message so that they can process it. If the message format is not important, wildcard subtopic can always be used to subscribe to any message format:

http://localhost:8185/channels/<channelID>/messages/home/temperature/*

Documentation

Overview

Package json contains JSON transformer.

Index

• Variables
• func Flatten(m map[string]interface{}) (map[string]interface{}, error)
• func New(tfs []TimeField) transformers.Transformer
• func ParseFlat(flat interface{}) interface{}
• type Message
• type Messages
• type Payload
• type TimeField

Examples

• Flatten
• ParseFlat

Variables

var (

	// ErrTransform represents an error during parsing message.
	ErrTransform = errors.New("unable to parse JSON object")
	// ErrInvalidKey represents the use of a reserved message field.
	ErrInvalidKey = errors.New("invalid object key")
	// ErrInvalidTimeField represents the use an invalid time field.
	ErrInvalidTimeField = errors.New("invalid time field")
)

Functions

func Flatten

func Flatten(m map[string]interface{}) (map[string]interface{}, error)

Flatten makes nested maps flat using composite keys created by concatenation of the nested keys.

Example

package main

import (
	"encoding/json"
	"fmt"

	smqjson "github.com/absmach/supermq/pkg/transformers/json"
)

func main() {
	in := map[string]interface{}{
		"key1": "value1",
		"key2": "value2",
		"key5": map[string]interface{}{
			"nested1": map[string]interface{}{
				"nested2": "value3",
				"nested3": "value4",
			},
			"nested2": map[string]interface{}{
				"nested4": "value5",
			},
		},
	}
	out, err := smqjson.Flatten(in)
	if err != nil {
		panic(err)
	}
	b, err := json.MarshalIndent(out, "", "  ")
	if err != nil {
		panic(err)
	}
	fmt.Println(string(b))
}

Output:

{
  "key1": "value1",
  "key2": "value2",
  "key5/nested1/nested2": "value3",
  "key5/nested1/nested3": "value4",
  "key5/nested2/nested4": "value5"
}

func New

func New(tfs []TimeField) transformers.Transformer

New returns a new JSON transformer.

func ParseFlat

func ParseFlat(flat interface{}) interface{}

ParseFlat receives flat map that represents complex JSON objects and returns the corresponding complex JSON object with nested maps. It's the opposite of the Flatten function.

Example

package main

import (
	"encoding/json"
	"fmt"

	smqjson "github.com/absmach/supermq/pkg/transformers/json"
)

func main() {
	in := map[string]interface{}{
		"key1":                 "value1",
		"key2":                 "value2",
		"key5/nested1/nested2": "value3",
		"key5/nested1/nested3": "value4",
		"key5/nested2/nested4": "value5",
	}

	out := smqjson.ParseFlat(in)
	b, err := json.MarshalIndent(out, "", "  ")
	if err != nil {
		panic(err)
	}
	fmt.Println(string(b))
}

Output:

{
  "key1": "value1",
  "key2": "value2",
  "key5": {
    "nested1": {
      "nested2": "value3",
      "nested3": "value4"
    },
    "nested2": {
      "nested4": "value5"
    }
  }
}

Types

type Message

type Message struct {
	Channel   string  `json:"channel,omitempty" db:"channel" bson:"channel"`
	Created   int64   `json:"created,omitempty" db:"created" bson:"created"`
	Subtopic  string  `json:"subtopic,omitempty" db:"subtopic" bson:"subtopic,omitempty"`
	Publisher string  `json:"publisher,omitempty" db:"publisher" bson:"publisher"`
	Protocol  string  `json:"protocol,omitempty" db:"protocol" bson:"protocol"`
	Payload   Payload `json:"payload,omitempty" db:"payload" bson:"payload,omitempty"`
}

Message represents a JSON messages.

type Messages

type Messages struct {
	Data   []Message
	Format string
}

Messages represents a list of JSON messages.

type Payload

type Payload map[string]interface{}

Payload represents JSON Message payload.

type TimeField

type TimeField struct {
	FieldName   string `toml:"field_name"`
	FieldFormat string `toml:"field_format"`
	Location    string `toml:"location"`
}

TimeField represents the message fields to use as timestamp.