utils

README

(C) 2020 Geotab Inc

All files and artifacts in this repository are licensed under the provisions of the license provided by the LICENSE file in this repository.

Reference compression design

The design is context aware, i. e. it knows that the payload is JSON, and the set of key-value pairs that can be expected. As the payload sizes are typically less than 100 characters, this is probably necessary to achieve high compression ratios. Tests with standard compression algorithms like GZIP, LZW, and DEFLATE showed comparably low compression ratios.

The overall compression consists of the following compression features:

1. Paths are replaced by a two byte index from the list of paths found in vsspathlist.json.
2. Fixed keywords, such as "action", "path", "get", etc., are replaced by a one byte value.
3. Timestamps are replaced by a four byte value.
4. The value part for the keys "value", "requestId", "subscriptionId" are replaced by a binary value, if possible.
5. JSON reserved characters are removed in the case their excistence in a certain position is a MUST according to the JSON format rules.
6. Payload data that does not fall under any of the features above is left uncompressed.

Below follows further details about the compression features.

1. The list vsspathist.json is generated by the VISSv2 server at every startup. The input to the server comes from the "vss_vissv2.binary" file, which is in its turn generated by the VSS Tools, that take the VSS tree as represented by the vspec files from the VSS repo as input. This means that if the VISSv2 server starts up with an updated VSS tree, then the clients should be aware of this, and get a copy of the updated vsspathlist.json file from the server. This could be handled by a request to the server asking for this file, but the VISSv2 spec does not currenly support that.
   The pathlist in the file generated by the VISSv2 server is ordered, so searches in the list can be optimised.
   The list index is represented by two bytes, which means that a maximum of 65536 paths can be handled. The list only contains paths for leaf nodes, so this is the maximum number of leaf nodes that the VSS tree can have for this compression feature to work.
   

2. The value replacing the keywords are the index from the kewordlist plus 128. The reason to add 128 is that an uncompressed UTF8 JSON string only contains byte values less than 128, see an ASCII table. The keywordlist original is found in the "common.go" source file. It is not expected to change. A client compression impl needs a copy of it.
   

3. The uncompressed timestamps follow the ISO8601 standard and has the format YYYY-MM-DDTHH:MM:SS.ssssssZ
   This is compressed as follows:

• The least significant year digit is represented by four bits.
• The month is represented by four bits.
• The day is represented by five bits.
• The hour is represented by five bits.
• The minute is represented by six bits.
• The second is represented by six bits.

Subsecond usage is not supported currently.

The sum of bits above is 30 bits, which fits into 4 bytes as shown below.
Byte 1 | Byte 2 | Byte 3 | Byte 4
00yyyymm | mmdddddh | hhhhmmmm | mmssssss

The timestamp encoding has the limitation that when the year goes from 2029 to 2030, any messages that are received by a client after the new year has begun, that contain a timestamp from the previous year, will be incorrectly decoded as they will be assigned the year 2039 instead of 2029. A solution to this could be to use Unix time instead, but then the 2038 problem will be coming. Another would be for clients to check for this 10-year problem around the time when it becomes likely. Vehicles “should not” respond with too stale data, so a few months after 2030 begins should be max what is needed.

1. The value types that are supported are integer, float32, and boolean. The first byte is and index from the keywordlist, which also contains this (+128). The type indicated also sets the number of bytes used for the representation, as shown below. The type "nuintx" means it is a negative uint value, as all integer values are represented as uints.
   Type | #bytes
   nuint8 | 1
   uint8 | 1
   nuint16 | 2
   uint16v | 2
   nuint24 | 3
   uint24 | 3
   nuint32 | 4
   uint32 | 4
   bool | 1
   float32 | 4
   

The absolute value of integers are compared to the max value of the different types, and the smallest possible size is selected. Values that are not of any of these types are kept uncompressed.

1. The following JSON reserved character usages can be removed as at decompression the JSON rules will infer their reinstatement.

• The leading and trailing curly brackets.
• The colon between key and value.
• The comma between JSON data entities.

1. The fact that data that does not relate to any of the mentioned compression features is left uncompressed leads to that the usage of this compression does not apply any restriction to the syntax scope of VISSv2. It also enables the client to send uncompressed requests, where the response will be compressed.

The current compression solution does not support service discovery requests.

Documentation

Index

• Constants
• Variables
• func AddKeyValue(message string, key string, value string) string
• func AnalyzeValueType(value string) int
• func BackendWSdataSession(conn *websocket.Conn, backendChannel chan string)
• func CloseLogFile()
• func CompressMessage(message []byte) []byte
• func DecompressMessage(message []byte) []byte
• func ExtractFromToken(token string, claim string) string
• func FileExists(filename string) bool
• func FinalizeMessage(responseMap map[string]interface{}) string
• func FrontendWSdataSession(conn *websocket.Conn, clientChannel chan string, backendChannel chan string)
• func GenerateHmac(input string, key string) string
• func GetModelIP(ipModel int) string
• func GetRfcTime() string
• func GetServerIP() string
• func InitDataSession(muxServer *http.ServeMux, regData RegData) (dataConn *websocket.Conn)
• func InitLog(filename string, logdir string, logFile bool, logLevel string)
• func MapRequest(request string, rMap *map[string]interface{}) int
• func NextQuoteMark(message []byte, offset int) int
• func PathToUrl(path string) string
• func ReadTransportSecConfig()
• func RegisterAsTransportMgr(regData *RegData, protocol string)
• func RemoveInternalData(response string) (string, int)
• func SetErrorResponse(reqMap map[string]interface{}, errRespMap map[string]interface{}, ...)
• func TrimLogFile(logFile *os.File)
• func UnpackFilter(filter interface{}, fList *[]FilterObject)
• func UrlToPath(url string) string
• func VerifyTokenSignature(token string, key string) bool
• type ClientHandler
• type ClientServer
• type CodeList
• type FilterObject
• type HttpChannel
• type HttpServer
  • func (server HttpServer) InitClientServer(muxServer *http.ServeMux)
• type HttpWSsession
  • func (httpCoreSocketSession HttpWSsession) TransportHubFrontendWSsession(dataConn *websocket.Conn, appClientChannel []chan string)
• type PathList
• type RegData
• type SecConfig
• type TransportHubFrontendWSSession
• type WsChannel
• type WsServer
  • func (server WsServer) InitClientServer(muxServer *http.ServeMux, serverIndex *int)
• type WsWSsession
  • func (wsCoreSocketSession WsWSsession) TransportHubFrontendWSsession(dataConn *websocket.Conn, appClientChannel []chan string)

Constants

const CODELISTINDEXPATH = 4 // must be set to the list index of the "path" element

const CODELISTINDEXREQID = 1 // must be set to the list index of the "requestId" element

const CODELISTINDEXSUBID = 5 // must be set to the list index of the "subscriptionId" element

const CODELISTINDEXTS = 3 // must be set to the list index of the "ts" element

const CODELISTINDEXVALUE = 2 // must be set to the list index of the "value" element

const CODELISTKEYS = 10 // must be set to the number of keys in the list

const CODELISTKEYVALUES = 15 // must be set to the number of keys plus values in the list (excl value types)

const IpEnvVarName = "GEN2MODULEIP"

const IpModel = 0 // IpModel = [0,1,2] = [localhost,extIP,envVarIP]

Variables

var (
	//    Trace   *log.Logger
	Info    *logrus.Logger
	Warning *logrus.Logger
	Error   *logrus.Logger
)

var AppClientChan = []chan string{
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
	make(chan string),
}

the number of channel array elements sets the limit for max number of parallel WS app clients

var HostIP string

var Logfile *os.File

const LOG_FILE = "servercore-log.txt"

var MuxServer = []*http.ServeMux{
	http.NewServeMux(),
	http.NewServeMux(),
	http.NewServeMux(),
}

var TransportErrorMessage string

var Upgrader = websocket.Upgrader{
	ReadBufferSize:  1024,
	WriteBufferSize: 1024,
}

Functions

func AddKeyValue

func AddKeyValue(message string, key string, value string) string

func AnalyzeValueType

func AnalyzeValueType(value string) int

func BackendWSdataSession

func BackendWSdataSession(conn *websocket.Conn, backendChannel chan string)

func CloseLogFile

func CloseLogFile()

func CompressMessage

func CompressMessage(message []byte) []byte

func DecompressMessage

func DecompressMessage(message []byte) []byte

func ExtractFromToken

func ExtractFromToken(token string, claim string) string

func FileExists

func FileExists(filename string) bool

func FinalizeMessage

func FinalizeMessage(responseMap map[string]interface{}) string

func FrontendWSdataSession

func FrontendWSdataSession(conn *websocket.Conn, clientChannel chan string, backendChannel chan string)

func GenerateHmac

func GenerateHmac(input string, key string) string

func GetModelIP

func GetModelIP(ipModel int) string

func GetRfcTime

func GetRfcTime() string

func GetServerIP

func GetServerIP() string

func InitDataSession

func InitDataSession(muxServer *http.ServeMux, regData RegData) (dataConn *websocket.Conn)

func InitLog

func InitLog(filename string, logdir string, logFile bool, logLevel string)

func MapRequest

func MapRequest(request string, rMap *map[string]interface{}) int

func NextQuoteMark

func NextQuoteMark(message []byte, offset int) int

func PathToUrl

func PathToUrl(path string) string

func ReadTransportSecConfig

func ReadTransportSecConfig()

func RegisterAsTransportMgr

func RegisterAsTransportMgr(regData *RegData, protocol string)

* * registerAsTransportMgr: * Registers with servercore as WebSocket protocol manager, and stores response in regData *

func RemoveInternalData

func RemoveInternalData(response string) (string, int)

func SetErrorResponse

func SetErrorResponse(reqMap map[string]interface{}, errRespMap map[string]interface{}, number string, reason string, message string)

func TrimLogFile

func TrimLogFile(logFile *os.File)

* * The log file is trimmed to 20% of its size when exceeding 10MB. *

func UnpackFilter

func UnpackFilter(filter interface{}, fList *[]FilterObject)

func UrlToPath

func UrlToPath(url string) string

func VerifyTokenSignature

func VerifyTokenSignature(token string, key string) bool

Types

type ClientHandler

type ClientHandler interface {
	// contains filtered or unexported methods
}

********************************************************************* Client response handlers *********************

type ClientServer

type ClientServer interface {
	InitClientServer(muxServer *http.ServeMux)
}

type CodeList

type CodeList struct {
	Code []string `json:"codes"`
}

type FilterObject

type FilterObject struct {
	Type  string
	Value string
}

type HttpChannel

type HttpChannel struct {
}

type HttpServer

type HttpServer struct {
}

func (HttpServer) InitClientServer

func (server HttpServer) InitClientServer(muxServer *http.ServeMux)

type HttpWSsession

type HttpWSsession struct {
}

func (HttpWSsession) TransportHubFrontendWSsession

func (httpCoreSocketSession HttpWSsession) TransportHubFrontendWSsession(dataConn *websocket.Conn, appClientChannel []chan string)

type PathList

type PathList struct {
	Path []string `json:"LeafPaths"`
}

type RegData

type RegData struct {
	Portnum int
	Urlpath string
	Mgrid   int
}

type SecConfig

type SecConfig struct {
	TransportSec  string `json:"transportSec"`  // "yes" or "no"
	SecPort       string `json:"secPort"`       // port number
	CaSecPath     string `json:"caSecPath"`     // relative path from the directory containing the transportSec.json file
	ServerSecPath string `json:"serverSecPath"` // relative path from the directory containing the transportSec.json file
	ServerCertOpt string `json:"serverCertOpt"` // one of  "NoClientCert"/"ClientCertNoVerification"/"ClientCertVerification"
	ClientSecPath string `json:"clientSecPath"` // relative path from the directory containing the transportSec.json file
}

type TransportHubFrontendWSSession

type TransportHubFrontendWSSession interface {
	// contains filtered or unexported methods
}

**********Server Core Communications *******************************************************************************

type WsChannel

type WsChannel struct {
	// contains filtered or unexported fields
}

type WsServer

type WsServer struct {
	ClientBackendChannel []chan string
}

func (WsServer) InitClientServer

func (server WsServer) InitClientServer(muxServer *http.ServeMux, serverIndex *int)

type WsWSsession

type WsWSsession struct {
	ClientBackendChannel []chan string
}

func (WsWSsession) TransportHubFrontendWSsession

func (wsCoreSocketSession WsWSsession) TransportHubFrontendWSsession(dataConn *websocket.Conn, appClientChannel []chan string)