x2j

package
v1.8.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 21, 2019 License: BSD-3-Clause, MIT, BSD-3-Clause Imports: 11 Imported by: 1

README 

x2j.go - Unmarshal arbitrary XML docs to map[string]interface{} or JSON and extract values (using wildcards, if necessary).

NOTICE:

26mar18: This is provided to aid transitioning applications using the "x2j" package to the "mxj" package.

USAGE

The package is fairly well self-documented. (http://godoc.org/github.com/clbanning/x2j)  
The one really useful function is:

    - Unmarshal(doc []byte, v interface{}) error  
      where v is a pointer to a variable of type 'map[string]interface{}', 'string', or
      any other type supported by xml.Unmarshal().

To retrieve a value for specific tag use: 

    - DocValue(doc, path string, attrs ...string) (interface{},error) 
    - MapValue(m map[string]interface{}, path string, attr map[string]interface{}, recast ...bool) (interface{}, error)

The 'path' argument is a period-separated tag hierarchy - also known as dot-notation.
It is the program's responsibility to cast the returned value to the proper type; possible 
types are the normal JSON unmarshaling types: string, float64, bool, []interface, map[string]interface{}.  

To retrieve all values associated with a tag occurring anywhere in the XML document use:

    - ValuesForTag(doc, tag string) ([]interface{}, error)
    - ValuesForKey(m map[string]interface{}, key string) []interface{}

    Demos: http://play.golang.org/p/m8zP-cpk0O
           http://play.golang.org/p/cIteTS1iSg
           http://play.golang.org/p/vd8pMiI21b

Returned values should be one of map[string]interface, []interface{}, or string.

All the values assocated with a tag-path that may include one or more wildcard characters - 
'*' - can also be retrieved using:

    - ValuesFromTagPath(doc, path string, getAttrs ...bool) ([]interface{}, error)
    - ValuesFromKeyPath(map[string]interface{}, path string, getAttrs ...bool) []interface{}

    Demos: http://play.golang.org/p/kUQnZ8VuhS
           http://play.golang.org/p/l1aMHYtz7G

NOTE: care should be taken when using "*" at the end of a path - i.e., "books.book.*".  See
the x2jpath_test.go case on how the wildcard returns all key values and collapses list values;
the same message structure can load a []interface{} or a map[string]interface{} (or an interface{}) 
value for a tag.

See the test cases in "x2jpath_test.go" and programs in "example" subdirectory for more.

XML PARSING CONVENTIONS

   - Attributes are parsed to map[string]interface{} values by prefixing a hyphen, '-',
     to the attribute label. [See https://godoc.org/github.com/clbanning/mxj#SetAttrPrefix for options.]
   - If the element is a simple element and has attributes, the element value
     is given the key '#text' for its map[string]interface{} representation.  (See
     the 'atomFeedString.xml' test data, below.)

BULK PROCESSING OF MESSAGE FILES

Sometime messages may be logged into files for transmission via FTP (e.g.) and subsequent
processing. You can use the bulk XML message processor to convert files of XML messages into 
map[string]interface{} values with custom processing and error handler functions.  See
the notes and test code for:

   - XmlMsgsFromFile(fname string, phandler func(map[string]interface{}) bool, ehandler func(error) bool,recast ...bool) error
     [See https://godoc.org/github.com/clbanning/mxj#HandleXmlReader for general version.]

IMPLEMENTATION NOTES

Nothing fancy here, just brute force.

   - Use xml.Decoder to parse the XML doc and build a tree.
   - Walk the tree and load values into a map[string]interface{} variable, 'm', as
     appropriate.
   - Use json.Marshaler to convert 'm' to JSON.

As for testing:

   - Copy an XML doc into 'x2j_test.xml'.
   - Run "go test" and you'll get a full dump.
     ("pathTestString.xml" and "atomFeedString.xml" are test data from "read_test.go"
     in the encoding/xml directory of the standard package library.)

MOTIVATION

I make extensive use of JSON for messaging and typically unmarshal the messages into
map[string]interface{} variables.  This is easily done using json.Unmarshal from the
standard Go libraries.  Unfortunately, many legacy solutions use structured
XML messages; in those environments the applications would have to be refitted to
interoperate with my components.

The better solution is to just provide an alternative HTTP handler that receives
XML doc messages and parses it into a map[string]interface{} variable and then reuse
all the JSON-based code.  The Go xml.Unmarshal() function does not provide the same
option of unmarshaling XML messages into map[string]interface{} variables. So I wrote
a couple of small functions to fill this gap.

Of course, once the XML doc was unmarshal'd into a map[string]interface{} variable it
was just a matter of calling json.Marshal() to provide it as a JSON string.  Hence 'x2j'
rather than just 'x2m'.

USES

   - putting a XML API on our message hub middleware (http://jsonhub.net)
   - loading XML data into NoSQL database, such as, mongoDB

Documentation

Overview

Unmarshal dynamic / arbitrary XML docs and extract values (using wildcards, if necessary). THIS IS ONLY PROVIDED TO FACILIATE MIGRATING TO "mxj" PACKAGE FROM "x2j" PACKAGE.

NOTICE: 03mar18, package mostly replicates github.com/clbanning/x2j using github.com/clbanning/mxj 

(Note: there is no concept of Node or Tree; only direct decoding to map[string]interface{}.)

One useful function is:

  • Unmarshal(doc []byte, v interface{}) error where v is a pointer to a variable of type 'map[string]interface{}', 'string', or any other type supported by xml.Unmarshal().

To retrieve a value for specific tag use:

  • DocValue(doc, path string, attrs ...string) (interface{},error)
  • MapValue(m map[string]interface{}, path string, attr map[string]interface{}, recast ...bool) (interface{}, error)

The 'path' argument is a period-separated tag hierarchy - also known as dot-notation. It is the program's responsibility to cast the returned value to the proper type; possible types are the normal JSON unmarshaling types: string, float64, bool, []interface, map[string]interface{}.

To retrieve all values associated with a tag occurring anywhere in the XML document use:

Returned values should be one of map[string]interface, []interface{}, or string.

All the values assocated with a tag-path that may include one or more wildcard characters - '*' - can also be retrieved using:

NOTE: care should be taken when using "*" at the end of a path - i.e., "books.book.*". See the x2jpath_test.go case on how the wildcard returns all key values and collapses list values; the same message structure can load a []interface{} or a map[string]interface{} (or an interface{}) value for a tag.

See the test cases in "x2jpath_test.go" and programs in "example" subdirectory for more.

XML PARSING CONVENTIONS

  • Attributes are parsed to map[string]interface{} values by prefixing a hyphen, '-', to the attribute label.

  • If the element is a simple element and has attributes, the element value is given the key '#text' for its map[string]interface{} representation. (See the 'atomFeedString.xml' test data, below.)

    io.Reader HANDLING

    ToMap(), ToJson(), and ToJsonIndent() provide parsing of messages from an io.Reader. If you want to handle a message stream, look at XmlMsgsFromReader().

    NON-UTF8 CHARACTER SETS

    Use the X2jCharsetReader variable to assign io.Reader for alternative character sets.

Index

Constants

This section is empty.

Variables

View Source 
var X2jCharsetReader func(charset string, input io.Reader) (io.Reader, error)

If X2jCharsetReader != nil, it will be used to decode the doc or stream if required 

import charset "code.google.com/p/go-charset/charset"
...
x2j.X2jCharsetReader = charset.NewReader
s, err := x2j.DocToJson(doc)

Functions

func ByteDocToJson 

func ByteDocToJson(doc []byte, recast ...bool) (string, error)

ByteDocToJson - return an XML doc as a JSON string. 

If the optional argument 'recast' is 'true', then values will be converted to boolean or float64 if possible.

func ByteDocToMap 

func ByteDocToMap(doc []byte, recast ...bool) (map[string]interface{}, error)

ByteDocToMap - convert an XML doc into a map[string]interface{}. (This is analogous to unmarshalling a JSON string to map[string]interface{} using json.Unmarshal().) 

If the optional argument 'recast' is 'true', then values will be converted to boolean or float64 if possible.
Note: recasting is only applied to element values, not attribute values.

func BytePathForTagShortest 

func BytePathForTagShortest(doc []byte, key string) (string, error)

Extract the shortest path from all possible paths - from PathsForTag(). Paths are strings using dot-notation.

func BytePathsForTag 

func BytePathsForTag(doc []byte, key string) ([]string, error)

Get all paths through the doc (in dot-notation) that terminate with the specified tag. Results can be used with ValuesAtTagPath() and ValuesFromTagPath().

func CastNanInf 

func CastNanInf(b bool)

Cast "Nan", "Inf", "-Inf" XML values to 'float64'. By default, these values will be decoded as 'string'.

func DocToJson 

func DocToJson(doc string, recast ...bool) (string, error)

DocToJson - return an XML doc as a JSON string. 

If the optional argument 'recast' is 'true', then values will be converted to boolean or float64 if possible.

func DocToJsonIndent 

func DocToJsonIndent(doc string, recast ...bool) (string, error)

DocToJsonIndent - return an XML doc as a prettified JSON string. 

If the optional argument 'recast' is 'true', then values will be converted to boolean or float64 if possible.
Note: recasting is only applied to element values, not attribute values.

func DocToMap 

func DocToMap(doc string, recast ...bool) (map[string]interface{}, error)

DocToMap - convert an XML doc into a map[string]interface{}. (This is analogous to unmarshalling a JSON string to map[string]interface{} using json.Unmarshal().) 

If the optional argument 'recast' is 'true', then values will be converted to boolean or float64 if possible.
Note: recasting is only applied to element values, not attribute values.

func DocValue 

func DocValue(doc, path string, attrs ...string) (interface{}, error)

DocValue - return a value for a specific tag 

'doc' is a valid XML message.
'path' is a hierarchy of XML tags, e.g., "doc.name".
'attrs' is an OPTIONAL list of "name:value" pairs for attributes.
Note: 'recast' is not enabled here. Use DocToMap(), NewAttributeMap(), and MapValue() calls for that.

func MapValue 

func MapValue(m map[string]interface{}, path string, attr map[string]interface{}, r ...bool) (interface{}, error)

MapValue - retrieves value based on walking the map, 'm'. 

'm' is the map value of interest.
'path' is a period-separated hierarchy of keys in the map.
'attr' is a map of attribute "name:value" pairs from NewAttributeMap().  May be 'nil'.
If the path can't be traversed, an error is returned.
Note: the optional argument 'r' can be used to coerce attribute values, 'attr', if done so for 'm'.

func NewAttributeMap 

func NewAttributeMap(kv ...string) (map[string]interface{}, error)

NewAttributeMap() - generate map of attributes=value entries as map["-"+string]string. 

'kv' arguments are "name:value" pairs that appear as attributes, name="value".
If len(kv) == 0, the return is (nil, nil).

func PathForKeyShortest 

func PathForKeyShortest(m map[string]interface{}, key string) string

Extract the shortest path from all possible paths - from PathsForKey(). Paths are strings using dot-notation.

func PathForTagShortest 

func PathForTagShortest(doc string, key string) (string, error)

Extract the shortest path from all possible paths - from PathsForTag(). Paths are strings using dot-notation.

func PathsForKey 

func PathsForKey(m map[string]interface{}, key string) []string

Get all paths through the map (in dot-notation) that terminate with the specified key. Results can be used with ValuesAtKeyPath() and ValuesFromKeyPath().

func PathsForTag 

func PathsForTag(doc string, key string) ([]string, error)

Get all paths through the doc (in dot-notation) that terminate with the specified tag. Results can be used with ValuesAtTagPath() and ValuesFromTagPath().

func ReaderValuesForTag 

func ReaderValuesForTag(rdr io.Reader, tag string) ([]interface{}, error)

ReaderValuesForTag - io.Reader version of ValuesForTag()

func ReaderValuesFromTagPath 

func ReaderValuesFromTagPath(rdr io.Reader, path string, getAttrs ...bool) ([]interface{}, error)

ReaderValuesFromTagPath - io.Reader version of ValuesFromTagPath()

func ToJson 

func ToJson(rdr io.Reader, recast ...bool) (string, error)

ToJson() - parse a XML io.Reader to a JSON string

func ToJsonIndent 

func ToJsonIndent(rdr io.Reader, recast ...bool) (string, error)

ToJsonIndent - the pretty form of ReaderToJson

func ToMap 

func ToMap(rdr io.Reader, recast ...bool) (map[string]interface{}, error)

ToMap() - parse a XML io.Reader to a map[string]interface{}

func Unmarshal 

func Unmarshal(doc []byte, v interface{}) error

Unmarshal - wraps xml.Unmarshal with handling of map[string]interface{} and string type variables. 

Usage: x2j.Unmarshal(doc,&m) where m of type map[string]interface{}
       x2j.Unmarshal(doc,&s) where s of type string (Overrides xml.Unmarshal().)
       x2j.Unmarshal(doc,&struct) - passed to xml.Unmarshal()
       x2j.Unmarshal(doc,&slice) - passed to xml.Unmarshal()

func ValuesAtKeyPath 

func ValuesAtKeyPath(m map[string]interface{}, path string, getAttrs ...bool) []interface{}

ValuesAtKeyPath - deliver all values at the same depth in a map[string]interface{} value 

If v := ValuesAtKeyPath(m,"x.y.z")
then there exists a _,vv := range v
such that v.(map[string]interface{})[z] == ValuesFromKeyPath(m,"x.y.z")

If there are no values for the path 'nil' is returned. 

'm' is the map to be walked
'path' is a dot-separated path of key values
'getAttrs' can be set 'true' to return attribute values for "*"-terminated path
       If a node is '*', then everything beyond is walked.
       E.g., see ValuesFromTagPath documentation.

func ValuesAtTagPath 

func ValuesAtTagPath(doc, path string, getAttrs ...bool) ([]interface{}, error)

ValuesAtTagPath - deliver all values at the same level of the document as the specified key. 

See ValuesAtKeyPath().

If there are no values for the path 'nil' is returned. A return value of (nil, nil) means that there were no values and no errors parsing the doc. 

'doc' is the XML document
'path' is a dot-separated path of tag nodes
'getAttrs' can be set 'true' to return attribute values for "*"-terminated path
       If a node is '*', then everything beyond is scanned for values.
       E.g., "doc.books' might return a single value 'book' of type []interface{}, but
             "doc.books.*" could return all the 'book' entries as []map[string]interface{}.
             "doc.books.*.author" might return all the 'author' tag values as []string - or
         		"doc.books.*.author.lastname" might be required, depending on he schema.

func ValuesForKey 

func ValuesForKey(m map[string]interface{}, key string) []interface{}

ValuesForKey - return all values in map associated with 'key' 

Returns nil if the 'key' does not occur in the map

func ValuesForTag 

func ValuesForTag(doc, tag string) ([]interface{}, error)

ValuesForTag - return all values in doc associated with 'tag'. 

Returns nil if the 'tag' does not occur in the doc.
If there is an error encounted while parsing doc, that is returned.
If you want values 'recast' use DocToMap() and ValuesForKey().

func ValuesFromKeyPath 

func ValuesFromKeyPath(m map[string]interface{}, path string, getAttrs ...bool) []interface{}

ValuesFromKeyPath - deliver all values for a path node from a map[string]interface{} If there are no values for the path 'nil' is returned. 

'm' is the map to be walked
'path' is a dot-separated path of key values
'getAttrs' can be set 'true' to return attribute values for "*"-terminated path
       If a node is '*', then everything beyond is walked.
       E.g., see ValuesFromTagPath documentation.

func ValuesFromTagPath 

func ValuesFromTagPath(doc, path string, getAttrs ...bool) ([]interface{}, error)

ValuesFromTagPath - deliver all values for a path node from a XML doc If there are no values for the path 'nil' is returned. A return value of (nil, nil) means that there were no values and no errors parsing the doc. 

'doc' is the XML document
'path' is a dot-separated path of tag nodes
'getAttrs' can be set 'true' to return attribute values for "*"-terminated path
       If a node is '*', then everything beyond is scanned for values.
       E.g., "doc.books' might return a single value 'book' of type []interface{}, but
             "doc.books.*" could return all the 'book' entries as []map[string]interface{}.
             "doc.books.*.author" might return all the 'author' tag values as []string - or
         		"doc.books.*.author.lastname" might be required, depending on he schema.

func WriteMap 

func WriteMap(m interface{}, offset ...int) string

WriteMap - dumps the map[string]interface{} for examination. 

'offset' is initial indentation count; typically: WriteMap(m).
NOTE: with XML all element types are 'string'.
But code written as generic for use with maps[string]interface{} values from json.Unmarshal().
Or it can handle a DocToMap(doc,true) result where values have been recast'd.

func XmlBufferToJson 

func XmlBufferToJson(b *bytes.Buffer, recast ...bool) (string, error)

XmlBufferToJson - process XML message from a bytes.Buffer 

'b' is the buffer
Optional argument 'recast' coerces values to float64 or bool where possible.

func XmlBufferToMap 

func XmlBufferToMap(b *bytes.Buffer, recast ...bool) (map[string]interface{}, error)

XmlBufferToMap - process XML message from a bytes.Buffer 

'b' is the buffer
Optional argument 'recast' coerces map values to float64 or bool where possible.

func XmlMsgsFromFile 

func XmlMsgsFromFile(fname string, phandler func(map[string]interface{}) bool, ehandler func(error) bool, recast ...bool) error

XmlMsgsFromFile() 

'fname' is name of file
'phandler' is the map processing handler. Return of 'false' stops further processing.
'ehandler' is the parsing error handler. Return of 'false' stops further processing and returns error.
Note: phandler() and ehandler() calls are blocking, so reading and processing of messages is serialized.
      This means that you can stop reading the file on error or after processing a particular message.
      To have reading and handling run concurrently, pass arguments to a go routine in handler and return true.

func XmlMsgsFromFileAsJson 

func XmlMsgsFromFileAsJson(fname string, phandler func(string) bool, ehandler func(error) bool, recast ...bool) error

XmlMsgsFromFileAsJson() 

'fname' is name of file
'phandler' is the JSON string processing handler. Return of 'false' stops further processing.
'ehandler' is the parsing error handler. Return of 'false' stops further processing and returns error.
Note: phandler() and ehandler() calls are blocking, so reading and processing of messages is serialized.
      This means that you can stop reading the file on error or after processing a particular message.
      To have reading and handling run concurrently, pass arguments to a go routine in handler and return true.

func XmlMsgsFromReader 

func XmlMsgsFromReader(rdr io.Reader, phandler func(map[string]interface{}) bool, ehandler func(error) bool, recast ...bool) error

XmlMsgsFromReader() - io.Reader version of XmlMsgsFromFile 

'rdr' is an io.Reader for an XML message (stream)
'phandler' is the map processing handler. Return of 'false' stops further processing.
'ehandler' is the parsing error handler. Return of 'false' stops further processing and returns error.
Note: phandler() and ehandler() calls are blocking, so reading and processing of messages is serialized.
      This means that you can stop reading the file on error or after processing a particular message.
      To have reading and handling run concurrently, pass arguments to a go routine in handler and return true.

func XmlMsgsFromReaderAsJson 

func XmlMsgsFromReaderAsJson(rdr io.Reader, phandler func(string) bool, ehandler func(error) bool, recast ...bool) error

XmlMsgsFromReaderAsJson() - io.Reader version of XmlMsgsFromFileAsJson 

'rdr' is an io.Reader for an XML message (stream)
'phandler' is the JSON string processing handler. Return of 'false' stops further processing.
'ehandler' is the parsing error handler. Return of 'false' stops further processing and returns error.
Note: phandler() and ehandler() calls are blocking, so reading and processing of messages is serialized.
      This means that you can stop reading the file on error or after processing a particular message.
      To have reading and handling run concurrently, pass arguments to a go routine in handler and return true.

Types

This section is empty.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.