xmlquery

package module
v1.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 24, 2024 License: MIT Imports: 12 Imported by: 519

README

xmlquery

Build Status GoDoc Go Report Card

Overview

xmlquery is an XPath query package for XML documents, allowing you to extract data or evaluate from XML documents with an XPath expression.

xmlquery has a built-in query object caching feature that caches recently used XPATH query strings. Enabling caching can avoid recompile XPath expression for each query.

You can visit this page to learn about the supported XPath(1.0/2.0) syntax. https://github.com/antchfx/xpath

htmlquery - Package for the HTML document query.

xmlquery - Package for the XML document query.

jsonquery - Package for the JSON document query.

Installation

 $ go get github.com/antchfx/xmlquery

Quick Starts

import (
	"github.com/antchfx/xmlquery"
)

func main(){
	s := `<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
<channel>
  <title>W3Schools Home Page</title>
  <link>https://www.w3schools.com</link>
  <description>Free web building tutorials</description>
  <item>
    <title>RSS Tutorial</title>
    <link>https://www.w3schools.com/xml/xml_rss.asp</link>
    <description>New RSS tutorial on W3Schools</description>
  </item>
  <item>
    <title>XML Tutorial</title>
    <link>https://www.w3schools.com/xml</link>
    <description>New XML tutorial on W3Schools</description>
  </item>
</channel>
</rss>`

	doc, err := xmlquery.Parse(strings.NewReader(s))
	if err != nil {
		panic(err)
	}
	channel := xmlquery.FindOne(doc, "//channel")
	if n := channel.SelectElement("title"); n != nil {
		fmt.Printf("title: %s\n", n.InnerText())
	}
	if n := channel.SelectElement("link"); n != nil {
		fmt.Printf("link: %s\n", n.InnerText())
	}
	for i, n := range xmlquery.Find(doc, "//item/title") {
		fmt.Printf("#%d %s\n", i, n.InnerText())
	}
}

Getting Started

Find specified XPath query.
list, err := xmlquery.QueryAll(doc, "a")
if err != nil {
	panic(err)
}
Parse an XML from URL.
doc, err := xmlquery.LoadURL("http://www.example.com/sitemap.xml")
Parse an XML from string.
s := `<?xml version="1.0" encoding="utf-8"?><rss version="2.0"></rss>`
doc, err := xmlquery.Parse(strings.NewReader(s))
Parse an XML from io.Reader.
f, err := os.Open("../books.xml")
doc, err := xmlquery.Parse(f)
Parse an XML in a stream fashion (simple case without elements filtering).
f, _ := os.Open("../books.xml")
p, err := xmlquery.CreateStreamParser(f, "/bookstore/book")
for {
	n, err := p.Read()
	if err == io.EOF {
		break
	}
	if err != nil {
		panic(err)
	}
	fmt.Println(n)
}

Notes: CreateStreamParser() used for saving memory if your had a large XML file to parse.

Parse an XML in a stream fashion (simple case advanced element filtering).
f, _ := os.Open("../books.xml")
p, err := xmlquery.CreateStreamParser(f, "/bookstore/book", "/bookstore/book[price>=10]")
for {
	n, err := p.Read()
	if err == io.EOF {
		break
	}
	if err != nil {
		panic(err)
	}
	fmt.Println(n)
}
Find authors of all books in the bookstore.
list := xmlquery.Find(doc, "//book//author")
// or
list := xmlquery.Find(doc, "//author")
Find the second book.
book := xmlquery.FindOne(doc, "//book[2]")
Find the last book.
book := xmlquery.FindOne(doc, "//book[last()]")
Find all book elements and only get id attribute.
list := xmlquery.Find(doc,"//book/@id")
fmt.Println(list[0].InnerText) // outout @id value
Find all books with id bk104.
list := xmlquery.Find(doc, "//book[@id='bk104']")
Find all books with price less than 5.
list := xmlquery.Find(doc, "//book[price<5]")
Evaluate total price of all books.
expr, err := xpath.Compile("sum(//book/price)")
price := expr.Evaluate(xmlquery.CreateXPathNavigator(doc)).(float64)
fmt.Printf("total price: %f\n", price)
Count the number of books.
expr, err := xpath.Compile("count(//book)")
count := expr.Evaluate(xmlquery.CreateXPathNavigator(doc)).(float64)
Calculate the total price of all book prices.
expr, err := xpath.Compile("sum(//book/price)")
price := expr.Evaluate(xmlquery.CreateXPathNavigator(doc)).(float64)

Advanced Features

Parse UTF-16 XML file with ParseWithOptions().
f, _ := os.Open(`UTF-16.XML`)
// Convert UTF-16 XML to UTF-8
utf16ToUtf8Transformer := unicode.UTF16(unicode.LittleEndian, unicode.IgnoreBOM).NewDecoder()
utf8Reader := transform.NewReader(f, utf16ToUtf8Transformer)
// Sets `CharsetReader`
options := xmlquery.ParserOptions{
	Decoder: &xmlquery.DecoderOptions{
		CharsetReader: func(charset string, input io.Reader) (io.Reader, error) {
			return input, nil
		},
	},
}
doc, err := xmlquery.ParseWithOptions(utf8Reader, options)
Query with custom namespace prefix.
s := `<?xml version="1.0" encoding="UTF-8"?>
<pd:ProcessDefinition xmlns:pd="http://xmlns.xyz.com/process/2003" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<pd:activity name="Invoke Request-Response Service">
<pd:type>RequestReplyActivity</pd:type>
<pd:resourceType>OpClientReqActivity</pd:resourceType>
<pd:x>300</pd:x>
<pd:y>80</pd:y>
</pd:activity>
</pd:ProcessDefinition>`
nsMap := map[string]string{
	"q": "http://xmlns.xyz.com/process/2003",
	"r": "http://www.w3.org/1999/XSL/Transform",
	"s": "http://www.w3.org/2001/XMLSchema",
}
expr, _ := xpath.CompileWithNS("//q:activity", nsMap)
node := xmlquery.QuerySelector(doc, expr)
Create XML document without call xml.Marshal.
doc := &xmlquery.Node{
	Type: xmlquery.DeclarationNode,
	Data: "xml",
	Attr: []xml.Attr{
		xml.Attr{Name: xml.Name{Local: "version"}, Value: "1.0"},
	},
}
root := &xmlquery.Node{
	Data: "rss",
	Type: xmlquery.ElementNode,
}
doc.FirstChild = root
channel := &xmlquery.Node{
	Data: "channel",
	Type: xmlquery.ElementNode,
}
root.FirstChild = channel
title := &xmlquery.Node{
	Data: "title",
	Type: xmlquery.ElementNode,
}
title_text := &xmlquery.Node{
	Data: "W3Schools Home Page",
	Type: xmlquery.TextNode,
}
title.FirstChild = title_text
channel.FirstChild = title

fmt.Println(doc.OutputXML(true))
fmt.Println(doc.OutputXMLWithOptions(WithOutputSelf()))

Output:

<?xml version="1.0"?><rss><channel><title>W3Schools Home Page</title></channel></rss>

FAQ

Find() vs QueryAll(), which is better?

Find and QueryAll both do the same thing: searches all of matched XML nodes. Find panics if provided with an invalid XPath query, while QueryAll returns an error.

Can I save my query expression object for the next query?

Yes, you can. We provide QuerySelector and QuerySelectorAll methods; they accept your query expression object.

Caching a query expression object avoids recompiling the XPath query expression, improving query performance.

Questions

Please let me know if you have any questions

Documentation

Overview

Package xmlquery provides extract data from XML documents using XPath expression.

Example 
package main

import (
	"fmt"
	"strings"

	"github.com/antchfx/xmlquery"
	"github.com/antchfx/xpath"
)

func main() {
	// XPATH syntax and functions see https://github.com/antchfx/xpath
	s := `<?xml version="1.0"?>
	<bookstore>
	   <book id="bk101">
		  <author>Gambardella, Matthew</author>
		  <title>XML Developer's Guide</title>
		  <genre>Computer</genre>
		  <price>44.95</price>
		  <publish_date>2000-10-01</publish_date>
	   </book>
	   <book id="bk102">
		  <author>Ralls, Kim</author>
		  <title>Midnight Rain</title>
		  <genre>Fantasy</genre>
		  <price>5.95</price>
		  <publish_date>2000-12-16</publish_date>
	   </book>
	   <book id="bk103">
		  <author>Corets, Eva</author>
		  <title>Maeve Ascendant</title>
		  <genre>Fantasy</genre>
		  <price>5.95</price>
		  <publish_date>2000-11-17</publish_date>
	   </book>
	</bookstore>`

	doc, err := xmlquery.Parse(strings.NewReader(s))
	if err != nil {
		panic(err)
	}
	// Quick query all books.
	books := xmlquery.Find(doc, `/bookstore/book`)
	for _, n := range books {
		fmt.Println(n)
	}

	// Find all books with price rather than 10.
	books = xmlquery.Find(doc, `//book[price < 10]`)
	fmt.Println(len(books))

	// Find books with @id=bk102 or @id=bk101
	books = xmlquery.Find(doc, `//book[@id = "bk102" or @id = "bk101"]`)
	fmt.Println(len(books))

	// Find books by author: Corets, Eva
	book := xmlquery.FindOne(doc, `//book[author = "Corets, Eva"]`)
	fmt.Println(book.SelectElement("title").InnerText()) // > Output: Maeve Ascendant

	// Calculate the total prices of all books
	nav := xmlquery.CreateXPathNavigator(doc)
	prices := xpath.MustCompile(`sum(//book/price)`).Evaluate(nav).(float64)
	fmt.Println(prices) // > Output: 56.85
}

Output:

Index

Examples

Constants

This section is empty.

Variables

View Source 
var DisableSelectorCache = false

DisableSelectorCache will disable caching for the query selector if value is true.

View Source 
var SelectorCacheMaxEntries = 50

SelectorCacheMaxEntries allows how many selector object can be caching. Default is 50. Will disable caching if SelectorCacheMaxEntries <= 0.

Functions

func AddAttr added in v1.3.0 

func AddAttr(n *Node, key, val string)

AddAttr adds a new attribute specified by 'key' and 'val' to a node 'n'.

func AddChild added in v1.3.0 

func AddChild(parent, n *Node)

AddChild adds a new node 'n' to a node 'parent' as its last child.

func AddSibling added in v1.3.0 

func AddSibling(sibling, n *Node)

AddSibling adds a new node 'n' as a sibling of a given node 'sibling'. Note it is not necessarily true that the new node 'n' would be added immediately after 'sibling'. If 'sibling' isn't the last child of its parent, then the new node 'n' will be added at the end of the sibling chain of their parent.

func FindEach 

func FindEach(top *Node, expr string, cb func(int, *Node))

FindEach searches the html.Node and calls functions cb. Important: this method is deprecated, instead, use for .. = range Find(){}.

func FindEachWithBreak 

func FindEachWithBreak(top *Node, expr string, cb func(int, *Node) bool)

FindEachWithBreak functions the same as FindEach but allows to break the loop by returning false from the callback function `cb`. Important: this method is deprecated, instead, use .. = range Find(){}.

func RemoveFromTree added in v1.3.0 

func RemoveFromTree(n *Node)

RemoveFromTree removes a node and its subtree from the document tree it is in. If the node is the root of the tree, then it's no-op.

Types

type Attr added in v1.3.4 

type Attr struct {
	Name         xml.Name
	Value        string
	NamespaceURI string
}

type DecoderOptions added in v1.3.6 

type DecoderOptions struct {
	Strict        bool
	AutoClose     []string
	Entity        map[string]string
	CharsetReader func(charset string, input io.Reader) (io.Reader, error)
}

DecoderOptions implement the very same options than the standard encoding/xml package. Please refer to this documentation: https://golang.org/pkg/encoding/xml/#Decoder

type Node 

type Node struct {
	Parent, FirstChild, LastChild, PrevSibling, NextSibling *Node

	Type         NodeType
	Data         string
	Prefix       string
	NamespaceURI string
	Attr         []Attr
	// contains filtered or unexported fields
}

A Node consists of a NodeType and some Data (tag name for element nodes, content for text) and are part of a tree of Nodes.

func Find 

func Find(top *Node, expr string) []*Node

Find is like QueryAll but panics if `expr` is not a valid XPath expression. See `QueryAll()` function.

func FindOne 

func FindOne(top *Node, expr string) *Node

FindOne is like Query but panics if `expr` is not a valid XPath expression. See `Query()` function.

func LoadURL 

func LoadURL(url string) (*Node, error)

LoadURL loads the XML document from the specified URL.

func Parse 

func Parse(r io.Reader) (*Node, error)

Parse returns the parse tree for the XML from the given Reader.

func ParseWithOptions added in v1.3.6 

func ParseWithOptions(r io.Reader, options ParserOptions) (*Node, error)

ParseWithOptions is like parse, but with custom options

func Query added in v1.1.0 

func Query(top *Node, expr string) (*Node, error)

Query searches the XML Node that matches by the specified XPath expr, and returns first matched element.

func QueryAll added in v1.1.0 

func QueryAll(top *Node, expr string) ([]*Node, error)

QueryAll searches the XML Node that matches by the specified XPath expr. Returns an error if the expression `expr` cannot be parsed.

func QuerySelector added in v1.1.0 

func QuerySelector(top *Node, selector *xpath.Expr) *Node

QuerySelector returns the first matched XML Node by the specified XPath selector.

func QuerySelectorAll added in v1.1.0 

func QuerySelectorAll(top *Node, selector *xpath.Expr) []*Node

QuerySelectorAll searches all of the XML Node that matches the specified XPath selectors.

func (*Node) InnerText 

func (n *Node) InnerText() string

InnerText returns the text between the start and end tags of the object.

func (*Node) Level added in v1.4.0 

func (n *Node) Level() int

func (*Node) OutputXML 

func (n *Node) OutputXML(self bool) string

OutputXML returns the text that including tags name.

func (*Node) OutputXMLWithOptions added in v1.3.13 

func (n *Node) OutputXMLWithOptions(opts ...OutputOption) string

OutputXMLWithOptions returns the text that including tags name.

func (*Node) RemoveAttr added in v1.3.13 

func (n *Node) RemoveAttr(key string)

RemoveAttr removes the attribute with the specified name.

func (*Node) SelectAttr 

func (n *Node) SelectAttr(name string) string

SelectAttr returns the attribute value with the specified name.

func (*Node) SelectElement 

func (n *Node) SelectElement(name string) *Node

SelectElement finds child elements with the specified name.

func (*Node) SelectElements 

func (n *Node) SelectElements(name string) []*Node

SelectElements finds child elements with the specified name.

func (*Node) SetAttr added in v1.3.13 

func (n *Node) SetAttr(key, value string)

SetAttr allows an attribute value with the specified name to be changed. If the attribute did not previously exist, it will be created.

type NodeNavigator 

type NodeNavigator struct {
	// contains filtered or unexported fields
}

func CreateXPathNavigator 

func CreateXPathNavigator(top *Node) *NodeNavigator

CreateXPathNavigator creates a new xpath.NodeNavigator for the specified XML Node.

func (*NodeNavigator) Copy 

func (x *NodeNavigator) Copy() xpath.NodeNavigator

func (*NodeNavigator) Current 

func (x *NodeNavigator) Current() *Node

func (*NodeNavigator) LocalName 

func (x *NodeNavigator) LocalName() string

func (*NodeNavigator) MoveTo 

func (x *NodeNavigator) MoveTo(other xpath.NodeNavigator) bool

func (*NodeNavigator) MoveToChild 

func (x *NodeNavigator) MoveToChild() bool

func (*NodeNavigator) MoveToFirst 

func (x *NodeNavigator) MoveToFirst() bool

func (*NodeNavigator) MoveToNext 

func (x *NodeNavigator) MoveToNext() bool

func (*NodeNavigator) MoveToNextAttribute 

func (x *NodeNavigator) MoveToNextAttribute() bool

func (*NodeNavigator) MoveToParent 

func (x *NodeNavigator) MoveToParent() bool

func (*NodeNavigator) MoveToPrevious 

func (x *NodeNavigator) MoveToPrevious() bool

func (*NodeNavigator) MoveToRoot 

func (x *NodeNavigator) MoveToRoot()

func (*NodeNavigator) NamespaceURL added in v1.2.1 

func (x *NodeNavigator) NamespaceURL() string

func (*NodeNavigator) NodeType 

func (x *NodeNavigator) NodeType() xpath.NodeType

func (*NodeNavigator) Prefix 

func (x *NodeNavigator) Prefix() string

func (*NodeNavigator) String 

func (x *NodeNavigator) String() string

func (*NodeNavigator) Value 

func (x *NodeNavigator) Value() string

type NodeType 

type NodeType uint

A NodeType is the type of a Node. 

const (
	// DocumentNode is a document object that, as the root of the document tree,
	// provides access to the entire XML document.
	DocumentNode NodeType = iota
	// DeclarationNode is the document type declaration, indicated by the
	// following tag (for example, <!DOCTYPE...> ).
	DeclarationNode
	// ElementNode is an element (for example, <item> ).
	ElementNode
	// TextNode is the text content of a node.
	TextNode
	// CharDataNode node <![CDATA[content]]>
	CharDataNode
	// CommentNode a comment (for example, <!-- my comment --> ).
	CommentNode
	// AttributeNode is an attribute of element.
	AttributeNode
	// NotationNode is a directive represents in document (for example, <!text...>).
	NotationNode
)

type OutputOption added in v1.3.13 

type OutputOption func(*outputConfiguration)

func WithEmptyTagSupport added in v1.3.13 

func WithEmptyTagSupport() OutputOption

WithEmptyTagSupport empty tags should be written as <empty/> and not as <empty></empty>

func WithOutputSelf added in v1.3.13 

func WithOutputSelf() OutputOption

WithOutputSelf configures the Node to print the root node itself

func WithPreserveSpace added in v1.3.18 

func WithPreserveSpace() OutputOption

WithPreserveSpace will preserve spaces in output

func WithoutComments added in v1.3.13 

func WithoutComments() OutputOption

WithoutComments will skip comments in output

type ParserOptions added in v1.3.6 

type ParserOptions struct {
	Decoder *DecoderOptions
}

type StreamParser added in v1.3.0 

type StreamParser struct {
	// contains filtered or unexported fields
}

StreamParser enables loading and parsing an XML document in a streaming fashion.

func CreateStreamParser added in v1.3.0 

func CreateStreamParser(r io.Reader, streamElementXPath string, streamElementFilter ...string) (*StreamParser, error)

CreateStreamParser creates a StreamParser. Argument streamElementXPath is required. Argument streamElementFilter is optional and should only be used in advanced scenarios.

Scenario 1: simple case: 

xml := `<AAA><BBB>b1</BBB><BBB>b2</BBB></AAA>`
sp, err := CreateStreamParser(strings.NewReader(xml), "/AAA/BBB")
if err != nil {
    panic(err)
}
for {
    n, err := sp.Read()
    if err != nil {
        break
    }
    fmt.Println(n.OutputXML(true))
}

Output will be: 

<BBB>b1</BBB>
<BBB>b2</BBB>

Scenario 2: advanced case: 

xml := `<AAA><BBB>b1</BBB><BBB>b2</BBB></AAA>`
sp, err := CreateStreamParser(strings.NewReader(xml), "/AAA/BBB", "/AAA/BBB[. != 'b1']")
if err != nil {
    panic(err)
}
for {
    n, err := sp.Read()
    if err != nil {
        break
    }
    fmt.Println(n.OutputXML(true))
}

Output will be: 

<BBB>b2</BBB>

As the argument names indicate, streamElementXPath should be used for providing xpath query pointing to the target element node only, no extra filtering on the element itself or its children; while streamElementFilter, if needed, can provide additional filtering on the target element and its children.

CreateStreamParser returns an error if either streamElementXPath or streamElementFilter, if provided, cannot be successfully parsed and compiled into a valid xpath query.

func CreateStreamParserWithOptions added in v1.3.6 

func CreateStreamParserWithOptions(
	r io.Reader,
	options ParserOptions,
	streamElementXPath string,
	streamElementFilter ...string,
) (*StreamParser, error)

CreateStreamParserWithOptions is like CreateStreamParser, but with custom options

func (*StreamParser) Read added in v1.3.0 

func (sp *StreamParser) Read() (*Node, error)

Read returns a target node that satisfies the XPath specified by caller at StreamParser creation time. If there is no more satisfying target nodes after reading the rest of the XML document, io.EOF will be returned. At any time, any XML parsing error encountered will be returned, and the stream parsing stopped. Calling Read() after an error is returned (including io.EOF) results undefined behavior. Also note, due to the streaming nature, calling Read() will automatically remove any previous target node(s) from the document tree.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.