yaml

package module
v1.8.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 10, 2020 License: MIT Imports: 20 Imported by: 1,044

README

YAML support for the Go language

GoDoc CircleCI codecov Go Report Card

Why a new library?

As of this writing, there already exists a defacto standard library for YAML processing Go: https://github.com/go-yaml/yaml. However we feel that some features are lacking, namely:

  • Pretty format for error notifications
  • Directly manipulate the YAML abstract syntax tree
  • Support Anchor and Alias when marshaling
  • Allow referencing elements declared in another file via anchors

Features

  • Pretty format for error notifications
  • Support Scanner or Lexer or Parser as public API
  • Support Anchor and Alias to Marshaler
  • Allow referencing elements declared in another file via anchors
  • Extract value or AST by YAMLPath ( YAMLPath is like a JSONPath )

Installation

go get -u github.com/goccy/go-yaml

Synopsis

1. Simple Encode/Decode

Support compatible interface to go-yaml/yaml by using reflect

var v struct {
	A int
	B string
}
v.A = 1
v.B = "hello"
bytes, err := yaml.Marshal(v)
if err != nil {
	//...
}
fmt.Println(string(bytes)) // "a: 1\nb: hello\n"

	yml := `
%YAML 1.2
---
a: 1
b: c
`
var v struct {
	A int
	B string
}
if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
	//...
}

To control marshal/unmarshal behavior, you can use the yaml tag

	yml := `---
foo: 1
bar: c
`
var v struct {
	A int    `yaml:"foo"`
	B string `yaml:"bar"`
}
if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
	//...
}

For convenience, we also accept the json tag. Note that not all options from the json tag will have significance when parsing YAML documents. If both tags exist, yaml tag will take precedence.

	yml := `---
foo: 1
bar: c
`
var v struct {
	A int    `json:"foo"`
	B string `json:"bar"`
}
if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
	//...
}

For custom marshal/unmarshaling, implement either Bytes or Interface variant of marshaler/unmarshaler. The difference is that while BytesMarshaler/BytesUnmarshaler behaves like encoding/json and InterfaceMarshaler/InterfaceUnmarshaler behaves like gopkg.in/yaml.v2.

Semantically both are the same, but they differ in performance. Because indentation matter in YAML, you cannot simply accept a valid YAML fragment from a Marshaler, and expect it to work when it is attached to the parent container's serialized form. Therefore when we receive use the BytesMarshaler, which returns []byte, we must decode it once to figure out how to make it work in the given context. If you use the InterfaceMarshaler, we can skip the decoding.

If you are repeatedly marshaling complex objects, the latter is always better performance wise. But if you are, for example, just providing a choice between a config file format that is read only once, the former is probably easier to code.

2. Reference elements in declared in another file

testdata directory includes anchor.yml file

├── testdata
   └── anchor.yml

And anchor.yml is defined the following.

a: &a
  b: 1
  c: hello

Then, if yaml.ReferenceDirs("testdata") option passed to yaml.Decoder, Decoder try to find anchor definition from YAML files the under testdata directory.

buf := bytes.NewBufferString("a: *a\n")
dec := yaml.NewDecoder(buf, yaml.ReferenceDirs("testdata"))
var v struct {
	A struct {
		B int
		C string
	}
}
if err := dec.Decode(&v); err != nil {
	//...
}
fmt.Printf("%+v\n", v) // {A:{B:1 C:hello}}

3. Encode with Anchor and Alias

3.1. Explicitly declaration Anchor name and Alias name

If you want to use anchor or alias, you can define it as a struct tag.

type T struct {
  A int
  B string
}
var v struct {
  C *T `yaml:"c,anchor=x"`
  D *T `yaml:"d,alias=x"`
}
v.C = &T{A: 1, B: "hello"}
v.D = v.C
bytes, err := yaml.Marshal(v)
if err != nil {
  panic(err)
}
fmt.Println(string(bytes))
/*
c: &x
  a: 1
  b: hello
d: *x
*/
3.2. Implicitly declared Anchor and Alias names

If you do not explicitly declare the anchor name, the default behavior is to use the equivalent of strings.ToLower($FieldName) as the name of the anchor.

If you do not explicitly declare the alias name AND the value is a pointer to another element, we look up the anchor name by finding out which anchor field the value is assigned to by looking up its pointer address.

type T struct {
	I int
	S string
}
var v struct {
	A *T `yaml:"a,anchor"`
	B *T `yaml:"b,anchor"`
	C *T `yaml:"c,alias"`
	D *T `yaml:"d,alias"`
}
v.A = &T{I: 1, S: "hello"}
v.B = &T{I: 2, S: "world"}
v.C = v.A // C has same pointer address to A
v.D = v.B // D has same pointer address to B
bytes, err := yaml.Marshal(v)
if err != nil {
	//...
}
fmt.Println(string(bytes)) 
/*
a: &a
  i: 1
  s: hello
b: &b
  i: 2
  s: world
c: *a
d: *b
*/
3.3 MergeKey and Alias

Merge key and alias ( <<: *alias ) can be used by embedding a structure with the inline,alias tag .

type Person struct {
	*Person `yaml:",omitempty,inline,alias"` // embed Person type for default value
	Name    string `yaml:",omitempty"`
	Age     int    `yaml:",omitempty"`
}
defaultPerson := &Person{
	Name: "John Smith",
	Age:  20,
}
people := []*Person{
	{
		Person: defaultPerson, // assign default value
		Name:   "Ken",         // override Name property
		Age:    10,            // override Age property
	},
	{
		Person: defaultPerson, // assign default value only
	},
}
var doc struct {
	Default *Person   `yaml:"default,anchor"`
	People  []*Person `yaml:"people"`
}
doc.Default = defaultPerson
doc.People = people
bytes, err := yaml.Marshal(doc)
if err != nil {
	//...
}
fmt.Println(string(bytes))
/*
default: &default
  name: John Smith
  age: 20
people:
- <<: *default
  name: Ken
  age: 10
- <<: *default
*/

4. Pretty Formatted Errors

Error values produced during parsing has two extra features over regular error values.

First by default they contain extra information on the location of the error from the source YAML document, to make it easier finding the error location.

Second, the error messages can optionally be colorized.

If you would like to control exactly how the output looks like, consider using yaml.FormatError, which accepts two boolean values to control turning on/off these features

5. Use YAMLPath

yml := `
store:
  book:
    - author: john
      price: 10
    - author: ken
      price: 12
  bicycle:
    color: red
    price: 19.95
`
path, err := yaml.PathString("$.store.book[*].author")
if err != nil {
  //...
}
var authors []string
if err := path.Read(strings.NewReader(yml), &authors); err != nil {
  //...
}
fmt.Println(authors)
// [john ken]
5.1 Print customized error with YAML source code
package main

import (
  "fmt"

  "github.com/goccy/go-yaml"
)

func main() {
  yml := `
a: 1
b: "hello"
`
  var v struct {
    A int
    B string
  }
  if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
    panic(err)
  }
  if v.A != 2 {
    // output error with YAML source
    path, err := yaml.PathString("$.a")
    if err != nil {
      panic(err)
    }
    source, err := path.AnnotateSource([]byte(yml), true)
    if err != nil {
      panic(err)
    }
    fmt.Printf("a value expected 2 but actual %d:\n%s\n", v.A, string(source))
  }
}

output result is the following.

Tools

ycat

print yaml file with color

ycat
Installation
go get -u github.com/goccy/go-yaml/cmd/ycat

License

MIT

Documentation

Index

Examples

Constants

View Source 
const (
	// DefaultIndentSpaces default number of space for indent
	DefaultIndentSpaces = 2
)
View Source 
const (
	// StructTagName tag keyword for Marshal/Unmarshal
	StructTagName = "yaml"
)

Variables

View Source 
var (
	ErrInvalidQuery      = xerrors.New("invalid query")
	ErrInvalidPath       = xerrors.New("invalid path instance")
	ErrInvalidPathString = xerrors.New("invalid path string")
)

Functions

func FormatError 

func FormatError(e error, colored, inclSource bool) string

FormatError is a utility function that takes advantage of the metadata stored in the errors returned by this package's parser.

If the second argument `colored` is true, the error message is colorized. If the third argument `inclSource` is true, the error message will contain snippets of the YAML source that was used.

func Marshal 

func Marshal(v interface{}) ([]byte, error)

Marshal serializes the value provided into a YAML document. The structure of the generated document will reflect the structure of the value itself. Maps and pointers (to struct, string, int, etc) are accepted as the in value.

Struct fields are only marshalled if they are exported (have an upper case first letter), and are marshalled using the field name lowercased as the default key. Custom keys may be defined via the "yaml" name in the field tag: the content preceding the first comma is used as the key, and the following comma-separated options are used to tweak the marshalling process. Conflicting names result in a runtime error.

The field tag format accepted is: 

`(...) yaml:"[<key>][,<flag1>[,<flag2>]]" (...)`

The following flags are currently supported: 

omitempty    Only include the field if it's not set to the zero
             value for the type or to empty slices or maps.
             Zero valued structs will be omitted if all their public
             fields are zero, unless they implement an IsZero
             method (see the IsZeroer interface type), in which
             case the field will be included if that method returns true.

flow         Marshal using a flow style (useful for structs,
             sequences and maps).

inline       Inline the field, which must be a struct or a map,
             causing all of its fields or keys to be processed as if
             they were part of the outer struct. For maps, keys must
             not conflict with the yaml keys of other struct fields.

anchor       Marshal with anchor. If want to define anchor name explicitly, use anchor=name style.
             Otherwise, if used 'anchor' name only, used the field name lowercased as the anchor name

alias        Marshal with alias. If want to define alias name explicitly, use alias=name style.
             Otherwise, If omitted alias name and the field type is pointer type,
             assigned anchor name automatically from same pointer address.

In addition, if the key is "-", the field is ignored.

For example: 

type T struct {
    F int `yaml:"a,omitempty"`
    B int
}
yaml.Marshal(&T{B: 2}) // Returns "b: 2\n"
yaml.Marshal(&T{F: 1}) // Returns "a: 1\nb: 0\n"

func MarshalWithOptions added in v1.8.0 

func MarshalWithOptions(v interface{}, opts ...EncodeOption) ([]byte, error)

MarshalWithOptions serializes the value provided into a YAML document with EncodeOptions.

func Unmarshal 

func Unmarshal(data []byte, v interface{}) error

Unmarshal decodes the first document found within the in byte slice and assigns decoded values into the out value.

Struct fields are only unmarshalled if they are exported (have an upper case first letter), and are unmarshalled using the field name lowercased as the default key. Custom keys may be defined via the "yaml" name in the field tag: the content preceding the first comma is used as the key, and the following comma-separated options are used to tweak the marshalling process (see Marshal). Conflicting names result in a runtime error.

For example: 

type T struct {
    F int `yaml:"a,omitempty"`
    B int
}
var t T
yaml.Unmarshal([]byte("a: 1\nb: 2"), &t)

See the documentation of Marshal for the format of tags and a list of supported tag options.

func UnmarshalWithOptions added in v1.8.0 

func UnmarshalWithOptions(data []byte, v interface{}, opts ...DecodeOption) error

UnmarshalWithOptions decodes with DecodeOptions the first document found within the in byte slice and assigns decoded values into the out value.

func ValueToNode added in v1.8.0 

func ValueToNode(v interface{}, opts ...EncodeOption) (ast.Node, error)

ValueToNode convert from value to ast.Node.

Types

type BytesMarshaler 

type BytesMarshaler interface {
	MarshalYAML() ([]byte, error)
}

BytesMarshaler interface may be implemented by types to customize their behavior when being marshaled into a YAML document. The returned value is marshaled in place of the original value implementing Marshaler.

If an error is returned by MarshalYAML, the marshaling procedure stops and returns with the provided error.

type BytesUnmarshaler 

type BytesUnmarshaler interface {
	UnmarshalYAML([]byte) error
}

BytesUnmarshaler interface may be implemented by types to customize their behavior when being unmarshaled from a YAML document.

type DecodeOption 

type DecodeOption func(d *Decoder) error

DecodeOption functional option type for Decoder

func DisallowDuplicateKey added in v1.6.1 

func DisallowDuplicateKey() DecodeOption

DisallowDuplicateKey causes an error when mapping keys that are duplicates

func DisallowUnknownField added in v1.1.3 

func DisallowUnknownField() DecodeOption

DisallowUnknownField causes the Decoder to return an error when the destination is a struct and the input contains object keys which do not match any non-ignored, exported fields in the destination.

func RecursiveDir 

func RecursiveDir(isRecursive bool) DecodeOption

RecursiveDir search yaml file recursively from passed dirs by ReferenceDirs option

func ReferenceDirs 

func ReferenceDirs(dirs ...string) DecodeOption

ReferenceDirs pass to Decoder that reference to anchor defined by files under the passed dirs

func ReferenceFiles 

func ReferenceFiles(files ...string) DecodeOption

ReferenceFiles pass to Decoder that reference to anchor defined by passed files

func ReferenceReaders 

func ReferenceReaders(readers ...io.Reader) DecodeOption

ReferenceReaders pass to Decoder that reference to anchor defined by passed readers

func Strict added in v1.6.1 

func Strict() DecodeOption

Strict enable DisallowUnknownField and DisallowDuplicateKey

func UseOrderedMap added in v1.5.0 

func UseOrderedMap() DecodeOption

UseOrderedMap can be interpreted as a map, and uses MapSlice ( ordered map ) aggressively if there is no type specification

func Validator 

func Validator(v StructValidator) DecodeOption

Validator set StructValidator instance to Decoder

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder reads and decodes YAML values from an input stream.

func NewDecoder 

func NewDecoder(r io.Reader, opts ...DecodeOption) *Decoder

NewDecoder returns a new decoder that reads from r.

func (*Decoder) Decode 

func (d *Decoder) Decode(v interface{}) error

Decode reads the next YAML-encoded value from its input and stores it in the value pointed to by v.

See the documentation for Unmarshal for details about the conversion of YAML into a Go value.

type EncodeOption 

type EncodeOption func(e *Encoder) error

EncodeOption functional option type for Encoder

func Flow 

func Flow(isFlowStyle bool) EncodeOption

Flow encoding by flow style

func Indent 

func Indent(spaces int) EncodeOption

Indent change indent number

func JSON added in v1.6.3 

func JSON() EncodeOption

JSON encode in JSON format

func MarshalAnchor added in v1.1.9 

func MarshalAnchor(callback func(*ast.AnchorNode, interface{}) error) EncodeOption

MarshalAnchor call back if encoder find an anchor during encoding

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder writes YAML values to an output stream.

func NewEncoder 

func NewEncoder(w io.Writer, opts ...EncodeOption) *Encoder

NewEncoder returns a new encoder that writes to w. The Encoder should be closed after use to flush all data to w.

func (*Encoder) Close 

func (e *Encoder) Close() error

Close closes the encoder by writing any remaining data. It does not write a stream terminating string "...".

func (*Encoder) Encode 

func (e *Encoder) Encode(v interface{}) error

Encode writes the YAML encoding of v to the stream. If multiple items are encoded to the stream, the second and subsequent document will be preceded with a "---" document separator, but the first will not.

See the documentation for Marshal for details about the conversion of Go values to YAML.

func (*Encoder) EncodeToNode added in v1.8.0 

func (e *Encoder) EncodeToNode(v interface{}) (ast.Node, error)

EncodeToNode convert v to ast.Node.

type FieldError 

type FieldError interface {
	StructField() string
}

FieldError need to implement StructField method only ( see https://godoc.org/gopkg.in/go-playground/validator.v9#FieldError )

type InterfaceMarshaler 

type InterfaceMarshaler interface {
	MarshalYAML() (interface{}, error)
}

InterfaceMarshaler interface has MarshalYAML compatible with github.com/go-yaml/yaml package.

type InterfaceUnmarshaler 

type InterfaceUnmarshaler interface {
	UnmarshalYAML(func(interface{}) error) error
}

InterfaceUnmarshaler interface has UnmarshalYAML compatible with github.com/go-yaml/yaml package.

type IsZeroer 

type IsZeroer interface {
	IsZero() bool
}

IsZeroer is used to check whether an object is zero to determine whether it should be omitted when marshaling with the omitempty flag. One notable implementation is time.Time.

type MapItem 

type MapItem struct {
	Key, Value interface{}
}

MapItem is an item in a MapSlice.

type MapSlice 

type MapSlice []MapItem

MapSlice encodes and decodes as a YAML map. The order of keys is preserved when encoding and decoding.

func (MapSlice) ToMap added in v1.8.0 

func (s MapSlice) ToMap() map[interface{}]interface{}

ToMap convert to map[interface{}]interface{}.

type Path added in v1.7.0 

type Path struct {
	// contains filtered or unexported fields
}

Path represent YAMLPath ( like a JSONPath ).

func PathString added in v1.7.0 

func PathString(s string) (*Path, error)

PathString create Path from string

YAMLPath rule $ : the root object/element . : child operator .. : recursive descent [num] : object/element of array by number [*] : all objects/elements for array.

func (*Path) AnnotateSource added in v1.8.0 

func (p *Path) AnnotateSource(source []byte, colored bool) ([]byte, error)

AnnotateSource add annotation to passed source ( see section 5.1 in README.md ).

Example 
yml := `
a: 1
b: "hello"
`
var v struct {
	A int
	B string
}
if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
	panic(err)
}
if v.A != 2 {
	// output error with YAML source
	path, err := yaml.PathString("$.a")
	if err != nil {
		log.Fatal(err)
	}
	source, err := path.AnnotateSource([]byte(yml), false)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("a value expected 2 but actual %d:\n%s\n", v.A, string(source))
}

Output:

a value expected 2 but actual 1:
>  2 | a: 1
          ^
   3 | b: "hello"

func (*Path) Filter added in v1.7.0 

func (p *Path) Filter(target, v interface{}) error

Filter filter from target by YAMLPath and set it to v.

func (*Path) FilterFile added in v1.7.0 

func (p *Path) FilterFile(f *ast.File) (ast.Node, error)

FilterFile filter from ast.File by YAMLPath.

func (*Path) FilterNode added in v1.7.0 

func (p *Path) FilterNode(node ast.Node) (ast.Node, error)

FilterNode filter from node by YAMLPath.

func (*Path) MergeFromFile added in v1.8.0 

func (p *Path) MergeFromFile(dst *ast.File, src *ast.File) error

MergeFromFile merge ast.File into ast.File.

func (*Path) MergeFromNode added in v1.8.0 

func (p *Path) MergeFromNode(dst *ast.File, src ast.Node) error

MergeFromNode merge ast.Node into ast.File.

func (*Path) MergeFromReader added in v1.8.0 

func (p *Path) MergeFromReader(dst *ast.File, src io.Reader) error

MergeFromReader merge YAML text into ast.File.

func (*Path) Read added in v1.7.0 

func (p *Path) Read(r io.Reader, v interface{}) error

Read decode from r and set extracted value by YAMLPath to v.

func (*Path) ReadNode added in v1.7.0 

func (p *Path) ReadNode(r io.Reader) (ast.Node, error)

ReadNode create AST from r and extract node by YAMLPath.

func (*Path) ReplaceWithFile added in v1.8.0 

func (p *Path) ReplaceWithFile(dst *ast.File, src *ast.File) error

ReplaceWithFile replace ast.File with ast.File.

func (*Path) ReplaceWithNode added in v1.8.0 

func (p *Path) ReplaceWithNode(dst *ast.File, node ast.Node) error

ReplaceNode replace ast.File with ast.Node.

func (*Path) ReplaceWithReader added in v1.8.0 

func (p *Path) ReplaceWithReader(dst *ast.File, src io.Reader) error

ReplaceWithReader replace ast.File with io.Reader.

func (*Path) String added in v1.7.0 

func (p *Path) String() string

String path to text.

type PathBuilder added in v1.7.0 

type PathBuilder struct {
	// contains filtered or unexported fields
}

PathBuilder represent builder for YAMLPath.

func (*PathBuilder) Build added in v1.7.0 

func (b *PathBuilder) Build() *Path

Build build YAMLPath.

func (*PathBuilder) Child added in v1.7.0 

func (b *PathBuilder) Child(name string) *PathBuilder

Child add '.name' to current path.

func (*PathBuilder) Index added in v1.7.0 

func (b *PathBuilder) Index(idx uint) *PathBuilder

Index add '[idx]' to current path.

func (*PathBuilder) IndexAll added in v1.7.0 

func (b *PathBuilder) IndexAll() *PathBuilder

IndexAll add '[*]' to current path.

func (*PathBuilder) Recursive added in v1.7.0 

func (b *PathBuilder) Recursive(selector string) *PathBuilder

Recursive add '..selector' to current path.

func (*PathBuilder) Root added in v1.7.0 

func (b *PathBuilder) Root() *PathBuilder

Root add '$' to current path.

type StructField 

type StructField struct {
	FieldName    string
	RenderName   string
	AnchorName   string
	AliasName    string
	IsAutoAnchor bool
	IsAutoAlias  bool
	IsOmitEmpty  bool
	IsFlow       bool
	IsInline     bool
}

StructField information for each the field in structure

type StructFieldMap 

type StructFieldMap map[string]*StructField

type StructValidator 

type StructValidator interface {
	Struct(interface{}) error
}

StructValidator need to implement Struct method only ( see https://godoc.org/gopkg.in/go-playground/validator.v9#Validate.Struct )

Source Files

View all Source files

Directories

Path Synopsis
cmd
ycat
internal
errors

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.