xsx

README

– eXtended S-eXpressions

import "git.fractalqb.de/fractalqb/xsx"

---

Package XSX provides tools for parsing something I call eXtended S-eXpressions. Extended means the following things compared to SEXP S-expressions:

1. Nested structures are delimited by a configurable set of balanced brackets e.g., '()', '[]' or '{}’ – not only by '()'.

2. XSX provides a notation for "Meta Values", i.e. XSXs that provide some sort of out-of-band information.

On the other hand some properties from SEXP were dropped, e.g. typing of the so called "octet strings". Things like that are completely left to the application.

Somewhat more formal description

Frist of all, XSX is not about datatypes, in this it is comparable to e.g. XML (No! don't leave… its much simpler). Instead, its building block is the atom i.e., nothing else than a sequence of characters, aka a 'string'. Atoms come as quoted atoms and as unquoted atoms. An atom has to be quoted when the atom's string contains characters that have a special meaning in XSX: White-space, brackets, the quote and the meta character.

Regexp style definition of Atom

ws           := UNICODE whitespace
o-brackets   := “set of opening brackets” – e.g. '(' | '[' | '{'
c-brackets   := “set of closing brackets” – e.g. ')' | ']' | '}'
quote        := “the quote char”          – e.g. "
meta         := “the meta char”           – e.g. \
syntax-chars := o-brackets | c-brackets | quote | meta

atom     := nq-atom | q-atom
nq-atom  := (^(syntax-chars|ws))+
q-atom   := quote(quote{2}|^quote)*quote

I.e. x is an atom and foo, bar and baz are atoms. An atom that contains space, a '"' or '\' would be quoted. Also "(" is an atom but ( is not an atom.

Sequences now BNF Style

Each atom is an XSX and from XSX'es one can build sequences:

XSX      := atom | sequence
sequence := o-bracket {ws} c-bracket | o-bracket {ws} xsxs {ws} c-bracket
xsxs     := XSX | XSX {ws} xsxs

Out-Of-Band Information with Meta XSXs

You can prefix each XSX with the meta char to make that expression a meta-expression. A meta-expression is not considered to be an XSX, i.e. you cannot create meta-meta-expressions or meta-meta-meta-expressions… hmm… and not even meta-meta-meta-meta-expressions! I think it became clear?

The special case when the meta char does not prefix an XSX e.g., “\ ” or (\), makes the meta char the special meta token void. One might perhaps use it to denote tha absence of something.

E.g. \4711 is a meta-atom and \{foo 1 bar false baz 3.1415} is a meta-sequence. What meta means is completely up to the application. Imagine (div hiho) and (div \{class green} hiho) to be a translation from <div>hiho</div> and <div class="green">hiho</div>.

Rationale

None! … despite the fact that I found it to be fun – and useful in some situations. Because XSX syntax so simple it is easy to use for proprietary data files.

The first implementation was inspired by the expat streaming parser that allows one to push some data into the paring machinery and it will fire appropriate callbacks when tokes are detected. Reimplementing it as a simple pulling scanner simplified the code dramatically. With Go routines one can easily turn that into the streaming and event driven model again.

So, if you are looking for something that's even simpler than JSON or YAML you might give it a try… Happy coding!

Documentation

Overview

Package xsx provides tools for parsing so called eXtended S-eXpressions. Extended means the following things compared to https://people.csail.mit.edu/rivest/sexp.html:

Nested structures are delimited by balanced braces '()', '[]' or '{}’ – not only by '()'.

XSX provides a notation for "Meta Values", i.e. XSX that provide some sort of meta information that is not part of the "normal" data.

On the other hand some properties from SEXP were dropped, e.g. typing of the so called "octet strings". Things like that are completely left to the application.

Example

p := NewDefaultScanner(strings.NewReader(` "xyz"foo `))
var tok Token
var err error
for err = p.Next(&tok, true); err == nil; err = p.Next(&tok, true) {
	fmt.Printf("%s [%s] %t %t\n", tok.Type, tok.String(), tok.Meta, tok.Quoted)
}
fmt.Println(err)

Output:

Space [ ] false false
Atom [xyz] false true
Atom [foo] false false
Space [ ] false false
EOF

Index

• type ExpectModifier
• type Expectation
  • func Expect(t *Token) *Expectation
  • func (ex *Expectation) AnyAtom(mod ExpectModifier) bool
  • func (ex *Expectation) Atom(s string, mod ExpectModifier) bool
  • func (ex *Expectation) Begin(mod ExpectModifier, bs ...rune) bool
  • func (ex *Expectation) End() bool
  • func (ex *Expectation) Failed() bool
  • func (ex *Expectation) Fails() Fails
  • func (ex *Expectation) Reset(t *Token) *Expectation
  • func (ex *Expectation) Space() bool
  • func (ex *Expectation) Void() bool
• type Fails
  • func (err Fails) Error() string
• type Scanner
  • func NewDefaultScanner(r io.Reader) *Scanner
  • func NewScanner(r io.Reader, s Syntax) (scn *Scanner, err error)
  • func NewStringDefaultScanner(str string) *Scanner
  • func NewStringScanner(str string, s Syntax) (*Scanner, error)
  • func (scn *Scanner) MustNext(tok *Token, space bool) bool
  • func (scn *Scanner) Next(tok *Token, space bool) error
  • func (scn *Scanner) Syntax() Syntax
• type Syntax
  • func DefaultSyntax() Syntax
  • func NewSyntax(begin, end string, quote, meta rune) (Syntax, error)
  • func (s Syntax) Begin() []rune
  • func (s Syntax) End() []rune
  • func (s Syntax) IsBegin(r rune) int
  • func (s Syntax) IsEnd(r rune) int
  • func (s Syntax) IsToken(r rune) bool
  • func (s Syntax) Meta() rune
  • func (s Syntax) Quote() rune
  • func (syn Syntax) QuoteIf(s string) (string, bool)
• type Token
  • func (t *Token) Rune() rune
• type TokenType
  • func (i TokenType) String() string
• type Writer
  • func NewDefaultWriter(w io.Writer) *Writer
  • func NewWriter(w io.Writer, s Syntax) (wr *Writer, err error)
  • func (w *Writer) Atom(a string, meta bool) (quoted bool, n int, err error)
  • func (w *Writer) Begin(b int, meta bool) (n int, err error)
  • func (w *Writer) End() (int, error)
  • func (w *Writer) Flush() error
  • func (w *Writer) Space(spc string) (n int, err error)
  • func (w *Writer) Void() (n int, err error)

Examples

• Package
• Expectation
• Syntax.QuoteIf
• Writer

Types

type ExpectModifier

type ExpectModifier int

const (
	IsMeta ExpectModifier = (1 << iota)
	NotMeta
	IsQuote
	NotQuote
)

type Expectation

type Expectation struct {
	// contains filtered or unexported fields
}

Example

var tok Token
scn := NewStringDefaultScanner("\\")
if err := scn.Next(&tok, false); err != nil {
	fmt.Println(err)
	return
}
ex := *Expect(&tok)
switch {
case ex.Begin(0, '('):
	fmt.Println("begin (")
case ex.Atom("foo", IsQuote|NotMeta):
	fmt.Println("found a \"foo\"")
default:
	fmt.Println(ex.Fails())
}
fmt.Println("Expectation failed:", ex.Failed())

Output:

[token is Void, not Begin] & [token is Void, not Atom]
Expectation failed: true

func Expect

func Expect(t *Token) *Expectation

func (*Expectation) AnyAtom

func (ex *Expectation) AnyAtom(mod ExpectModifier) bool

func (*Expectation) Atom

func (ex *Expectation) Atom(s string, mod ExpectModifier) bool

func (*Expectation) Begin

func (ex *Expectation) Begin(mod ExpectModifier, bs ...rune) bool

func (*Expectation) End

func (ex *Expectation) End() bool

func (*Expectation) Failed

func (ex *Expectation) Failed() bool

func (*Expectation) Fails

func (ex *Expectation) Fails() Fails

func (*Expectation) Reset

func (ex *Expectation) Reset(t *Token) *Expectation

func (*Expectation) Space

func (ex *Expectation) Space() bool

func (*Expectation) Void

func (ex *Expectation) Void() bool

type Fails

type Fails []error

func (Fails) Error

func (err Fails) Error() string

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

func NewDefaultScanner

func NewDefaultScanner(r io.Reader) *Scanner

func NewScanner

func NewScanner(r io.Reader, s Syntax) (scn *Scanner, err error)

func NewStringDefaultScanner

func NewStringDefaultScanner(str string) *Scanner

func NewStringScanner

func NewStringScanner(str string, s Syntax) (*Scanner, error)

func (*Scanner) MustNext

func (scn *Scanner) MustNext(tok *Token, space bool) bool

func (*Scanner) Next

func (scn *Scanner) Next(tok *Token, space bool) error

func (*Scanner) Syntax

func (scn *Scanner) Syntax() Syntax

type Syntax

type Syntax struct {
	// contains filtered or unexported fields
}

func DefaultSyntax

func DefaultSyntax() Syntax

func NewSyntax

func NewSyntax(begin, end string, quote, meta rune) (Syntax, error)

func (Syntax) Begin

func (s Syntax) Begin() []rune

func (Syntax) End

func (s Syntax) End() []rune

func (Syntax) IsBegin

func (s Syntax) IsBegin(r rune) int

func (Syntax) IsEnd

func (s Syntax) IsEnd(r rune) int

func (Syntax) IsToken

func (s Syntax) IsToken(r rune) bool

func (Syntax) Meta

func (s Syntax) Meta() rune

func (Syntax) Quote

func (s Syntax) Quote() rune

func (Syntax) QuoteIf

func (syn Syntax) QuoteIf(s string) (string, bool)

Example

syn := DefaultSyntax()
fmt.Println(syn.QuoteIf(""))
fmt.Println(syn.QuoteIf("foo"))
fmt.Println(syn.QuoteIf("foo bar"))
fmt.Println(syn.QuoteIf("(foo)"))
fmt.Println(syn.QuoteIf("\\foo"))
fmt.Println(syn.QuoteIf(`a "foo"`))

Output:

"" true
foo false
"foo bar" true
"(foo)" true
"\foo" true
"a ""foo""" true

type Token

type Token struct {
	strings.Builder
	Type         TokenType
	Meta, Quoted bool
}

func (*Token) Rune

func (t *Token) Rune() rune

Returns unicode.MaxRune+1 if no rune is available

type TokenType

type TokenType int

const (
	Space TokenType = iota + 1
	Atom
	Begin
	End
	Void
)

func (TokenType) String

func (i TokenType) String() string

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Example

syn, _ := NewSyntax("<", ">", '\'', '^')
w, _ := NewWriter(os.Stdout, syn)
w.Begin(0, false)
w.Space("\n\t")
w.Atom("foo 'n' bar", true)
w.Space("\n\t")
w.Atom("4711", false)
w.Space("\n")
w.End()
w.Flush()

Output:

<
	^'foo ''n'' bar'
	4711
>

func NewDefaultWriter

func NewDefaultWriter(w io.Writer) *Writer

func NewWriter

func NewWriter(w io.Writer, s Syntax) (wr *Writer, err error)

func (*Writer) Atom

func (w *Writer) Atom(a string, meta bool) (quoted bool, n int, err error)

func (*Writer) Begin

func (w *Writer) Begin(b int, meta bool) (n int, err error)

func (*Writer) End

func (w *Writer) End() (int, error)

func (*Writer) Flush

func (w *Writer) Flush() error

func (*Writer) Space

func (w *Writer) Space(spc string) (n int, err error)

func (*Writer) Void

func (w *Writer) Void() (n int, err error)