record_reader

Documentation

Index

• Variables
• type Options
• type RecordGroup
  • func ReadAll(rr RecordReader) (RecordGroup, error)
• type RecordReader
  • func NewCsv(in io.Reader, opts *Options) RecordReader
  • func NewJson(in io.Reader, opts *Options) RecordReader

Variables

var ErrIncompleteRecord = errors.New("stream finished in the middle of a record")

Types

type Options

type Options struct {
	// The max number of records in a RecordGroup.
	// Higher numbers will use more memory.
	GroupSize int
}

type RecordGroup

type RecordGroup struct {
	// The bytes of all the records in the group.
	Bytes []byte
	// The number of records present in the bytes.
	Count int
}

RecordGroup is a complete chunk of records read from the reader. Its bytes can be safely written.

func ReadAll

func ReadAll(rr RecordReader) (RecordGroup, error)

ReadAll returns a record group that has read every record from the given reader, stopping after the first error.

Note that unlike RecordReader.Next, if an error occurs, the RecordGroup would have non-empty bytes (if there are any records in the reader stream).

type RecordReader

type RecordReader interface {
	// Next returns the bytes for the next chunk of records,
	// or an error. error will never be non-nil
	// if the group bytes are > 0.
	Next() (RecordGroup, error)
}

func NewCsv

func NewCsv(in io.Reader, opts *Options) RecordReader

NewCsv returns a record reader that reports each CSV row as a record. Note that CSV rows are not just delimited via '\n'- there is in fact no way to know whether a character is inline a record without having parsed the file (consider that newlines can be within cells, and quotes are escaped via doubling-up, so even something like `","` may refer to the string `","` within a cell, or `"",""` in the CSV).

func NewJson

func NewJson(in io.Reader, opts *Options) RecordReader

NewJson returns a record reader that parses Apex-flavor JSON and converts it into bytes of JSONLines format in its records.

Note this is not a generic JSON parser- we cannot parse the JSON both because it'd be slower, but especially because it's not really possible to parse JSON like `[{}, {}]` incrementally. So we read lines and figure out when we are between objects.

Becuase of this, only Apex-flavor pretty JSON is supported. Like:

[ {
     "x": 1
}, {

     "x: 2
} ]