dsv

Documentation

Overview

Package dsv parses Delimiter-separated DataSources

Index

• type Parser
  • func CreateParser(conf *ParserConf) *Parser
  • func (p *Parser) Parse(r io.Reader, source sif.DataSource, schema sif.Schema, onIteratorEnd func()) (sif.PartitionIterator, error)
  • func (p *Parser) PartitionSize() int
• type ParserConf

Types

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Parser produces partitions from DSV data

func CreateParser

func CreateParser(conf *ParserConf) *Parser

CreateParser returns a new DSV Parser

func (*Parser) Parse

func (p *Parser) Parse(r io.Reader, source sif.DataSource, schema sif.Schema, onIteratorEnd func()) (sif.PartitionIterator, error)

Parse parses DSV data to produce Partitions

func (*Parser) PartitionSize

func (p *Parser) PartitionSize() int

PartitionSize returns the maximum size in rows of Partitions produced by this Parser

type ParserConf

type ParserConf struct {
	PartitionSize int    // The maximum number of rows per Partition. Defaults to 128.
	HeaderLines   int    // The number of lines to ignore from the beginning of each file. Defaults to 0.
	Delimiter     rune   // The delimiter separating columns in the file. Defaults to ,
	Comment       rune   // Lines beginning with the comment character are ignored. Cannot be equal to the Delimiter. Defaults to no comment character.
	NilValue      string // A special string which represents nil values in the dataset. Defaults to "" (the empty string).
}

ParserConf configures a DSV Parser