flatfile

README

flatfile

This package provides a utility to read a fixed-width record from a flat-file database or a text file. The intent is to eliminate boilerplate code for reading data from a flat file and mapping it to the fields in a struct.

This package allows you to define your record layout mapping using struct tags.

This packge provides a method Unmarshal which will convert a record (slice of bytes) into a struct.

Usage

Download

Use your favourite dependency tool to pull the code. dep, go mod, etc.

go get github.com/ahmedalhulaibi/flatfile

Examples

Please refer to the examples folder in repo for all examples.

Let's say I have a text file customers.txt which contains customer data. There are 2 records:

AMY1900-01-01019123 FAKE STREETCA
BOB1800-01-01037456 OLD STREET US

The file layout is:

Columns  1, Length  3   = Customer Name
Columns  4, Length 10   = Customer Join/Open Date
Columns 14, Length  3   = Customer Age
Columns 17, Length 15   = Customer Street Address
Columns 32, Length  2   = Customer Country

Each field in a struct can be mapped to a single field in a record using a struct tag.

Struct tags are in the form flatfile:"col,len" or for a slice field flatfile:"col,len,occurrences".

I can directly translate these to fields in a struct:

type CustomerRecord struct {
    //flatfile is one indexed, column starts at 1
    Name        string `flatfile:"1,3"`
    OpenDate    string `flatfile:"4,10"`
    Age         uint   `flatfile:"14,3"`
    Address     string `flatfile:"17,15"`
    CountryCode string `flatfile:"32,2"`
}

Once your layout is defined we can read a line from a file and unmarshal the data:

type CustomerRecord struct {
	//flatfile is one indexed, column starts at 1
	Name        string `flatfile:"1,3"`
	OpenDate    string `flatfile:"4,10"`
	Age         uint   `flatfile:"14,3"`
	Address     string `flatfile:"17,15"`
	CountryCode string `flatfile:"32,2"`
}

func main() {
	file, err := os.Open("customers.txt")
	checkError(err)
	defer file.Close()

	reader := bufio.NewReader(file)

	eof := false
	for !eof {
		data := readLine(reader)
		if data == nil {
			eof = true
		} else {
			fileRecord := &CustomerRecord{}
            //unmarhsal text data to struct
			err := flatfile.Unmarshal(data, fileRecord, 0, 0)
			fmt.Printf("%v\n", fileRecord)
			checkError(err)
		}
	}
	checkError(err)
}

func readLine(reader *bufio.Reader) []byte {
	str, _, err := reader.ReadLine()
	if err == io.EOF {
		return nil
	}

	return str
}

func checkError(err error) {
	if err != nil {
		panic(err)
	}
}

Features

Data type support:

• bool

• string

• int

• int8

• int16

• int32

• int64

• uint

• uint8

• uint16

• uint32

• uint64

• float32

• float64

• rune

• byte

• Slice

• Array

• Nested struct

• Nested pointer (to any support type including struct)

• Slice, Array support AKA Emulate COBOL occurs clause

• Offset feature to support reading long lines of data. Example

• Byte and Rune support using type override.

  These are aliases for uint8 and int32 respectively. These require an override option to be supplied.

• Flat File abstraction

• Support for conditional unmarshal

  if field(col,len) == "text" do unmarshal else skip. This is done by supplying the condition option. The syntax is a bit funky:

  col-len-VALUE e.g. 1-10-TENLETTERS This means the unmarshal will only occur if the data from column 1, length 10 is equals to TENLETTERS.

  This is useful for flat files where there are multiple record layouts within the same file.

Documentation

Index

• func CalcNumFieldsToUnmarshal(data []byte, v interface{}, fieldOffset int) (int, []byte, error)
• func Examine(v interface{})
• func ShouldUnmarshal(ffpTag *flatfileTag, data []byte) bool
• func Unmarshal(data []byte, v interface{}, startFieldIdx int, numFieldsToUnmarshal int, ...) error
• type FlatFile
  • func New(reader *bufio.Reader, objectLayout interface{}) (*FlatFile, error)
  • func (f *FlatFile) Read() (err error)

Functions

func CalcNumFieldsToUnmarshal

func CalcNumFieldsToUnmarshal(data []byte, v interface{}, fieldOffset int) (int, []byte, error)

CalcNumFieldsToUnmarshal determines how many fields can be unmarshalled successfully This currently will not return an accurate result for overlapping fields For example:

type Profile struct {
		FirstName string `flatfile:"1,10"`
		LastName  string `flatfile:"11,10"`
		FullName  string `flatfile:"1,20"`
}

Expected output is that 3 fields can be unmarshalled successfully The result will be 2 Another example: type Profile struct

type Profile struct {
		FirstName string `flatfile:"1,10"`
		LastName  string `flatfile:"11,10"`
		FullName  string `flatfile:"1,20"`
		Random    string `flatfile:"7,9"`
}

This function would have to be redesigned to handle multiple scenarios of overlapping fields

func Examine

func Examine(v interface{})

Examine traverses all elements of a type and uses the reflect pkg to print type and kind

func ShouldUnmarshal

func ShouldUnmarshal(ffpTag *flatfileTag, data []byte) bool

ShouldUnmarshal returns true if the condition

func Unmarshal

func Unmarshal(data []byte, v interface{}, startFieldIdx int, numFieldsToUnmarshal int, isPartialUnmarshal bool) error

Unmarshal will read data and convert it into a struct based on a schema/map defined by struct tags

Struct tags are in the form `flatfile:"col,len"`. col and len should be integers > 0

startFieldIdx: index can be passed to indicate which struct field to start the unmarshal. Zero indexed.

numFieldsToUnmarshal: can be passed to indicate how many fields to unmarshal starting from startFieldIdx

isPartialUnmarshal: flag should be set to true if the data[0] is intended to be unmarshalled at startFieldIdx

If startFieldIdx == 0 and umFieldsToMarshal == 0 then Unmarshal will attempt to unmarshal all fields with an ffp tag

Types

type FlatFile

type FlatFile struct {
	// contains filtered or unexported fields
}

FlatFile is an abstraction for a flat file containing structured data

func New

func New(reader *bufio.Reader, objectLayout interface{}) (*FlatFile, error)

New returns a new FlatFile reader object

func (*FlatFile) Read

func (f *FlatFile) Read() (err error)

Read will read a line from a bufio.Reader and call flatfile.Unmarshal to convert the read in data into FlatFile.objectLayout