csv

README

What is csv package?

The csv package is a library for performing validation when reading CSV or TSV files. Validation rules are specified using struct tags. The csv package read returns which columns of which rows do not adhere to the specified rules.

We are implementing internationalization (i18n) for error messages.

Supported languages

• English
• Japanese
• Russian

If you want to add a new language, please create a pull request. Ref. https://github.com/nao1215/csv/pull/8

Why need csv package?

I was frustrated with error-filled CSV files written by non-engineers.

I encountered a use case of "importing one CSV file into multiple DB tables". Unfortunately, I couldn't directly import the CSV file into the DB tables. So, I attempted to import the CSV file through a Go-based application.

What frustrated me was not knowing where the errors in the CSV file were. Existing libraries didn't provide output like "The value in the Mth column of the Nth row is incorrect". I attempted to import multiple times and processed error messages one by one. Eventually, I started writing code to parse each column, which wasted a considerable amount of time.

Based on the above experience, I decided to create a generic CSV validation tool.

How to use

Please attach the "validate:" tag to your structure and write the validation rules after it. It's crucial that the "order of columns" matches the "order of field definitions" in the structure. The csv package does not automatically adjust the order.

When using csv.Decode, please pass a pointer to a slice of structures tagged with struct tags. The csv package will perform validation based on the struct tags and save the read results to the slice of structures if there are no errors. If there are errors, it will return them as []error.

Example: english error message

package main

import (
	"bytes"
	"fmt"

	"github.com/nao1215/csv"
)

func main() {
	input := `id,name,age
1,Gina,23
a,Yulia,25
3,Den1s,30
`
	buf := bytes.NewBufferString(input)
	c, err := csv.NewCSV(buf)
	if err != nil {
		panic(err)
	}

	type person struct {
		ID   int    `validate:"numeric"`
		Name string `validate:"alpha"`
		Age  int    `validate:"gt=24"`
	}
	people := make([]person, 0)

	errs := c.Decode(&people)
	if len(errs) != 0 {
		for _, err := range errs {
			fmt.Println(err.Error())
		}
	}

	// Output:
	// line:2 column age: target is not greater than the threshold value: threshold=24, value=23
	// line:3 column id: target is not a numeric character: value=a
	// line:4 column name: target is not an alphabetic character: value=Den1s
}

Example: japanese error message

func main() {
	input := `id,name,age
1,Gina,23
a,Yulia,25
3,Den1s,30
`
	buf := bytes.NewBufferString(input)
	c, err := csv.NewCSV(buf, csv.WithJapaneseLanguage()) // Set Japanese language option
	if err != nil {
		panic(err)
	}

	type person struct {
		ID   int    `validate:"numeric"`
		Name string `validate:"alpha"`
		Age  int    `validate:"gt=24"`
	}
	people := make([]person, 0)

	errs := c.Decode(&people)
	if len(errs) != 0 {
		for _, err := range errs {
			fmt.Println(err.Error())
		}
	}

	// Output:
	// line:2 column age: 値がしきい値より大きくありません: threshold=24, value=23
	// line:3 column id: 値が数字ではありません: value=a
	// line:4 column name: 値がアルファベット文字ではありません: value=Den1s
}

Struct tags

You set the validation rules following the "validate:" tag according to the rules in the table below. If you need to set multiple rules, please enumerate them separated by commas.

Strings

Tag Name	Description
alpha	Check whether value is alphabetic or not
alphanumeric	Check whether value is alphanumeric or not
ascii	Check whether value is ASCII or not
boolean	Check whether value is boolean or not.
contains	Check whether value contains the specified substring e.g. validate:"contains=abc"
containsany	Check whether value contains any of the specified characters e.g. validate:"containsany=abc def"
lowercase	Check whether value is lowercase or not
numeric	Check whether value is numeric or not
uppercase	Check whether value is uppercase or not

Format

Tag Name	Description
email	Check whether value is an email address or not

Comparisons

Tag Name	Description
eq	Check whether value is equal to the specified value. e.g. validate:"eq=1"
gt	Check whether value is greater than the specified value e.g. validate:"gt=1"
gte	Check whether value is greater than or equal to the specified value e.g. validate:"gte=1"
lt	Check whether value is less than the specified value e.g. validate:"lt=1"
lte	Check whether value is less than or equal to the specified value e.g. validate:"lte=1"
ne	Check whether value is not equal to the specified value e.g. validate:"ne=1"

Other

Tag Name	Description
len	Check whether the length of the value is equal to the specified value e.g. validate:"len=10"
max	Check whether value is less than or equal to the specified value e.g. validate:"max=100"
min	Check whether value is greater than or equal to the specified value e.g. validate:"min=1"
oneof	Check whether value is included in the specified values e.g. validate:"oneof=male female prefer_not_to"
required	Check whether value is empty or not

License

MIT License

Contribution

First off, thanks for taking the time to contribute! See CONTRIBUTING.md for more information. Contributions are not only related to development. For example, GitHub Star motivates me to develop! Please feel free to contribute to this project.

Special Thanks

I was inspired by the following OSS. Thank you for your great work!

• go-playground/validator
• shogo82148/go-header-csv

Contributors ✨

Thanks goes to these wonderful people (emoji key):

CHIKAMATSU Naohiro 📖

Add your contributions

This project follows the all-contributors specification. Contributions of any kind welcome!

Documentation

Overview

Package csv returns which columns have syntax errors on a per-line basis when reading CSV. It also has the capability to convert the character encoding to UTF-8 if the CSV character encoding is not UTF-8.

Index

• Variables
• type CSV
  • func NewCSV(r io.Reader, opts ...Option) (*CSV, error)
  • func (c *CSV) Decode(structSlicePointer any) []error
• type Error
  • func NewError(localizer *i18n.Localizer, id, subMessage string) *Error
  • func (e *Error) Error() string
  • func (e *Error) Is(target error) bool
• type Option
  • func WithHeaderless() Option
  • func WithJapaneseLanguage() Option
  • func WithRussianLanguage() Option
  • func WithTabDelimiter() Option

Examples

• CSV
• WithJapaneseLanguage
• WithRussianLanguage

Variables

var (
	// ErrStructSlicePointerID is the error ID used when the value is not a pointer to a struct slice.
	ErrStructSlicePointerID = "ErrStructSlicePointer"
	// ErrInvalidOneOfFormatID is the error ID used when the target is not one of the specified values.
	ErrInvalidOneOfFormatID = "ErrInvalidOneOfFormat"
	// ErrInvalidThresholdFormatID is the error ID used when the threshold format is invalid.
	ErrInvalidThresholdFormatID = "ErrInvalidThresholdFormat"
	// ErrInvalidBooleanID is the error ID used when the target is not a boolean.
	ErrInvalidBooleanID = "ErrInvalidBoolean"
	// ErrInvalidAlphabetID is the error ID used when the target is not an alphabetic character.
	ErrInvalidAlphabetID = "ErrInvalidAlphabet"
	// ErrInvalidNumericID is the error ID used when the target is not a numeric character.
	ErrInvalidNumericID = "ErrInvalidNumeric"
	// ErrInvalidAlphanumericID is the error ID used when the target is not an alphanumeric character.
	ErrInvalidAlphanumericID = "ErrInvalidAlphanumeric"
	// ErrRequiredID is the error ID used when the target is required but is empty.
	ErrRequiredID = "ErrRequired"
	// ErrEqualID is the error ID used when the target is not equal to the threshold value.
	ErrEqualID = "ErrEqual"
	// ErrInvalidThresholdID is the error ID used when the threshold value is invalid.
	ErrInvalidThresholdID = "ErrInvalidThreshold"
	// ErrNotEqualID is the error ID used when the target is equal to the threshold value.
	ErrNotEqualID = "ErrNotEqual"
	// ErrGreaterThanID is the error ID used when the target is not greater than the threshold value.
	ErrGreaterThanID = "ErrGreaterThan"
	// ErrGreaterThanEqualID is the error ID used when the target is not greater than or equal to the threshold value.
	ErrGreaterThanEqualID = "ErrGreaterThanEqual"
	// ErrLessThanID is the error ID used when the target is not less than the threshold value.
	ErrLessThanID = "ErrLessThan"
	// ErrLessThanEqualID is the error ID used when the target is not less than or equal to the threshold value.
	ErrLessThanEqualID = "ErrLessThanEqual"
	// ErrMinID is the error ID used when the target is less than the minimum value.
	ErrMinID = "ErrMin"
	// ErrMaxID is the error ID used when the target is greater than the maximum value.
	ErrMaxID = "ErrMax"
	// ErrLengthID is the error ID used when the target length is not equal to the threshold value.
	ErrLengthID = "ErrLength"
	// ErrOneOfID is the error ID used when the target is not one of the specified values.
	ErrOneOfID = "ErrOneOf"
	// ErrInvalidStructID is the error ID used when the target is not a struct.
	ErrInvalidStructID = "ErrInvalidStruct"
	// ErrUnsupportedTypeID is the error ID used when the target is an unsupported type.
	ErrUnsupportedTypeID = "ErrUnsupportedType"
	// ErrLowercaseID is the error ID used when the target is not a lowercase character.
	ErrLowercaseID = "ErrLowercase"
	// ErrUppercaseID is the error ID used when the target is not an uppercase character.
	ErrUppercaseID = "ErrUppercase"
	// ErrASCIIID is the error ID used when the target is not an ASCII character.
	ErrASCIIID = "ErrASCII"
	// ErrEmailID is the error ID used when the target is not an email.
	ErrEmailID = "ErrEmail"
	// ErrContainsID is the error ID used when the target does not contain the specified value.
	ErrContainsID = "ErrContains"
	// ErrInvalidContainsFormatID is the error ID used when the contains format is invalid.
	ErrInvalidContainsFormatID = "ErrInvalidContainsFormat"
	// ErrContainsAnyID is the error ID used when the target does not contain any of the specified values.
	ErrContainsAnyID = "ErrContainsAny"
	// ErrInvalidContainsAnyFormatID is the error ID used when the contains any format is invalid.
	ErrInvalidContainsAnyFormatID = "ErrInvalidContainsAnyFormat"
)

var LocaleFS embed.FS

Types

type CSV

type CSV struct {
	// contains filtered or unexported fields
}

CSV is a struct that implements CSV Reader and Writer.

Example

package main

import (
	"bytes"
	"fmt"

	"github.com/nao1215/csv"
)

func main() {
	input := `id,name,age
1,Gina,23
a,Yulia,25
3,Den1s,30
`
	buf := bytes.NewBufferString(input)
	c, err := csv.NewCSV(buf)
	if err != nil {
		panic(err)
	}

	type person struct {
		ID   int    `validate:"numeric"`
		Name string `validate:"alpha"`
		Age  int    `validate:"gt=24"`
	}
	people := make([]person, 0)

	errs := c.Decode(&people)
	if len(errs) != 0 {
		for _, err := range errs {
			fmt.Println(err.Error())
		}
	}

}

Output:

line:2 column age: target is not greater than the threshold value: threshold=24, value=23
line:3 column id: target is not a numeric character: value=a
line:4 column name: target is not an alphabetic character: value=Den1s

func NewCSV

func NewCSV(r io.Reader, opts ...Option) (*CSV, error)

NewCSV returns a new CSV struct.

func (*CSV) Decode

func (c *CSV) Decode(structSlicePointer any) []error

Decode reads the CSV and returns the columns that have syntax errors on a per-line basis. The strutSlicePointer is a pointer to structure slice where validation rules are set in struct tags.

type Error

type Error struct {
	// contains filtered or unexported fields
}

Error is an error that is used to localize error messages.

func NewError

func NewError(localizer *i18n.Localizer, id, subMessage string) *Error

NewError returns a new Error.

func (*Error) Error

func (e *Error) Error() string

Error returns the localized error message.

func (*Error) Is

func (e *Error) Is(target error) bool

Is reports whether the target error is the same as the error.

type Option

type Option func(c *CSV) error

Option is a function that sets a configuration option for CSV struct.

func WithHeaderless

func WithHeaderless() Option

WithHeaderless is an Option that sets the headerless flag to true.

func WithJapaneseLanguage

func WithJapaneseLanguage() Option

WithJapaneseLanguage is an Option that sets the i18n bundle to Japanese.

Example

package main

import (
	"bytes"
	"fmt"

	"github.com/nao1215/csv"
)

func main() {
	input := `id,name,age
1,Gina,23
a,Yulia,25
3,Den1s,30
`
	buf := bytes.NewBufferString(input)
	c, err := csv.NewCSV(buf, csv.WithJapaneseLanguage())
	if err != nil {
		panic(err)
	}

	type person struct {
		ID   int    `validate:"numeric"`
		Name string `validate:"alpha"`
		Age  int    `validate:"gt=24"`
	}
	people := make([]person, 0)

	errs := c.Decode(&people)
	if len(errs) != 0 {
		for _, err := range errs {
			fmt.Println(err.Error())
		}
	}

}

Output:

line:2 column age: 値がしきい値より大きくありません: threshold=24, value=23
line:3 column id: 値が数字ではありません: value=a
line:4 column name: 値がアルファベット文字ではありません: value=Den1s

func WithRussianLanguage

func WithRussianLanguage() Option

WithRussianLanguage is an Option that sets the i18n bundle to Russian.

Example

package main

import (
	"bytes"
	"fmt"

	"github.com/nao1215/csv"
)

func main() {
	input := `id,name,age
1,Gina,23
a,Yulia,25
3,Den1s,30
`
	buf := bytes.NewBufferString(input)
	c, err := csv.NewCSV(buf, csv.WithRussianLanguage())
	if err != nil {
		panic(err)
	}

	type person struct {
		ID   int    `validate:"numeric"`
		Name string `validate:"alpha"`
		Age  int    `validate:"gt=24"`
	}
	people := make([]person, 0)

	errs := c.Decode(&people)
	if len(errs) != 0 {
		for _, err := range errs {
			fmt.Println(err.Error())
		}
	}

}

Output:

line:2 column age: целевое значение не больше порогового значения: threshold=24, value=23
line:3 column id: целевое значение не является числовым символом: value=a
line:4 column name: целевое значение не является алфавитным символом: value=Den1s

func WithTabDelimiter

func WithTabDelimiter() Option

WithTabDelimiter is an Option that sets the delimiter to a tab character.