content

Documentation

Index

• Constants
• type ContentReader
  • func MergeReaders(arr []*ContentReader, arrayKey string) (*ContentReader, error)
  • func NewContentReader(filePath string, arrayKey string) *ContentReader
  • func NewEmptyContentReader(arrayKey string) *ContentReader
  • func (cr *ContentReader) Close() error
  • func (cr *ContentReader) GetError() error
  • func (cr *ContentReader) GetFilePath() string
  • func (cr *ContentReader) IsEmpty() bool
  • func (cr *ContentReader) Length() (int, error)
  • func (cr *ContentReader) NextRecord(recordOutput interface{}) error
  • func (cr *ContentReader) Reset()
• type ContentWriter
  • func NewContentWriter(arrayKey string, isCompleteFile, useStdout bool) (*ContentWriter, error)
  • func (rw *ContentWriter) Close() error
  • func (rw *ContentWriter) GetArrayKey() string
  • func (rw *ContentWriter) GetError() error
  • func (rw *ContentWriter) GetFilePath() string
  • func (rw *ContentWriter) IsEmpty() bool
  • func (rw *ContentWriter) RemoveOutputFilePath() error
  • func (rw *ContentWriter) SetArrayKey(arrKey string) *ContentWriter
  • func (rw *ContentWriter) Write(record interface{})

Constants

const (
	DefaultKey = "results"
)

Types

type ContentReader

type ContentReader struct {
	// contains filtered or unexported fields
}

Open and read JSON file, find the array key inside it and load its value into the memory in small chunks. Currently, 'ContentReader' only support extracting a single value for a given key (arrayKey), other keys are ignored. The value must of of type array. Each array value can be fetched using 'GetRecord' (thread-safe). This technique solves the limit of memory size which may be too small to fit large JSON.

func MergeReaders

func MergeReaders(arr []*ContentReader, arrayKey string) (*ContentReader, error)

func NewContentReader

func NewContentReader(filePath string, arrayKey string) *ContentReader

func NewEmptyContentReader

func NewEmptyContentReader(arrayKey string) *ContentReader

func (*ContentReader) Close

func (cr *ContentReader) Close() error

Cleanup the reader data.

func (*ContentReader) GetError

func (cr *ContentReader) GetError() error

func (*ContentReader) GetFilePath

func (cr *ContentReader) GetFilePath() string

func (*ContentReader) IsEmpty

func (cr *ContentReader) IsEmpty() bool

func (*ContentReader) Length

func (cr *ContentReader) Length() (int, error)

Number of element in the array.

func (*ContentReader) NextRecord

func (cr *ContentReader) NextRecord(recordOutput interface{}) error

Each call to 'NextRecord()' will returns a single element from the channel. Only the first call invokes a goroutine to read data from the file and push it into the channel. 'io.EOF' will be returned if no data is left.

func (*ContentReader) Reset

func (cr *ContentReader) Reset()

Prepare the reader to read the file all over again (not thread-safe).

type ContentWriter

type ContentWriter struct {
	// contains filtered or unexported fields
}

Write a JSON file in small chunks. Only a single JSON key can be written to the file, and array as its value. The array's values could be any JSON value types (number, string, etc...). Once the first 'Write" call is made, the file will stay open, waiting for the next struct to be written (thread-safe). Finally, 'Close' will fill the end of the JSON file and the operation will be completed.

func NewContentWriter

func NewContentWriter(arrayKey string, isCompleteFile, useStdout bool) (*ContentWriter, error)

func (*ContentWriter) Close

func (rw *ContentWriter) Close() error

Finish writing to the file.

func (*ContentWriter) GetArrayKey

func (rw *ContentWriter) GetArrayKey() string

func (*ContentWriter) GetError

func (rw *ContentWriter) GetError() error

func (*ContentWriter) GetFilePath

func (rw *ContentWriter) GetFilePath() string

func (*ContentWriter) IsEmpty

func (rw *ContentWriter) IsEmpty() bool

func (*ContentWriter) RemoveOutputFilePath

func (rw *ContentWriter) RemoveOutputFilePath() error

func (*ContentWriter) SetArrayKey

func (rw *ContentWriter) SetArrayKey(arrKey string) *ContentWriter

func (*ContentWriter) Write

func (rw *ContentWriter) Write(record interface{})

Write a single item to the JSON array.