content

Documentation

Index

• Constants
• func ConvertToStruct(record, recordOutput interface{}) error
• type ContentReader
  • func MergeReaders(arr []*ContentReader, arrayKey string) (contentReader *ContentReader, err error)
  • func MergeSortedReaders(readerRecord SortableContentItem, sortedReaders []*ContentReader, ...) (contentReader *ContentReader, err error)
  • func NewContentReader(filePath string, arrayKey string) *ContentReader
  • func NewEmptyContentReader(arrayKey string) *ContentReader
  • func NewMultiSourceContentReader(filePaths []string, arrayKey string) *ContentReader
  • func SortAndSaveBufferToFile(keysToContentItems map[string]SortableContentItem, allKeys []string, ...) (contentReader *ContentReader, err error)
  • func SortContentReader(readerRecord SortableContentItem, reader *ContentReader, ascendingOrder bool) (*ContentReader, error)
  • func SortContentReaderByCalculatedKey(reader *ContentReader, getKeyFunc keyCalculationFunc, ascendingOrder bool) (contentReader *ContentReader, err error)
  • func (cr *ContentReader) Close() error
  • func (cr *ContentReader) GetError() error
  • func (cr *ContentReader) GetFilesPaths() []string
  • func (cr *ContentReader) IsEmpty() bool
  • func (cr *ContentReader) Length() (int, error)
  • func (cr *ContentReader) NextRecord(recordOutput interface{}) error
  • func (cr *ContentReader) Reset()
• type ContentWriter
  • func NewContentWriter(arrayKey string, isCompleteFile, useStdout bool) (*ContentWriter, error)
  • func (rw *ContentWriter) Close() error
  • func (rw *ContentWriter) GetArrayKey() string
  • func (rw *ContentWriter) GetError() error
  • func (rw *ContentWriter) GetFilePath() string
  • func (rw *ContentWriter) IsEmpty() bool
  • func (rw *ContentWriter) RemoveOutputFilePath() error
  • func (rw *ContentWriter) SetArrayKey(arrKey string) *ContentWriter
  • func (rw *ContentWriter) Write(record interface{})
• type SortRecord
  • func (sr SortRecord) GetSortKey() string
• type SortableContentItem

Constants

const (
	DefaultKey = "results"
)

Functions

func ConvertToStruct

func ConvertToStruct(record, recordOutput interface{}) error

Types

type ContentReader

type ContentReader struct {
	// contains filtered or unexported fields
}

Open and read JSON files, find the array key inside it and load its value into the memory in small chunks. Currently, 'ContentReader' only support extracting a single value for a given key (arrayKey), other keys are ignored. The value must be of type array. Each array value can be fetched using 'NextRecord' (thread-safe). This technique solves the limit of memory size which may be too small to fit large JSON.

func MergeReaders

func MergeReaders(arr []*ContentReader, arrayKey string) (contentReader *ContentReader, err error)

func MergeSortedReaders

func MergeSortedReaders(readerRecord SortableContentItem, sortedReaders []*ContentReader, ascendingOrder bool) (contentReader *ContentReader, err error)

Merge a slice of sorted content-readers into a single sorted content-reader.

func NewContentReader

func NewContentReader(filePath string, arrayKey string) *ContentReader

func NewEmptyContentReader

func NewEmptyContentReader(arrayKey string) *ContentReader

func NewMultiSourceContentReader

func NewMultiSourceContentReader(filePaths []string, arrayKey string) *ContentReader

func SortAndSaveBufferToFile

func SortAndSaveBufferToFile(keysToContentItems map[string]SortableContentItem, allKeys []string, increasingOrder bool) (contentReader *ContentReader, err error)

func SortContentReader

func SortContentReader(readerRecord SortableContentItem, reader *ContentReader, ascendingOrder bool) (*ContentReader, error)

Sort a content-reader in the required order (ascending or descending). Performs a merge-sort on the reader, splitting the reader to multiple readers of size 'utils.MaxBufferSize'. Sort each of the split readers, and merge them into a single sorted reader. In case of multiple items with the same key - all the items will appear in the sorted reader, but their order is not guaranteed to be preserved.

func SortContentReaderByCalculatedKey

func SortContentReaderByCalculatedKey(reader *ContentReader, getKeyFunc keyCalculationFunc, ascendingOrder bool) (contentReader *ContentReader, err error)

Sort a ContentReader, according to a key generated by getKeyFunc. getKeyFunc gets an item from the reader and returns the key of the item. Attention! In case of multiple items with the same key - only the first item in the original reader will appear in the sorted one! The other items will be removed. Also pay attention that the order of the fields inside the objects might change.

func (*ContentReader) Close

func (cr *ContentReader) Close() error

Cleanup the reader data.

func (*ContentReader) GetError

func (cr *ContentReader) GetError() error

func (*ContentReader) GetFilesPaths

func (cr *ContentReader) GetFilesPaths() []string

func (*ContentReader) IsEmpty

func (cr *ContentReader) IsEmpty() bool

func (*ContentReader) Length

func (cr *ContentReader) Length() (int, error)

Number of element in the array.

func (*ContentReader) NextRecord

func (cr *ContentReader) NextRecord(recordOutput interface{}) error

Each call to 'NextRecord()' will return a single element from the channel. Only the first call invokes a goroutine to read data from the file and push it into the channel. 'io.EOF' will be returned if no data is left.

func (*ContentReader) Reset

func (cr *ContentReader) Reset()

Prepare the reader to read the file all over again (not thread-safe).

type ContentWriter

type ContentWriter struct {
	// contains filtered or unexported fields
}

Write a JSON file in small chunks. Only a single JSON key can be written to the file, and array as its value. The array's values could be any JSON value types (number, string, etc...). Once the first 'Write" call is made, the file will stay open, waiting for the next struct to be written (thread-safe). Finally, 'Close' will fill the end of the JSON file and the operation will be completed.

func NewContentWriter

func NewContentWriter(arrayKey string, isCompleteFile, useStdout bool) (*ContentWriter, error)

func (*ContentWriter) Close

func (rw *ContentWriter) Close() error

Finish writing to the file.

func (*ContentWriter) GetArrayKey

func (rw *ContentWriter) GetArrayKey() string

func (*ContentWriter) GetError

func (rw *ContentWriter) GetError() error

func (*ContentWriter) GetFilePath

func (rw *ContentWriter) GetFilePath() string

func (*ContentWriter) IsEmpty

func (rw *ContentWriter) IsEmpty() bool

func (*ContentWriter) RemoveOutputFilePath

func (rw *ContentWriter) RemoveOutputFilePath() error

func (*ContentWriter) SetArrayKey

func (rw *ContentWriter) SetArrayKey(arrKey string) *ContentWriter

func (*ContentWriter) Write

func (rw *ContentWriter) Write(record interface{})

Write a single item to the JSON array.

type SortRecord

type SortRecord struct {
	Key    string      `json:"key,omitempty"`
	Record interface{} `json:"record,omitempty"`
}

func (SortRecord) GetSortKey

func (sr SortRecord) GetSortKey() string

type SortableContentItem

type SortableContentItem interface {
	GetSortKey() string
}