content

Documentation

Index

• Constants
• type ContentReader
  • func MergeReaders(arr []*ContentReader, arrayKey string) (*ContentReader, error)
  • func MergeSortedReaders(readerRecord SortableContentItem, sortedReaders []*ContentReader, ...) (*ContentReader, error)
  • func NewContentReader(filePath string, arrayKey string) *ContentReader
  • func NewEmptyContentReader(arrayKey string) *ContentReader
  • func NewMultiSourceContentReader(filePaths []string, arrayKey string) *ContentReader
  • func SortAndSaveBufferToFile(keysToContentItems map[string]SortableContentItem, allKeys []string, ...) (*ContentReader, error)
  • func SortContentReader(readerRecord SortableContentItem, reader *ContentReader, ascendingOrder bool) (*ContentReader, error)
  • func (cr *ContentReader) Close() error
  • func (cr *ContentReader) GetError() error
  • func (cr *ContentReader) GetFilesPaths() []string
  • func (cr *ContentReader) IsEmpty() bool
  • func (cr *ContentReader) Length() (int, error)
  • func (cr *ContentReader) NextRecord(recordOutput interface{}) error
  • func (cr *ContentReader) Reset()
• type ContentWriter
  • func NewContentWriter(arrayKey string, isCompleteFile, useStdout bool) (*ContentWriter, error)
  • func (rw *ContentWriter) Close() error
  • func (rw *ContentWriter) GetArrayKey() string
  • func (rw *ContentWriter) GetError() error
  • func (rw *ContentWriter) GetFilePath() string
  • func (rw *ContentWriter) IsEmpty() bool
  • func (rw *ContentWriter) RemoveOutputFilePath() error
  • func (rw *ContentWriter) SetArrayKey(arrKey string) *ContentWriter
  • func (rw *ContentWriter) Write(record interface{})
• type SortableContentItem

Constants

const (
	DefaultKey = "results"
)

Types

type ContentReader

type ContentReader struct {
	// contains filtered or unexported fields
}

Open and read JSON files, find the array key inside it and load its value into the memory in small chunks. Currently, 'ContentReader' only support extracting a single value for a given key (arrayKey), other keys are ignored. The value must be of type array. Each array value can be fetched using 'NextRecord' (thread-safe). This technique solves the limit of memory size which may be too small to fit large JSON.

func MergeReaders

func MergeReaders(arr []*ContentReader, arrayKey string) (*ContentReader, error)

func MergeSortedReaders

func MergeSortedReaders(readerRecord SortableContentItem, sortedReaders []*ContentReader, ascendingOrder bool) (*ContentReader, error)

Merge a slice of sorted content-readers into a single sorted content-reader.

func NewContentReader

func NewContentReader(filePath string, arrayKey string) *ContentReader

func NewEmptyContentReader

func NewEmptyContentReader(arrayKey string) *ContentReader

func NewMultiSourceContentReader

func NewMultiSourceContentReader(filePaths []string, arrayKey string) *ContentReader

func SortAndSaveBufferToFile

func SortAndSaveBufferToFile(keysToContentItems map[string]SortableContentItem, allKeys []string, increasingOrder bool) (*ContentReader, error)

func SortContentReader

func SortContentReader(readerRecord SortableContentItem, reader *ContentReader, ascendingOrder bool) (*ContentReader, error)

Sort a content-reader in the required order (ascending or descending). Performs a merge-sort on the reader, splitting the reader to multiple readers of size 'utils.MaxBufferSize'. Sort each of the split readers, and merge them into a single sorted reader.

func (*ContentReader) Close

func (cr *ContentReader) Close() error

Cleanup the reader data.

func (*ContentReader) GetError

func (cr *ContentReader) GetError() error

func (*ContentReader) GetFilesPaths

func (cr *ContentReader) GetFilesPaths() []string

func (*ContentReader) IsEmpty

func (cr *ContentReader) IsEmpty() bool

func (*ContentReader) Length

func (cr *ContentReader) Length() (int, error)

Number of element in the array.

func (*ContentReader) NextRecord

func (cr *ContentReader) NextRecord(recordOutput interface{}) error

Each call to 'NextRecord()' will returns a single element from the channel. Only the first call invokes a goroutine to read data from the file and push it into the channel. 'io.EOF' will be returned if no data is left.

func (*ContentReader) Reset

func (cr *ContentReader) Reset()

Prepare the reader to read the file all over again (not thread-safe).

type ContentWriter

type ContentWriter struct {
	// contains filtered or unexported fields
}

Write a JSON file in small chunks. Only a single JSON key can be written to the file, and array as its value. The array's values could be any JSON value types (number, string, etc...). Once the first 'Write" call is made, the file will stay open, waiting for the next struct to be written (thread-safe). Finally, 'Close' will fill the end of the JSON file and the operation will be completed.

func NewContentWriter

func NewContentWriter(arrayKey string, isCompleteFile, useStdout bool) (*ContentWriter, error)

func (*ContentWriter) Close

func (rw *ContentWriter) Close() error

Finish writing to the file.

func (*ContentWriter) GetArrayKey

func (rw *ContentWriter) GetArrayKey() string

func (*ContentWriter) GetError

func (rw *ContentWriter) GetError() error

func (*ContentWriter) GetFilePath

func (rw *ContentWriter) GetFilePath() string

func (*ContentWriter) IsEmpty

func (rw *ContentWriter) IsEmpty() bool

func (*ContentWriter) RemoveOutputFilePath

func (rw *ContentWriter) RemoveOutputFilePath() error

func (*ContentWriter) SetArrayKey

func (rw *ContentWriter) SetArrayKey(arrKey string) *ContentWriter

func (*ContentWriter) Write

func (rw *ContentWriter) Write(record interface{})

Write a single item to the JSON array.

type SortableContentItem

type SortableContentItem interface {
	GetSortKey() string
}