Documentation
¶
Overview ¶
Package memorystream provides a DataSource which streams data from an in-memory buffer. Primarily for testing/mocking purposes.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CreateDataFrame ¶
func CreateDataFrame(generators []func() []byte, batchSize int, parser sif.DataSourceParser, schema sif.Schema) sif.DataFrame
CreateDataFrame is a factory for DataSources
Types ¶
type DataSource ¶
type DataSource struct {
// contains filtered or unexported fields
}
DataSource is a buffer containing data which will be manipulating according to a DataFrame
func (*DataSource) Analyze ¶
func (ms *DataSource) Analyze() (sif.PartitionMap, error)
Analyze returns a PartitionMap, describing how the source data will be divided into Partitions
func (*DataSource) DeserializeLoader ¶
func (ms *DataSource) DeserializeLoader(bytes []byte) (sif.PartitionLoader, error)
DeserializeLoader creates a PartitionLoader for this DataSource from a serialized representation
func (*DataSource) IsStreaming ¶
func (ms *DataSource) IsStreaming() bool
IsStreaming returns true iff this DataSource provides a continuous stream of data
type PartitionLoader ¶
type PartitionLoader struct {
// contains filtered or unexported fields
}
PartitionLoader is capable of loading partitions of data from a file
func (*PartitionLoader) GobDecode ¶
func (pl *PartitionLoader) GobDecode(in []byte) error
GobDecode deserializes a PartitionLoader
func (*PartitionLoader) GobEncode ¶
func (pl *PartitionLoader) GobEncode() ([]byte, error)
GobEncode serializes a PartitionLoader
func (*PartitionLoader) Load ¶
func (pl *PartitionLoader) Load(parser sif.DataSourceParser) (sif.PartitionIterator, error)
Load is capable of loading partitions of data from a file
func (*PartitionLoader) ToString ¶
func (pl *PartitionLoader) ToString() string
ToString returns a string representation of this PartitionLoader
type PartitionMap ¶
type PartitionMap struct {
// contains filtered or unexported fields
}
PartitionMap is an iterator producing a sequence of PartitionLoaders
func (*PartitionMap) HasNext ¶
func (pm *PartitionMap) HasNext() bool
HasNext returns true iff there is another PartitionLoader remaining
func (*PartitionMap) Next ¶
func (pm *PartitionMap) Next() sif.PartitionLoader
Next returns the next PartitionLoader for a data generator
Source Files