mem

Documentation

Index

• func GetMemFileFs() afero.Fs
• func NewMemFileWriter(name string, f OnCloseFunc) (source.ParquetFile, error)
• func SetInMemFileFs(fs *afero.Fs)
• type MemFile
  • func (fs *MemFile) Close() error
  • func (fs *MemFile) Create(name string) (source.ParquetFile, error)
  • func (fs *MemFile) Open(name string) (source.ParquetFile, error)
  • func (fs *MemFile) Read(b []byte) (cnt int, err error)
  • func (fs *MemFile) Seek(offset int64, pos int) (int64, error)
  • func (fs *MemFile) Write(b []byte) (n int, err error)
• type OnCloseFunc

Functions

func GetMemFileFs

func GetMemFileFs() afero.Fs

GetMemFileFs - returns the current memory file-system being used by ParquetFile

func NewMemFileWriter

func NewMemFileWriter(name string, f OnCloseFunc) (source.ParquetFile, error)

NewMemFileWriter - intiates and creates an instance of MemFiles NOTE: there is no NewMemFileReader as this particular type was written to handle in-memory conversions and offloading. The results of conversion can then be stored and read via HDFS, LocalFS, etc without the need for loading the file back into memory directly

func SetInMemFileFs

func SetInMemFileFs(fs *afero.Fs)

SetInMemFileFs - overrides local in-memory fileSystem NOTE: this is set by NewMemFileWriter is created and memFs is still nil

Types

type MemFile

type MemFile struct {
	FilePath string
	File     afero.File
	OnClose  OnCloseFunc
}

MemFile - ParquetFile type for in-memory file operations

func (*MemFile) Close

func (fs *MemFile) Close() error

Close - close file and execute OnCloseFunc

func (*MemFile) Create

func (fs *MemFile) Create(name string) (source.ParquetFile, error)

Create - create in-memory file

func (*MemFile) Open

func (fs *MemFile) Open(name string) (source.ParquetFile, error)

Open - open file in-memory

func (*MemFile) Read

func (fs *MemFile) Read(b []byte) (cnt int, err error)

Read - read file

func (*MemFile) Seek

func (fs *MemFile) Seek(offset int64, pos int) (int64, error)

Seek - seek function

func (*MemFile) Write

func (fs *MemFile) Write(b []byte) (n int, err error)

Write - write file in-memory

type OnCloseFunc

type OnCloseFunc func(string, io.Reader) error

OnCloseFunc function type, handles what to do after converted file is closed in-memory. Close() will pass the filename string and data as io.reader