README
¶
sophie 
A sequencial data processing library.
Documentation
¶
Overview ¶
Package sophie provides an raw mechanism for serializing data.
It aims at more efficiency than other serialization methods because of the following reasons:
* Maximum of reusing objects, allocation and GC are avoided * No reflections
Since the serialization is flexible, one can also make some trade-offs between efficiency and convinience. E.g., if the data structure may be changed in the future, in the ReadFrom/WriteTo, God codec can be used to provide future compatibility.
Sub packages:
mr MapReduce library kv A file format storing key-value pairs.
Index ¶
- Constants
- Variables
- func ReadStringSlice(r Reader, sl *[]string) (err error)
- func WriteStringSlice(w Writer, sl []string) error
- type BufferedFileReader
- type BufferedFileWriter
- type ByteSlice
- type CollectCloser
- type CollectCloserStruct
- type Collector
- type CollectorF
- type EmptyClose
- type FileSystem
- type FsPath
- func (fp FsPath) Create() (WriteCloser, error)
- func (fp FsPath) Join(sub string) FsPath
- func (fp FsPath) Mkdir(perm os.FileMode) error
- func (fp FsPath) Open() (ReadCloser, error)
- func (fp FsPath) ReadDir() ([]os.FileInfo, error)
- func (fp FsPath) Remove() error
- func (fp FsPath) Stat() (os.FileInfo, error)
- type Int32
- type IterateCloser
- type IterateCloserStruct
- type Iterator
- type Null
- type RawByteSlice
- type RawString
- type RawVInt
- type ReadCloser
- type Reader
- type SophieReader
- type SophieWriter
- type Sophier
- type String
- type Time
- type VInt
- type WriteCloser
- type Writer
Constants ¶
const UNKNOWN_LEN = -1
The constants for unknown length. @see SophieReader.ReadFrom
Variables ¶
var ( // Returned if some Sophie file is found to have bad format. ErrBadFormat = errors.New("Bad Sophie format") )
var NullCollectCloser = nullCollectCloser{}
A helper variable with a CollectCloser ignoring every thing collecting to it
Functions ¶
func ReadStringSlice ¶
A helper function that reads a slice String from a Reader. The data was serialized by WriteStringSlice function.
func WriteStringSlice ¶
A helper function that writes a slice of Strings to a Writer. Serialized data can be read by ReadStringSlice function.
Types ¶
type BufferedFileReader ¶
BufferedFileReader is a sophie.ReadCloser with buffer
func (BufferedFileReader) Close ¶
func (b BufferedFileReader) Close() error
sophie.ReadCloser interface
type BufferedFileWriter ¶
BufferedFileWriter is a sophie.WriteCloser with buffer.
func (BufferedFileWriter) Close ¶
func (b BufferedFileWriter) Close() error
sophie.WriteCloser interface
type ByteSlice ¶
type ByteSlice []byte
*ByteSlice implements Sophier interface.
type CollectCloser ¶
CollectCloser is Collector + io.Closer
type CollectCloserStruct ¶
type CollectCloserStruct struct {
CollectF func(SophieWriter, SophieWriter) error
CloseF func() error
}
CollectCloserStruct is a struct whose pointer implements CollectCloser interface
func (*CollectCloserStruct) Close ¶
func (c *CollectCloserStruct) Close() error
func (*CollectCloserStruct) Collect ¶
func (c *CollectCloserStruct) Collect(k, v SophieWriter) error
type Collector ¶
type Collector interface {
Collect(key, val SophieWriter) error
}
Collector is an interface for collecting Sophie key-value pairs.
type CollectorF ¶
type CollectorF func(key, val SophieWriter) error
A func type implementing Collector interface. A nil func is a no-op Collector.
func (CollectorF) Collect ¶
func (c CollectorF) Collect(key, val SophieWriter) error
Collector interface.
type EmptyClose ¶
type EmptyClose struct{}
EmptyClose is a helper type defining an empty Close method
type FileSystem ¶
type FileSystem interface {
// Create creates a file of a specified name.
Create(fn string) (WriteCloser, error)
// Mkdir makes the directory and its parents if necessary of specifiy path
// and perm.
Mkdir(path string, perm os.FileMode) error
// Open opens a ReadCloser for reading the file of a specified name.
Open(fn string) (ReadCloser, error)
// ReadDir reads the FileInfos of the files under a specified directory.
ReadDir(dir string) ([]os.FileInfo, error)
// Stat returns the FileInfo of a file/directory of a specified name.
Stat(fn string) (os.FileInfo, error)
// Remove deletes a file or a directory(and all its file/directories in it)
Remove(fn string) error
}
An interface defining some actions an file-system should have.
var ( // LocalFS is a FileSystem representing the local file-system. LocalFS FileSystem = localFileSystem{} )
type FsPath ¶
type FsPath struct {
// The FileSystem
Fs FileSystem
// The path
Path string
}
FsPath is a pair of FileSystem and a path
func LocalFsPath ¶
LocalFsPath returns an FsPath of the LocalFS and the specified path
func TempDirPath ¶
func TempDirPath() FsPath
TempDirPath returns the OS temporary dir as the fs path.
func (FsPath) Create ¶
func (fp FsPath) Create() (WriteCloser, error)
Calls FileSystem.Create with the path
func (FsPath) Join ¶
Join returns a new FsPath with the same FileSystem and the path joined with sub
func (FsPath) Open ¶
func (fp FsPath) Open() (ReadCloser, error)
Calls FileSystem.Open with the path
type IterateCloser ¶
sohpie.IterateCloser is Iterator + io.Closer
type IterateCloserStruct ¶
type IterateCloserStruct struct {
NextF func(key, val SophieReader) error
CloserF func() error
}
A struct implementing IterateCloser by funcs.
func (*IterateCloserStruct) Next ¶
func (ics *IterateCloserStruct) Next(key, val SophieReader) error
Iterator interface
type Iterator ¶
type Iterator interface {
Next(key, val SophieReader) error
}
Iterator is an interface for iterating Sophier kv pairs.
type RawByteSlice ¶
type RawByteSlice []byte
*RawByteSlice implements Sophier interface. It encodes byte-slice assuming the length of buffer will be known when decoding.
type RawString ¶
type RawString string
*RawString implements Sophie interface. It assumes the length to be known.
type RawVInt ¶
type RawVInt int
*RawVInt implements Sophie interface and serializing as a vint. It assumes the length to be known.
type ReadCloser ¶
sophie.ReadCloser is sohpie.Reader + io.Closer
type Reader ¶
type Reader interface {
io.Reader
io.ByteReader
// Skip skips n bytes, returns the number of actually skipped
Skip(n int64) (int64, error)
}
sophie.Reader is an interface extended from io.Reader + io.ByteReader
type SophieReader ¶
type SophieReader interface {
// ReadFrom reads fields from a Reader. If l is not UNKNOWN_LEN, it can be
// used to determine the border of the serialized data.
// @param l the number of bytes to read. UNKNOWN_LEN(-1) means unknown
// length
ReadFrom(r Reader, l int) error
}
SophieReader is the interface for some data structure that can reads fields from a Reader. The data in the Reader could have known length. For all predefined Sophies with prefix Raw, the length must be specified.
type SophieWriter ¶
SophieWriter is the interface for some data structure that can write fields to a Writer.
type Sophier ¶
type Sophier interface {
SophieReader
SophieWriter
}
Sophier is a basic data structure for serialization. It is SophieReader + SophieWriter
func NewRawByteSlice ¶
func NewRawByteSlice() Sophier
Returns a new instance of *RawByteSlice as a Sophier
func NewRawString ¶
func NewRawString() Sophier
Returns a new instance of *RawByteSlice as a Sophier
func ReturnNULL ¶
func ReturnNULL() Sophier
type String ¶
type String string
*String implements Sophie interface
func ReadString ¶
A helper function that reads a String from a Reader.
type WriteCloser ¶
sophie.WriteCloser is sophie.Writer + io.Closer
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package kv supporting read and write of a simple file formating for Sophie, which stores key-value pairs.
|
Package kv supporting read and write of a simple file formating for Sophie, which stores key-value pairs. |
|
Package mr provides a local concurrent computing model(MapReduce) using Sophie serialization.
|
Package mr provides a local concurrent computing model(MapReduce) using Sophie serialization. |