stores

Documentation

Index

• type Binary
• type Delimited
• type Document
• type File
• type MongoDocument
• type Relational
• type Spreadsheet
• type Store
  • func (s *Store) MarshalJSON() ([]byte, error)
  • func (s *Store) UnmarshalJSON(b []byte) error
• type Text

Types

type Binary

type Binary struct {
	File

	// The byte position, range or series of bytes and ranges.
	Bytes string `json:"bytes,omitempty" bson:"bytes,omitempty"`
}

Store for binary files or text files which are not line-based. The location of the data is defined by byte positions and ranges.

type Delimited

type Delimited struct {
	Text

	// The delimiter used which denotes the structure
	Delimiter string `json:"delimiter,omitempty" bson:"delimiter,omitempty"`

	// A line number or range of lines representing the header. Note, this
	// should only be supplied if the columns are guaranteed consistent for
	// all rows in the file.
	Header string `json:"header,omitempty" bson:"header,omitempty"`

	// The column index or name (if a header exists) where the data is
	// defined. For indexes, this can be a range. For column names this
	// can be a comma-separated list of names.
	Column string `json:"column,omitempty" bson:"column,omitempty"`
}

Store for delimited text files. The location is denoted by the line and column fields. Examples includes CSV and tab-delimited files.

type Document

type Document struct {
	Store

	// A forward slash-delimited path to the value
	Path string `json:"path,omitempty" bson:"path,omitempty"`
}

Store for JSON-based data

type File

type File struct {
	Store

	// The name of the file. If the uri is supplied with a path, the name of
	// the file will be extracted if not supplied.
	Name string `json:"name,omitempty" bson:"name,omitempty"`
}

The base type for file-based stores

type MongoDocument

type MongoDocument struct {
	Document

	// The database name. If not supplied the database name will attempt to be
	// extracted from the uri
	Database string `json:"database,omitempty" bson:"database,omitempty"`

	// The collection name where the document is stored.
	Collection string `json:"collection,omitempty" bson:"collection,omitempty"`
}

Store for documents stored in MongoDB databases

type Relational

type Relational struct {
	Store

	// The database name
	Database string `json:"database,omitempty" bson:"database,omitempty"`

	// The schema name for databases that support them, e.g. PostgreSQL
	Schema string `json:"schema,omitempty" bson:"schema,omitempty"`

	// The table name
	Table string `json:"table,omitempty" bson:"table,omitempty"`

	// Object containing the lookup of the row
	Row map[string]interface{} `json:"row,omitempty" bson:"row,omitempty"`

	// The column name
	Column string `json:"column,omitempty" column:"column,omitempty"`
}

Store representing a data based on the relational model. Examples include PostgreSQL, SQLite, Oracle, MySQL.

type Spreadsheet

type Spreadsheet struct {
	File

	// The index or name of the sheet
	Sheet string `json:"sheet",bson:"sheet"`

	// A line number or range of lines representing the header.
	Header string `json:"header",bson:"header"`

	// The line, line range or series of lines and ranges.
	Row string `json:"row",bson:"row"`

	// The column index or name (if a header exists) where the data is
	// defined. For indexes, this can be a range. For column names this
	// can be a comma-separated list of names.
	Column string `json:"column",bson:"column"`
}

Store for spreadsheet files such as Microsoft Excel, Google Docs Spreadsheet, and OpenOffice Spreadsheet. Data is tabular.

type Store

type Store struct {
	// Represents the Mongo ID
	Id bson.ObjectId `json:"-" bson:"_id,omitempty"`

	// URI which denotes where the data lives, where came from (streamed) or
	// where can be accessed in the future. The latter value is preferred
	// since it theoretically enables future access. The completeness and
	// value of this is store dependent.
	Uri string `json:"uri,omitempty" bson:"uri,omitempty"`

	// The store type as a string. This is used when messages are parsed and
	// for downstream processing. If not defined, the type will attempted to
	// be inferred by keys supplied.
	Type string `json:"type,omitempty" bson:"type,omitempty"`

	// The value or array of values processed. For sources this would
	// typically be the pre-transformed data while for targets this would be
	// post-processed data. Supplying the value here is typically unnecessary
	// since the value could be accessed using the other information supplied
	// in store data. However, for systems that treat this as an audit log or
	// if the target system does not perform an versioning of data of it's
	// own, this could act as primitive store of values.
	Value interface{} `json:"value,omitempty" bson:"value,omitempty"`

	// Extra data from decoded JSON. This enables clients to store additional
	// metadata about the store.
	Extra map[string]interface{} `json:"-" bson:",inline,omitempty"`
}

Base which contains general information about the store

func (*Store) MarshalJSON

func (s *Store) MarshalJSON() ([]byte, error)

Satisfies the `json.Marshaler` interface

func (*Store) UnmarshalJSON

func (s *Store) UnmarshalJSON(b []byte) error

Satisfies the `json.Unmarshaler` interface

type Text

type Text struct {
	File

	// The line, line range or series of lines and ranges.
	Line string `json:"line,omitempty" bson:"line,omitempty"`
}

Store for text files. The location of the data is defined by lines and chars.