frontend

package
v4.1.15 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 16, 2022 License: Apache-2.0 Imports: 22 Imported by: 0

README

MarketStore frontend API

Transport Methods

MarketStore communicates with its clients through standard HTTP in Messagepack RPC (Messagepack version of JSON-RPC 2.0).

DataService.ListSymbols()

Input

no parameters

Output

list of string for each unique symbol stored in the server.

DataService.Query()

Input

Query() interface accepts a list of "requests", each of which is a map with the following fields.

  • destination (string)

    A string path of the query target. A TimeBucketKey contains a Symbol, Timeframe, and an AttributeGroup. For example, "TSLA/1Min/OHLCV" is an example TimeBucketKey. In this example, TSLA is the Symbol, 1Min is the TimeFrame, and OHLCV is the AttributeGroup. Moreover, a single destination can include multiple symbols split by commas for a multi-symbol query. For example, "TSLA,F,NVDA/1Min/OHLCV" will query data for Symbols TSLA, F, and NVDA all across the same TimeFrame, AttributeGroup.

  • epoch_start (int64)

    An integer epoch seconds from Unix epoch time. Rows timestamped equal to or after this time will be returned.

  • epoch_end (int64)

    An integer epoch seconds from Unix epoch time. Rows timestamped equal to or before this time will be returned.

  • limit_record_count (int)

    An integer to limit the number of rows to be returned from the query.

  • limit_from_start (bool)

    A boolean value to indicate if limit_recourd_count should be counted from the lower side of result set or upper. Default to false, meaning from the upper.

Note: It is also possible to query multiple TimeBucketKeys at once. The requests parameter is passed a list of query structures (See examples).

Output

The output returns the same number of "responses" as the requests, each of which has the following fields.

  • result

    A MultiDataset type. See below for this type.

DataService.Write()

Input

  • dataset

    A MultiDataset type. See below for this type.

  • is_variable_length (bool)

    A boolean value for telling MarketStore if the write procedure will be dynamic in length.

Output

The API will return an empty response on success. Should the write call fail, the response will include the original input as well as an error returned by the server.

MultiDataset type

This is the common wire format to represent a series of columns containing multiple slices (horizontal partitions). It is a map with the following fields that represents set of column-oriented data

  • types ([]string)

    a list of strings for the column types compatible with numpy dtypes (e.g., 'i4', 'f8')

  • names ([]string)

    a list of strings for the column names

  • data ([][]byte)

    a list of byte arrays with each being the binary column data

  • startindex ([]int)

    a list of integer to indicate which element each slice starts at

  • lengths ([]int)

    a list of integer to indicate how many elements each slice has

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var Queryable uint32 // treated as bool

Functions

func NewDataShapeVector 

func NewDataShapeVector(dataShapes []*proto.DataShape) (dsv []io.DataShape, err error)

NewDataShapeVector returns a new array of io.DataShape for the given array of proto.DataShape inputs.

func NewServer 

func NewServer(rootDir string, catDir *catalog.Directory, aggRunner *sqlparser.AggRunner,
	w Writer, q QueryInterface,
) (*RPCServer, *DataService)

func Profile 

func Profile(address string)

func ToNumpyMultiDataSet 

func ToNumpyMultiDataSet(p *proto.NumpyMultiDataset) *io.NumpyMultiDataset

func ToProtoNumpyMultiDataSet 

func ToProtoNumpyMultiDataSet(nmds *io.NumpyMultiDataset) *proto.NumpyMultiDataset

Types

type CreateRequest 

type CreateRequest struct {
	// bucket key string. e.g. "TSLA/1Min/OHLC"
	Key string `msgpack:"key"`
	// a list of type strings such as i4 and f8
	ColumnTypes []string `msgpack:"column_types"`
	// a list of column names
	ColumnNames      []string `msgpack:"column_names"`
	IsVariableLength bool     `msgpack:"is_variable_length"`
}

Create: Creates a new time bucket in the DB

type DataService 

type DataService struct {
	// contains filtered or unexported fields
}

func NewDataService added in v4.1.1 

func NewDataService(rootDir string, catDir *catalog.Directory, aggRunner *sqlparser.AggRunner,
	w Writer, q QueryInterface,
) *DataService

func (*DataService) Create 

func (s *DataService) Create(_ *http.Request, reqs *MultiCreateRequest, response *MultiServerResponse) (err error)

func (*DataService) Destroy 

func (s *DataService) Destroy(_ *http.Request, reqs *MultiKeyRequest, response *MultiServerResponse) (err error)

func (*DataService) GetInfo 

func (s *DataService) GetInfo(_ *http.Request, reqs *MultiKeyRequest, response *MultiGetInfoResponse) (err error)

func (*DataService) Init 

func (s *DataService) Init()

func (*DataService) ListSymbols 

func (s *DataService) ListSymbols(r *http.Request, req *ListSymbolsRequest, response *ListSymbolsResponse) (err error)

func (*DataService) Query 

func (s *DataService) Query(r *http.Request, reqs *MultiQueryRequest, response *MultiQueryResponse) (err error)

func (*DataService) Write 

func (s *DataService) Write(_ *http.Request, reqs *MultiWriteRequest, response *MultiServerResponse) (err error)

type GRPCService 

type GRPCService struct {
	// contains filtered or unexported fields
}

GRPCService is the implementation of GRPC API for Marketstore. All grpc/protobuf-related logics and models are defined in this file.

func NewGRPCService added in v4.1.1 

func NewGRPCService(rootDir string, catDir *catalog.Directory, aggRunner *sqlparser.AggRunner,
	w Writer, q QueryInterface,
) *GRPCService

func (GRPCService) Create added in v4.1.1 

func (s GRPCService) Create(ctx context.Context, req *proto.MultiCreateRequest) (*proto.MultiServerResponse, error)

func (GRPCService) Destroy 

func (s GRPCService) Destroy(ctx context.Context, req *proto.MultiKeyRequest) (*proto.MultiServerResponse, error)

func (GRPCService) ListSymbols 

func (s GRPCService) ListSymbols(ctx context.Context, req *proto.ListSymbolsRequest,
) (*proto.ListSymbolsResponse, error)

func (GRPCService) Query 

func (s GRPCService) Query(_ context.Context, reqs *proto.MultiQueryRequest) (*proto.MultiQueryResponse, error)

func (GRPCService) ServerVersion 

func (s GRPCService) ServerVersion(ctx context.Context, req *proto.ServerVersionRequest,
) (*proto.ServerVersionResponse, error)

func (GRPCService) Write 

func (s GRPCService) Write(ctx context.Context, reqs *proto.MultiWriteRequest) (*proto.MultiServerResponse, error)

type GetInfoResponse 

type GetInfoResponse struct {
	LatestYear int
	TimeFrame  time.Duration
	DSV        []io.DataShape
	RecordType io.EnumRecordType
	ServerResp ServerResponse
}

type HeartbeatMessage 

type HeartbeatMessage struct {
	Status  string `json:"status"`
	Version string `json:"version"`
	GitHash string `json:"git_hash"`
	Uptime  string `json:"uptime"`
}

type KeyRequest 

type KeyRequest struct {
	Key string `msgpack:"key"`
}

type ListSymbolsRequest added in v4.0.1 

type ListSymbolsRequest struct {
	// "symbol", or "tbk"
	Format string `msgpack:"format,omitempty"`
}

type ListSymbolsResponse 

type ListSymbolsResponse struct {
	Results []string
}

type MultiCreateRequest 

type MultiCreateRequest struct {
	Requests []CreateRequest `msgpack:"requests"`
}

type MultiGetInfoResponse 

type MultiGetInfoResponse struct {
	Responses []GetInfoResponse `msgpack:"responses"`
}

type MultiKeyRequest 

type MultiKeyRequest struct {
	Requests []KeyRequest `msgpack:"requests"`
}

type MultiQueryRequest 

type MultiQueryRequest struct {
	/*
		A multi-request allows for different Timeframes and record formats for each request
	*/
	Requests []QueryRequest `msgpack:"requests"`
}

type MultiQueryResponse 

type MultiQueryResponse struct {
	Responses []QueryResponse `msgpack:"responses"`
	Version   string          `msgpack:"version"`  // Server Version
	Timezone  string          `msgpack:"timezone"` // Server Timezone
}

func (*MultiQueryResponse) ToColumnSeriesMap 

func (resp *MultiQueryResponse) ToColumnSeriesMap() (*io.ColumnSeriesMap, error)

ToColumnSeriesMap converts a MultiQueryResponse to a ColumnSeriesMap, returning an error if there is any issue encountered while converting.

type MultiServerResponse 

type MultiServerResponse struct {
	Responses []ServerResponse `msgpack:"responses"`
}

type MultiWriteRequest 

type MultiWriteRequest struct {
	/*
		A multi-request allows for different Timeframes and record formats for each request
	*/
	Requests []WriteRequest `msgpack:"requests"`
}

type QueryInterface added in v4.1.2 

type QueryInterface interface {
	ExecuteQuery(tbk *io.TimeBucketKey, start, end time.Time, LimitRecordCount int,
		LimitFromStart bool, columns []string,
	) (io.ColumnSeriesMap, error)
}

type QueryRequest 

type QueryRequest struct {
	// Note: SQL is not fully supported
	IsSQLStatement bool   `msgpack:"is_sqlstatement"` // If this is a SQL request, Only SQLStatement is relevant
	SQLStatement   string `msgpack:"sql_statement"`

	// Destination is <symbol>/<timeframe>/<attributegroup>
	Destination string `msgpack:"destination"`
	// This is not usually set, defaults to Symbol/Timeframe/AttributeGroup
	KeyCategory string `msgpack:"key_category,omitempty"`
	// Lower time predicate (i.e. index >= start) in unix epoch second
	EpochStart *int64 `msgpack:"epoch_start,omitempty"`
	// Nanosecond of the lower time predicate
	EpochStartNanos *int64 `msgpack:"epoch_start_nanos,omitempty"`
	// Upper time predicate (i.e. index <= end) in unix epoch second
	EpochEnd *int64 `msgpack:"epoch_end,omitempty"`
	// Nanosecond of the upper time predicate
	EpochEndNanos *int64 `msgpack:"epoch_end_nanos,omitempty"`
	// Number of max returned rows from lower/upper bound
	LimitRecordCount *int `msgpack:"limit_record_count,omitempty"`
	// Set to true if LimitRecordCount should be from the lower
	LimitFromStart *bool `msgpack:"limit_from_start,omitempty"`
	// Array of column names to be returned
	Columns []string `msgpack:"columns,omitempty"`

	// Support for functions is experimental and subject to change
	Functions []string `msgpack:"functions,omitempty"`
}

This is the parameter interface for DataService.Query method.

type QueryRequestBuilder 

type QueryRequestBuilder struct {
	// contains filtered or unexported fields
}

QueryRequestBuilder is a builder for QueryRequest to set various parameters flexibly. 

qr := NewQueryRequestBuilder("TSLA/1D/OHLCV").LimitRecourdCount(100).End()

func NewQueryRequestBuilder 

func NewQueryRequestBuilder(destination string) *QueryRequestBuilder

func (*QueryRequestBuilder) End 

func (b *QueryRequestBuilder) End() QueryRequest

func (*QueryRequestBuilder) EpochEnd 

func (b *QueryRequestBuilder) EpochEnd(value int64) *QueryRequestBuilder

func (*QueryRequestBuilder) EpochStart 

func (b *QueryRequestBuilder) EpochStart(value int64) *QueryRequestBuilder

func (*QueryRequestBuilder) Functions 

func (b *QueryRequestBuilder) Functions(value []string) *QueryRequestBuilder

func (*QueryRequestBuilder) LimitFromStart 

func (b *QueryRequestBuilder) LimitFromStart(value bool) *QueryRequestBuilder

func (*QueryRequestBuilder) LimitRecordCount 

func (b *QueryRequestBuilder) LimitRecordCount(value int) *QueryRequestBuilder

type QueryResponse 

type QueryResponse struct {
	Result *io.NumpyMultiDataset `msgpack:"result"`
}

type QueryService added in v4.1.2 

type QueryService struct {
	// contains filtered or unexported fields
}

func NewQueryService added in v4.1.2 

func NewQueryService(catDir *catalog.Directory) *QueryService

func (*QueryService) ExecuteQuery added in v4.1.2 

func (qs *QueryService) ExecuteQuery(tbk *io.TimeBucketKey, start, end time.Time, limitRecordCount int,
	limitFromStart bool, columns []string,
) (io.ColumnSeriesMap, error)

type RPCServer added in v4.1.13 

type RPCServer struct {
	*rpc.Server
}

func (*RPCServer) ServeHTTP added in v4.1.13 

func (s *RPCServer) ServeHTTP(w http.ResponseWriter, r *http.Request)

type ServerResponse 

type ServerResponse struct {
	Error   string `msgpack:"error"`
	Version string `msgpack:"version"` // Server Version
}

type UtilityAPIHandlers added in v4.1.13 

type UtilityAPIHandlers struct {
	// contains filtered or unexported fields
}

func NewUtilityAPIHandlers added in v4.1.1 

func NewUtilityAPIHandlers(startTime time.Time) *UtilityAPIHandlers

func (*UtilityAPIHandlers) Handle added in v4.1.13 

func (uah *UtilityAPIHandlers) Handle(url string) error

type WriteRequest 

type WriteRequest struct {
	Data             *io.NumpyMultiDataset `msgpack:"dataset"`
	IsVariableLength bool                  `msgpack:"is_variable_length"`
}

type Writer added in v4.1.2 

type Writer interface {
	WriteCSM(csm io.ColumnSeriesMap, isVariableLength bool) error
}

Source Files

View all Source files

Directories

Path Synopsis
Package stream implements websocket interface for streaming in the server core.
 Package stream implements websocket interface for streaming in the server core.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.