README
¶
MarketStore
MarketStore is a database server optimized for financial timeseries data. You can think of it as an extensible DataFrame service that is accessible from anywhere in your system, at higher scalability.
It is designed from the ground up to address scalability issues around handling large amounts of financial market data used in algorithmic trading backtesting, charting, and analyzing price history with data spanning many years, including tick-level for the all US equities or the exploding crypto currencies space. If you are struggling with managing lots of HDF5 files, this is perfect solution to your problem.
The batteries are included so you can start pulling crypto price data from GDAX right after you install MarketStore. Then you can query DataFrame content over the network at as low latency as your local HDF5 files from disk, and appending new data to the end is two orders of magnitude faster than DataFrame would be. This is because the storage format is optimized for the type of data and use cases as well as for modern filesystem/hardware characteristics.
Install
MarketStore is in pure Go (with some CGO code), so you can build it from source pretty easily. If you want to start right away, use our docker image.
Build From Source
Prerequisite
You need go 1.9+ and dep.
$ go get -u github.com/golang/dep/...
$ mkdir -p /go/src/github.com/alpacahq
$ cd /go/src/github.com/alpacahq
$ git clone https://github.com/alpacahq/marketstore.git
$ cd marketstore
$ make configure
$ make all plugins
Test
$ make unittest
Tutorial
Python Client
pymarketstore is the standard python client.
In [1]: import pymarketstore as pymkts
## query data
In [2]: param = pymkts.Params('BTC', '1Min', 'OHLCV', limit=10)
In [3]: cli = pymkts.Client()
In [4]: reply = cli.query(param)
In [5]: reply.first().df()
Out[5]:
Open High Low Close Volume
Epoch
2018-01-17 17:19:00+00:00 10400.00 10400.25 10315.00 10337.25 7.772154
2018-01-17 17:20:00+00:00 10328.22 10359.00 10328.22 10337.00 14.206040
2018-01-17 17:21:00+00:00 10337.01 10337.01 10180.01 10192.15 7.906481
2018-01-17 17:22:00+00:00 10199.99 10200.00 10129.88 10160.08 28.119562
2018-01-17 17:23:00+00:00 10140.01 10161.00 10115.00 10115.01 11.283704
2018-01-17 17:24:00+00:00 10115.00 10194.99 10102.35 10194.99 10.617131
2018-01-17 17:25:00+00:00 10194.99 10240.00 10194.98 10220.00 8.586766
2018-01-17 17:26:00+00:00 10210.02 10210.02 10101.00 10138.00 6.616969
2018-01-17 17:27:00+00:00 10137.99 10138.00 10108.76 10124.94 9.962978
2018-01-17 17:28:00+00:00 10124.95 10142.39 10124.94 10142.39 2.262249
## write data
In [7]: import numpy as np
In [8]: import pandas as pd
In [9]: data = np.array([(pd.Timestamp('2017-01-01 00:00').value / 10**9, 10.0)], dtype=[('Epoch', 'i8'), ('Ask', 'f4')])
In [10]: cli.write(data, 'TEST/1Min/Tick')
Out[10]: {'responses': None}
In [11]: cli.query(pymkts.Params('TEST', '1Min', 'Tick')).first().df()
Out[11]:
Ask
Epoch
2017-01-01 00:00:00+00:00 10.0
Command-Line Client mkts
Let's test out marketstore by running the runtest.sh example under cmd/tools/mkts/examples.
This test script will create a bucket, load example tick data into the bucket, and run a simple query.
The last few lines of output should match the following:
============================= ========== ========== ==========
Epoch Bid Ask Nanoseconds
============================= ========== ========== ==========
2016-12-31 02:37:57 +0000 UTC 1.05185 1.05197 139999810
2016-12-31 02:38:02 +0000 UTC 1.05185 1.05198 389999832
2016-12-31 02:38:09 +0000 UTC 1.05188 1.052 389999583
2016-12-31 02:38:09 +0000 UTC 1.05189 1.05201 889999385
2016-12-31 02:38:10 +0000 UTC 1.05186 1.05197 139999706
2016-12-31 02:38:10 +0000 UTC 1.05186 1.05192 389999188
2016-12-31 02:38:10 +0000 UTC 1.05181 1.05189 639999508
2016-12-31 02:38:10 +0000 UTC 1.05182 1.0519 889999829
2016-12-31 02:38:11 +0000 UTC 1.05181 1.05189 389999631
2016-12-31 02:38:18 +0000 UTC 1.0518 1.0519 139999900
============================= ========== ========== ==========
Elapsed parse time: 19.523 ms
Elapsed query time: 4.707 ms
Configuration
In order to run MarketStore, a configuration .yaml file is needed. A default file is included in the codebase above and is called mkts.yml. This path to this file is passed in to the launcher binary with the '-config' flag, or by default it finds a file with that name in the directory it is running from. This file should look as follows:
root_directory: /project/data/mktsdb
listen_port: 5993
log_level: info
queryable: true
stop_grace_period: 0
wal_rotate_interval: 5
stale_threshold: 5
enable_add: true
enable_remove: false
- root_directory: allows the user to specify the directory in which the MarketStore database resides (string)
- listen_port: specifies the port that MarketStore will serve through (integer)
- timezone: system timezone by name of TZ database (e.g. America/New_York) default=UTC
- log_level: allows the user to specify the log level (string: info, warning, error)
- queryable: allows the user to run MarketStore in polling-only mode, where it will not respond to query (bool)
- stop_grace_period: sets the amount of time MarketStore will wait to shutdown after a SIGINT signal is received (integer: seconds)
- wal_rotate_interval: frequency at which the WAL file will be trimmed after being flushed to disk (integer, minutes)
- stale_threshold: threshold by which MarketStore will declare a symbol stale (integer, days)
- enable_add: flag allowing new symbols to be added to DB via /write API
- enable_remove: flag allowing symbols to be removed from DB via /write API
- triggers: list of trigger plugins
- bgworkers: list of background worker plugins
GDAX Data Fetcher
You can start pulling data from GDAX if you configure the data poller. For more information, see GDAX Feeder Document
On-Disk Aggregate
You only need to collect tick/minute level data. Time-based aggregation on disk can be done via On-Disk Aggregate
Query Parameter Spec
Please see another doc
Plug-in Architecture
We know the needs and requirements in this space are diverse. TheMarketStore provides strong core functionality with flexible plug-in architecture. If you want to build your own, look around plugins
Bug Report & Contribution
If you are interested in improving MarketStore, you are more than welcome! Just file issues or requests in github or contact oss@alpaca.markets.
Is This Production-Ready?
Yes, absolutely! It has been used in production for years in serious business, but we also never feel it is complete. You can use it for your own purpose and give us more feedback.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
contrib
|
|
|
ondiskagg
This is a shim package for buiding a plugin module wrapping the importable aggtrigger package.
|
This is a shim package for buiding a plugin module wrapping the importable aggtrigger package. |
|
ondiskagg/aggtrigger
OnDiskAgg implements a trigger to downsample base timeframe data and write to disk.
|
OnDiskAgg implements a trigger to downsample base timeframe data and write to disk. |
|
ondiskagg/calendar
Package calendar provides market calendar, with which you can check if the market is open at specific point of time.
|
Package calendar provides market calendar, with which you can check if the market is open at specific point of time. |
|
buffile
package buffile helps batch write by writes to the tempobary in-memory buffer under the assumption that many writes come in to the part of single file frequently.
|
package buffile helps batch write by writes to the tempobary in-memory buffer under the assumption that many writes come in to the part of single file frequently. |
|
bgworker
Package bgworker provides interface for bgworker plugins.
|
Package bgworker provides interface for bgworker plugins. |
|
trigger
Package trigger provides interface for trigger plugins.
|
Package trigger provides interface for trigger plugins. |