xignitefeeder

README

Xignite Feeder

• This plugin retrieves quotes data for all symbols in (a) specified exchange(s) by QUICK Xignite API and store it to the local marketstore server.
• You need an API token to call Xignite API. Register to Xignite and generate your API token first.
• This plugin is also able to collect daily candlestick chart data from a specified date and backfill it to the local marketstore. this historical backfill process can be executed when marketstore is started and the configured time everyday (see update_time config )

Example configuration

bgworkers:
  # -----------------------
  # XigniteFeeder gets the realtime stock data by Xignite API (https://www.quick.co.jp/qxapis/index.php)
  # and writes its to the local marketstore at fixed intervals.
  # -----------------------
  - module: xignitefeeder.so
    config:
      # exchange list
      exchanges:
        - XTKS # Tokyo Stock Exchange
        #- XNGO # Nagoya Stock Exchange
        #- XSAP # Sapporo Stock Exchange
        #- XFKA # Fukuoka Stock Exchange
      # Xignite feeder also retrieves data of Index Symbols (ex. ＴＯＰＩＸ（東証１部株価指数）) every day.
      # To get target indices, index groups that the indices belong are necessary.
      # (cf. https://www.marketdata-cloud.quick-co.jp/Products/QUICKIndexHistorical/Overview/ListSymbols )
      index_groups:
        - INDXJPX # JAPAN EXCHANGE GROUP
        - IND_NIKKEI # NIKKEI INDICES
      # Symbols in not_quote_stock_list won't be queried or stored in the database. 
      # This config can be manually overridden by 
      # XIGNITE_FEEDER_NOT_QUOTE_STOCK_LIST (=comma-separated list of stock codes) environmental variable.
      # (e.g. XIGNITE_FEEDER_NOT_QUOTE_STOCK_LIST="1111,2222,3333")
      not_quote_stock_list: 
        #- "1111"
        #- "2222"
        #- "3333"
      # time when target symbols in the exchanges are updated everyday.
      # this time is also used for the historical data backfill (UTC)
      # This config can be manually overridden by "XIGNITE_FEEDER_UPDATE_TIME" environmental variable.
      update_time: "22:00:00" # (UTC). = every day at 07:00:00 (JST)
      # XigniteFeeder writes data to "{identifier}/{timeframe}/TICK" TimeBucketKey
      timeframe: "1Sec"
      # Base URL of the API
      base_url: "https://stg-api.mk-smapi.jp/"
      # Endpoint of each Xigntie API
      endpoint:
        equity_realtime_get_quotes: "MINKABUEquityRealTime.json/GetQuotes"
        equity_realtime_list_symbols: "MINKABUEquityRealTime.json/ListSymbols"
        equity_realtime_get_bars: "MINKABUEquityRealTime.json/GetBars"
        equity_historical_get_quotes_range: "MINKABUEquityHistorical.json/GetQuotesRange"
        index_realtime_get_bars: "MINKABUIndexRealTime.json/GetBars"
        index_historical_list_symbols: "MINKABUIndexHistorical.json/ListSymbols"
        index_historical_get_quotes_range: "MINKABUIndexHistorical.json/GetQuotesRange"
      # Auth token for Xignite API
      # This config can be manually overridden by "XIGNITE_FEEDER_API_TOKEN" environmental variable.
      token: "D***0"
      # Timeout [sec] for Xignite API
      timeout: 10
      # Interval [sec] to call Xignite API
      interval: 10
      # The data-feeding is executed when 'minute' of the current time matches off_hours_schedule
      # even when the market is cloded. Example: "10" -> execute at 00:10, 01:10, 02:10,...,23:10
      # Numbers separated by commas are allowed.  Example: "0,15,30,45" -> execute every 15 minutes.
      # Whitespaces are ignored.
      off_hours_schedule: "0,15,30,45"
      # XigniteFeeder runs from openTime ~ closeTime (UTC)
      openTime: "23:00:00" # 08:00 (JST)
      closeTime: "06:10:00" # 15:10 (JST)
      # XigniteFeeder doesn't run on the following days
      closedDaysOfTheWeek:
        - "Saturday"
        - "Sunday"
      # XigniteFeeder doesn't run on the closed dates (in JST)
      # (cf. https://www.jpx.co.jp/corporate/about-jpx/calendar/ )
      closedDays:
        - "2019-01-01"
        - "2019-01-02"
        - "2019-01-03"
        - "2019-01-14"
        - "2019-02-11"
        - "2019-03-21"
        - "2019-04-29"
        - "2019-04-30"
        - "2019-05-01"
        - "2019-05-02"
        - "2019-05-03"
        - "2019-05-04"
        - "2019-05-05"
        - "2019-05-06"
        - "2019-07-15"
        - "2019-08-11"
        - "2019-08-12"
        - "2019-09-16"
        - "2019-09-23"
        - "2019-10-14"
        - "2019-10-22"
        - "2019-11-03"
        - "2019-11-04"
        - "2019-11-23"
        - "2019-12-31"
        - "2020-01-01"
        - "2020-01-02"
        - "2020-01-03"
        - "2020-01-13"
        - "2020-02-11"
        - "2020-02-23"
        - "2020-02-24"
        - "2020-03-20"
        - "2020-04-29"
        - "2020-05-03"
        - "2020-05-04"
        - "2020-05-05"
        - "2020-05-06"
        - "2020-07-23"
        - "2020-07-24"
        - "2020-08-10"
      # if backfill is enabled, historical daily chart data for all symbols in the target exchanges
      # are aggregated using Xignite API (=GetQuotesRange endpoint) and stored to "{symbol}/{timeframe}/OHLCV" bucket.
      backfill:
        enabled: true
        since: "2008-01-01"
        timeframe: "1D"
      # In addition to the daily-chart backfill above,
      # Xignite Feeder can feed 5-minute chart data of the target symbols for the past X business days. The data is stored to {symbol}/{timeframe}/OHLCV bucket (e.g. "1400/5Min/OHLCV" )
      recentBackfill:
        enabled: false
        days: 7 # Xignite Feeder feeds the data for {days} business days
        timeframe: "5Min"

Build

Run

DaitonoMacBook-puro:marketstore[xignitefeeder]$ pwd
/Users/dakimura/go/src/github.com/dakimura/marketstore

DaitonoMacBook-puro:marketstore[xignitefeeder]$ make plugins
(...)
go build -o /Users/dakimura/go/bin/iex.so -buildmode=plugin .
/Library/Developer/CommandLineTools/usr/bin/make -C contrib/xignitefeeder
go build -o /Users/dakimura/go/bin/xignitefeeder.so -buildmode=plugin .
go: finding github.com/modern-go/concurrent latest

DaitonoMacBook-puro:marketstore[xignitefeeder]$ ./marketstore start
{"level":"info","timestamp":"2019-05-16T10:14:03.586+0900","msg":"Running single threaded"}
(...)
{"level":"debug","timestamp":"2019-05-16T10:14:27.043+0900","msg":"[Xignite API] Delay(sec) in GetQuotes response= 0.297678"}
{"level":"debug","timestamp":"2019-05-16T10:14:27.308+0900","msg":"[Xignite API] request url=https://api.marketdata-cloud.quick-co.jp/QUICKEquityRealTime.json/GetQuotes"}
{"level":"debug","timestamp":"2019-05-16T10:14:27.437+0900","msg":"Data has been saved to marketstore successfully."}
{"level":"debug","timestamp":"2019-05-16T10:14:27.993+0900","msg":"[Xignite API] Delay(sec) in GetQuotes response= 0.240362"}
{"level":"debug","timestamp":"2019-05-16T10:14:28.310+0900","msg":"[Xignite API] request url=https://api.marketdata-cloud.quick-co.jp/QUICKEquityRealTime.json/GetQuotes"}
{"level":"debug","timestamp":"2019-05-16T10:14:28.402+0900","msg":"Data has been saved to marketstore successfully."}

Check the stored data

DaitonoMacBook-puro:marketstore[xignitefeeder]$ ./marketstore connect --url localhost:5993
{"level":"info","timestamp":"2019-05-16T10:18:43.751+0900","msg":"Running single threaded"}
Connected to remote instance at: http://localhost:5993
Type `\help` to see command options
» \show 7203/1Sec/TICK 1970-01-01
=============================  ==========  ==========  
                        Epoch  Ask         Bid         
=============================  ==========  ==========  
2019-05-15 09:49:47 +0000 UTC    6496        6495        
2019-05-15 09:49:52 +0000 UTC    6496        6495        
2019-05-15 09:49:53 +0000 UTC    6496        6495        
2019-05-15 09:49:54 +0000 UTC    6496        6495        
2019-05-15 09:49:56 +0000 UTC    6496        6495        
2019-05-15 09:49:57 +0000 UTC    6496        6495        
2019-05-15 09:49:58 +0000 UTC    6496        6495 
(...)

# to get the backfill data...
» \show 7203/1D/OHLCV 1970-01-01
(...)