README
¶
SHARKSTORE
SharkStore is a distributed, persistent key-value store, whose database layer is based on the rocksdb and the replication works are based on raft.
Modules
- master-server(golang) -- metadata server
- data-server(c++) -- database server
- gateway-server(golang) -- proxy
- console(golang) -- web administration
Directory
|-- README.md
|-- console web administration tool
|-- data-server business data storage service, where data is stored
|-- glide.yaml
|-- master-server metadata service, where you get metadata
|-- model
|-- pkg-go common module
|-- proxy
|-- gateway-server sql/restful proxy
Architecture Design
See arch.md .
Install
See INSTALL.md .
Performance
We created a table named 'metric' witch 12 columns on 3 data servers with NVMe disk, and each of the columns is integer type. The first four columns(named salt, key, host, ts) are made a unified index together as the key. On the other hand, the table is pre-splitted to 100 ranges with 3 replicas running on raft.
-
Batch Insert
100-row records inserted each time, 61,000,000,000 records inserted totally.
average response time: 54ms, best tps: 800,000
-
Select
-
select one record by salt+key+host+ts
average response time: 5ms
-
select 10-100 records by salt+key+host
average response time: 50ms(before caching) VS 8ms(after caching)
-
Features
-
SQL syntax compatible and restful api supported
Users can access by sql client directly, and also by restful api. -
Dynamic table scheme
Table columns is allowed to be added or renamed dynamically. -
Pre-sharding
Pre-sharding is supported when creating a new table. -
Globally sorted data
User can scan tables by primary key globally. -
Strong consistency.
Strong consistency is guaranteed by the data replication works on raft group. -
Online scalability | Auto failover | Auto rebalance | Auto schedule
-
NVMe+SPDK Supported
Under testing...
License
Under the Apache 2.0 license. See the LICENSE file for details.
Directories
¶
| Path | Synopsis |
|---|---|
|
console
|
|
|
models
*
|
* |
|
routers
*
|
* |
|
service
*
|
* |
|
master-server
|
|
|
pkg/alarmpb
Package alarmpb is a generated protocol buffer package.
|
Package alarmpb is a generated protocol buffer package. |
|
pkg/aspb
Package aspb is a generated protocol buffer package.
|
Package aspb is a generated protocol buffer package. |
|
pkg/errorpb
Package errorpb is a generated protocol buffer package.
|
Package errorpb is a generated protocol buffer package. |
|
pkg/eventpb
Package eventpb is a generated protocol buffer package.
|
Package eventpb is a generated protocol buffer package. |
|
pkg/funcpb
Package funcpb is a generated protocol buffer package.
|
Package funcpb is a generated protocol buffer package. |
|
pkg/kvrpcpb
Package kvrpcpb is a generated protocol buffer package.
|
Package kvrpcpb is a generated protocol buffer package. |
|
pkg/lockpb
Package lockrpcpb is a generated protocol buffer package.
|
Package lockrpcpb is a generated protocol buffer package. |
|
pkg/metapb
Package metapb is a generated protocol buffer package.
|
Package metapb is a generated protocol buffer package. |
|
pkg/ms_raftcmdpb
Package ms_raftcmdpb is a generated protocol buffer package.
|
Package ms_raftcmdpb is a generated protocol buffer package. |
|
pkg/mspb
Package mspb is a generated protocol buffer package.
|
Package mspb is a generated protocol buffer package. |
|
pkg/raft_cmdpb
Package raft_cmdpb is a generated protocol buffer package.
|
Package raft_cmdpb is a generated protocol buffer package. |
|
pkg/redispb
Package redispb is a generated protocol buffer package.
|
Package redispb is a generated protocol buffer package. |
|
pkg/schpb
Package schpb is a generated protocol buffer package.
|
Package schpb is a generated protocol buffer package. |
|
pkg/statspb
Package statspb is a generated protocol buffer package.
|
Package statspb is a generated protocol buffer package. |
|
pkg/taskpb
Package taskpb is a generated protocol buffer package.
|
Package taskpb is a generated protocol buffer package. |
|
pkg/timestamp
Package timestamp is a generated protocol buffer package.
|
Package timestamp is a generated protocol buffer package. |
|
pkg-go
|
|
|
proxy
|
|
|
gateway-server/mysql
Package query provides functions to transform queries.
|
Package query provides functions to transform queries. |
|
gateway-server/sqltypes
Package sqltypes implements interfaces and types that represent SQL values.
|
Package sqltypes implements interfaces and types that represent SQL values. |
|
boom/boomer
Package boomer provides commands to run load tests and display results.
|
Package boomer provides commands to run load tests and display results. |
|
client-ds
Command hey is an HTTP load generator.
|
Command hey is an HTTP load generator. |
|
client-ds/requester
Package requester provides commands to run load tests and display results.
|
Package requester provides commands to run load tests and display results. |
|
apd
Package apd implements arbitrary-precision decimals.
|
Package apd implements arbitrary-precision decimals. |
|
assert
Package assert contains functions for making assertions in unit tests From github.com/docker/docker/pkg/testutil/assert
|
Package assert contains functions for making assertions in unit tests From github.com/docker/docker/pkg/testutil/assert |
|
deepcopy
deepcopy deep copies maps, slices, etc.
|
deepcopy deep copies maps, slices, etc. |
|
hlc
Package hlc implements the Hybrid Logical Clock outlined in "Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases", available online at http://www.cse.buffalo.edu/tech-reports/2014-04.pdf.
|
Package hlc implements the Hybrid Logical Clock outlined in "Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases", available online at http://www.cse.buffalo.edu/tech-reports/2014-04.pdf. |
|
structure/maps/treemap
Implements a map backed by llrb tree.
|
Implements a map backed by llrb tree. |
|
structure/trees/llrbtree
Implements Left-Leaning Red Black trees.
|
Implements Left-Leaning Red Black trees. |