README
¶
Vitess
Vitess is a set of servers and tools meant to facilitate scaling of MySQL databases for the web. It's currently used as a fundamental component of YouTube's MySQL infrastructure.
sougou presented Vitess at Fosdem '14 in the go devroom. Here are the slides, and here is the video.
Vitess is currently not ready for unsupervised use yet. There's good deal of documentation missing. Also, some functionality is under development and some APIs are still changing. But if you feel adventurous, you're welcome to try it. If you run into issues, please subscribe and post to our mailing list. We'll do our best to help you.
- Vision
- Concepts
- General Concepts
- Replication Graph
- Serving graph
- Tools
- Getting Started
- Operations
- Preparing for production
- Reparenting
- Resharding
- Schema management
- Zookeeper data
vttablet
TODO: Move this content to its own markdown.
Smart middleware sitting in front of MySQL and serving clients requests.
- Connection pooling.
- SQL parser: Although very close, the vtocc SQL parser is not SQL-92 compliant. It has left out constructs that are deemed uncommon or OLTP-unfriendly. It should, however, allow most queries used by a well-behaved web application.
- Query rewrite and sanitation (adding limits, avoiding non-deterministic updates).
- Query consolidation: reuse the results of an in-flight query to any subsequent requests that were received while the query was still executing.
- Rowcache: the mysql buffer cache is optimized for range scans over indices and tables. Unfortunately, it’s not good for random access by primary key. The rowcache will instead maintain a row based cache (using memcached as its backend) and keep it consistent by fielding all DMLs that could potentially affect them.
- Update stream: A server that streams the list of rows that are changing in the database, which can be used as a mechanism to continuously export the data to another data store.
- Integrated query killer for queries that take too long to return data.
- Discard idle backend connections to avoid offline db errors.
- Transaction management: Ability to limit the number of concurrent transactions and manage deadlines.
License
Unless otherwise noted, the vitess source files are distributed under the BSD-style license found in the LICENSE file.
Directories
¶
| Path | Synopsis |
|---|---|
|
data
|
|
|
go
|
|
|
bufio2
Package bufio implements buffered I/O. It wraps an io.Reader or io.Writer object, creating another object (Reader or AsyncWriter) that also implements the interface but provides buffering and some help for textual I/O.
|
Package bufio implements buffered I/O. It wraps an io.Reader or io.Writer object, creating another object (Reader or AsyncWriter) that also implements the interface but provides buffering and some help for textual I/O. |
|
cache
The implementation borrows heavily from SmallLRUCache (originally by Nathan Schrenk).
|
The implementation borrows heavily from SmallLRUCache (originally by Nathan Schrenk). |
|
cmd/normalizer
normalizer takes a file of sql statements as input and converts the statements into normalized sql statements with bind variables.
|
normalizer takes a file of sql statements as input and converts the statements into normalized sql statements with bind variables. |
|
cmd/vtprimecache
vtprimecache is a standalone version of primecache
|
vtprimecache is a standalone version of primecache |
|
cmd/vttablet
vt tablet server: Serves queries and performs housekeeping jobs.
|
vt tablet server: Serves queries and performs housekeeping jobs. |
|
executil
Only build on Linux, since the use of procfs is platform specific.
|
Only build on Linux, since the use of procfs is platform specific. |
|
fileutil
filutil contains a few utility functions related to files and path
|
filutil contains a few utility functions related to files and path |
|
flagutil
Package flagutil contains flags that parse string lists and string maps.
|
Package flagutil contains flags that parse string lists and string maps. |
|
netutil
This packages contains a few utility functions for network related functions.
|
This packages contains a few utility functions for network related functions. |
|
pools
Package pools provides functionality to manage and reuse resources like connections.
|
Package pools provides functionality to manage and reuse resources like connections. |
|
proc
Package proc allows you to configure servers to be restarted with negligible downtime.
|
Package proc allows you to configure servers to be restarted with negligible downtime. |
|
rpcplus
Package rpc provides access to the exported methods of an object across a network or other I/O connection.
|
Package rpc provides access to the exported methods of an object across a network or other I/O connection. |
|
rpcplus/jsonrpc
Package jsonrpc implements a JSON-RPC ClientCodec and ServerCodec for the rpc package.
|
Package jsonrpc implements a JSON-RPC ClientCodec and ServerCodec for the rpc package. |
|
rpcplus/pbrpc
Package pbrpc implements a ClientCodec and ServerCodec for the rpc package using proto.
|
Package pbrpc implements a ClientCodec and ServerCodec for the rpc package using proto. |
|
sqltypes
Package sqltypes implements interfaces and types that represent SQL values.
|
Package sqltypes implements interfaces and types that represent SQL values. |
|
stats
Package stats is a wrapper for expvar.
|
Package stats is a wrapper for expvar. |
|
timer
Package timer provides timer functionality that can be controlled by the user.
|
Package timer provides timer functionality that can be controlled by the user. |
|
vt/client2/tablet
API compliant to the requirements of database/sql Open expects name to be "hostname:port/keyspace/shard" For query arguments, we assume place-holders in the query string in the form of :v0, :v1, etc.
|
API compliant to the requirements of database/sql Open expects name to be "hostname:port/keyspace/shard" For query arguments, we assume place-holders in the query string in the form of :v0, :v1, etc. |
|
vt/dbconfigs
Package dbconfigs is reusable by vt tools to load the db configs file.
|
Package dbconfigs is reusable by vt tools to load the db configs file. |
|
vt/primecache
primecache primes the MySQL buffer cache with the rows that are going to be modified by the replication stream.
|
primecache primes the MySQL buffer cache with the rows that are going to be modified by the replication stream. |
|
vt/topo/helpers
helpers package contains a few utility classes to handle topo.Server objects, and transitions from one topo implementation to another.
|
helpers package contains a few utility classes to handle topo.Server objects, and transitions from one topo implementation to another. |
|
vt/topo/test
package test contains utilities to test topo.Server implementations.
|
package test contains utilities to test topo.Server implementations. |
|
vt/topo/test/faketopo
faketopo contains utitlities for tests that have to interact with a Vitess topology.
|
faketopo contains utitlities for tests that have to interact with a Vitess topology. |
|
vt/topotools
topotools contains high level functions based on vt/topo and vt/actionnode.
|
topotools contains high level functions based on vt/topo and vt/actionnode. |
|
vt/vtgate
Package vtgate provides query routing rpc services for vttablets.
|
Package vtgate provides query routing rpc services for vttablets. |
|
vt/vtgate/gorpcvtgateservice
Package gorpcvtgateservice provides to go rpc glue for vtgate
|
Package gorpcvtgateservice provides to go rpc glue for vtgate |
|
vt/vttablet
Package vttablet contains the meat of the vttablet binary.
|
Package vttablet contains the meat of the vttablet binary. |
|
vt/worker
'worker' package contains the framework, utility methods and core functions for long running actions.
|
'worker' package contains the framework, utility methods and core functions for long running actions. |
|
vt/wrangler
wrangler contains the Wrangler object to manage complex topology actions.
|
wrangler contains the Wrangler object to manage complex topology actions. |
|
zk
Emulate a "global" namespace across n zk quorums.
|
Emulate a "global" namespace across n zk quorums. |
|
zk/fakezk
Package fakezk is a pretty complete mock implementation of a Zookeper connection (see go/zk/zk.Conn).
|
Package fakezk is a pretty complete mock implementation of a Zookeper connection (see go/zk/zk.Conn). |
|
zk/zkns/pdns
To be used with PowerDNS (pdns) as a "pipe backend" CoProcess.
|
To be used with PowerDNS (pdns) as a "pipe backend" CoProcess. |
|
zk/zkocc
cache for zkocc
|
cache for zkocc |
|
test
|
|
|
third_party
|
|
|
go/launchpad.net/gozk/zookeeper
gozk - ZooKeeper support for the Go language
|
gozk - ZooKeeper support for the Go language |
Click to show internal directories.
Click to hide internal directories.