README
¶
Vitess
Vitess is a set of servers and tools meant to facilitate scaling of MySQL databases for the web. It's been developed since 2011, and is currently used as a fundamental component of YouTube's MySQL infrastructure, serving thousands of QPS (per server). If you want to find out whether Vitess is a good fit for your project, please read our helicopter overview.
There are a couple of videos from sougou that you can watch: a short intro prepared for Google I/O 2014 and a more detailed presentation from @Scale '14.
Overview
Vitess consists of a number servers, command line utilities, and a consistent metadata store. Taken together, they allow you to serve more database traffic, and add features like sharding, which normally you would have to implement in your application.
vttablet is a server that sits in front of a MySQL database, making it more robust and available in the face of high traffic. Among other things, it adds a connection pool, has a row based cache, and it rewrites SQL queries to be safer and nicer to the underlying database.
vtgate is a very light proxy that routes database traffic from your app to the right vttablet, basing on the sharding scheme, latency required, and health of the vttablets. This allows the client to be very simple, as all it needs to be concerned about is finding the closest vtgate.
The topology is a metadata store that contains information about running servers, the sharding scheme, and replication graph. It is backed by a consistent data store, like Apache ZooKeeper. The topology backends are plugin based, allowing you to write your own if ZooKeeper doesn't fit your needs. You can explore the topology through vtctld, a webserver (not shown in the diagram).
vtctl is a command line utility that allows a human or a script to easily interact with the system.
All components communicate using a lightweight RPC system based on BSON. The RPC system is plugin based, so you can easily write your own backend (at Google we use a Protocol Buffers based protocol). We provide a client implementation for three languages: Python, Go, and Java. Writing a client for your language should not be difficult, as it's a matter of implementing only a few API calls (please send us a pull request if you do!).
Trying it out
Vitess is not entirely ready for unsupervised use yet. Some functionality is still under development, APIs may change, and parts of the code are undocumented. However, if you feel adventurous, you're more than welcome to try it. We know that there are some rough edges, so please don't hesitate to reach out to us through our mailing list if you run into any issues. Warnings aside, please take a look at our Getting Started guide.
Documentation
Intro
- Helicopter overview: high level overview of Vitess that should tell you whether Vitess is for you.
- Frequently Asked Questions.
- Vision: principles guiding the design of Vitess.
Using Vitess
- Getting Started: how to set your environment to work with Vitess.
- Tools: all Vitess tools and servers.
- vttablet/vtocc: information about the most important Vitess server.
- Reparenting: performing master failover.
- Resharding: adding more shards to your cluster.
- Preparing for production (wip).
- Schema management: managing your database schema using Vitess.
Reference
License
Unless otherwise noted, the vitess source files are distributed under the BSD-style license found in the LICENSE file.
Directories
¶
| Path | Synopsis |
|---|---|
|
data
|
|
|
go
|
|
|
acl
Package acl contains functions to enforce access control lists.
|
Package acl contains functions to enforce access control lists. |
|
bson
Package bson implements encoding and decoding of BSON objects.
|
Package bson implements encoding and decoding of BSON objects. |
|
bufio2
Package bufio2 implements buffered I/O.
|
Package bufio2 implements buffered I/O. |
|
bytes2
Package bytes2 provides alternate implementations of functionality similar to go's bytes package.
|
Package bytes2 provides alternate implementations of functionality similar to go's bytes package. |
|
cache
Package cache implements a LRU cache.
|
Package cache implements a LRU cache. |
|
cgzip
Package cgzip wraps the C library for gzip.
|
Package cgzip wraps the C library for gzip. |
|
cmd/bsongen
bsongen will generate bson encoders and decoders for a given go type.
|
bsongen will generate bson encoders and decoders for a given go type. |
|
cmd/vtprimecache
vtprimecache is a standalone version of primecache
|
vtprimecache is a standalone version of primecache |
|
cmd/vttablet
vt tablet server: Serves queries and performs housekeeping jobs.
|
vt tablet server: Serves queries and performs housekeeping jobs. |
|
db
Package db defines an alternate (and simplified) db api compared to go's database/sql.
|
Package db defines an alternate (and simplified) db api compared to go's database/sql. |
|
event
Package event provides a reflect-based framework for low-frequency global dispatching of events, which are values of any arbitrary type, to a set of listener functions, which are usually registered by plugin packages during init().
|
Package event provides a reflect-based framework for low-frequency global dispatching of events, which are values of any arbitrary type, to a set of listener functions, which are usually registered by plugin packages during init(). |
|
event/syslogger
Package syslogger uses the event package to listen for any event that implements the Syslogger interface.
|
Package syslogger uses the event package to listen for any event that implements the Syslogger interface. |
|
executil
Package executil provides information about running processes.
|
Package executil provides information about running processes. |
|
exit
Package exit provides an alternative to os.Exit(int) that executes deferred functions before exiting.
|
Package exit provides an alternative to os.Exit(int) that executes deferred functions before exiting. |
|
fileutil
Package fileutil contains utility functions related to files and paths.
|
Package fileutil contains utility functions related to files and paths. |
|
flagutil
Package flagutil contains flags that parse string lists and string maps.
|
Package flagutil contains flags that parse string lists and string maps. |
|
hack
Package hack gives you some efficient functionality at the cost of breaking some Go rules.
|
Package hack gives you some efficient functionality at the cost of breaking some Go rules. |
|
history
Package history implements a circular buffer with adjacent-item deduplication.
|
Package history implements a circular buffer with adjacent-item deduplication. |
|
ioutil2
Package ioutil2 provides extra functionality along similar lines to io/ioutil.
|
Package ioutil2 provides extra functionality along similar lines to io/ioutil. |
|
jscfg
Package jscfg implements a simple API for reading and writing JSON files.
|
Package jscfg implements a simple API for reading and writing JSON files. |
|
memcache
Package memcache is a client for memcached.
|
Package memcache is a client for memcached. |
|
mysql
Package mysql wraps the C client library for MySQL.
|
Package mysql wraps the C client library for MySQL. |
|
netutil
Package netutil contains network-related utility functions.
|
Package netutil contains network-related utility functions. |
|
pools
Package pools provides functionality to manage and reuse resources like connections.
|
Package pools provides functionality to manage and reuse resources like connections. |
|
proc
Package proc allows you to configure servers to be restarted with negligible downtime.
|
Package proc allows you to configure servers to be restarted with negligible downtime. |
|
rpcplus
Package rpcplus provides access to the exported methods of an object across a network or other I/O connection.
|
Package rpcplus provides access to the exported methods of an object across a network or other I/O connection. |
|
rpcplus/jsonrpc
Package jsonrpc implements a JSON-RPC ClientCodec and ServerCodec for the rpcplus package.
|
Package jsonrpc implements a JSON-RPC ClientCodec and ServerCodec for the rpcplus package. |
|
rpcplus/pbrpc
Package pbrpc implements a ClientCodec and ServerCodec for the rpcplus package using Protocol Buffers.
|
Package pbrpc implements a ClientCodec and ServerCodec for the rpcplus package using Protocol Buffers. |
|
sqltypes
Package sqltypes implements interfaces and types that represent SQL values.
|
Package sqltypes implements interfaces and types that represent SQL values. |
|
stats
Package stats is a wrapper for expvar.
|
Package stats is a wrapper for expvar. |
|
streamlog
Package streamlog provides a non-blocking message broadcaster.
|
Package streamlog provides a non-blocking message broadcaster. |
|
sync2
Package sync2 provides extra functionality along the same lines as sync.
|
Package sync2 provides extra functionality along the same lines as sync. |
|
tb
Package tb exposes some handy traceback functionality buried in the runtime.
|
Package tb exposes some handy traceback functionality buried in the runtime. |
|
terminal
Package terminal detects whether a file descriptor is a terminal.
|
Package terminal detects whether a file descriptor is a terminal. |
|
testfiles
Package testfiles locates test files within the Vitess directory tree.
|
Package testfiles locates test files within the Vitess directory tree. |
|
timer
Package timer provides timer functionality that can be controlled by the user.
|
Package timer provides timer functionality that can be controlled by the user. |
|
umgmt
Package umgmt (micromanagment) provides a tiny server running on a unix domain socket.
|
Package umgmt (micromanagment) provides a tiny server running on a unix domain socket. |
|
vt/binlog/binlogplayer
Package binlogplayer contains the code that plays a filtered replication stream on a client database.
|
Package binlogplayer contains the code that plays a filtered replication stream on a client database. |
|
vt/client2/tablet
API compliant to the requirements of database/sql Open expects name to be "hostname:port/keyspace/shard" For query arguments, we assume place-holders in the query string in the form of :v0, :v1, etc.
|
API compliant to the requirements of database/sql Open expects name to be "hostname:port/keyspace/shard" For query arguments, we assume place-holders in the query string in the form of :v0, :v1, etc. |
|
vt/dbconfigs
Package dbconfigs is reusable by vt tools to load the db configs file.
|
Package dbconfigs is reusable by vt tools to load the db configs file. |
|
vt/dbconnpool
Package dbconnpool exposes a single DBConnection object with wrapped access to a single DB connection, and a ConnectionPool object to pool these DBConnections.
|
Package dbconnpool exposes a single DBConnection object with wrapped access to a single DB connection, and a ConnectionPool object to pool these DBConnections. |
|
vt/events
Package events defines common structures used for events dispatched from various other package.
|
Package events defines common structures used for events dispatched from various other package. |
|
vt/primecache
primecache primes the MySQL buffer cache with the rows that are going to be modified by the replication stream.
|
primecache primes the MySQL buffer cache with the rows that are going to be modified by the replication stream. |
|
vt/status
Package status defines a few useful functions for our binaries, mainly to link the status page with a vtctld instance.
|
Package status defines a few useful functions for our binaries, mainly to link the status page with a vtctld instance. |
|
vt/tabletmanager
Package agent exports the ActionAgent object.
|
Package agent exports the ActionAgent object. |
|
vt/topo/events
Package events defines the structures used for events relating directly to individual topology elements, like keyspaces, shards, and tablets.
|
Package events defines the structures used for events relating directly to individual topology elements, like keyspaces, shards, and tablets. |
|
vt/topo/helpers
helpers package contains a few utility classes to handle topo.Server objects, and transitions from one topo implementation to another.
|
helpers package contains a few utility classes to handle topo.Server objects, and transitions from one topo implementation to another. |
|
vt/topo/test
package test contains utilities to test topo.Server implementations.
|
package test contains utilities to test topo.Server implementations. |
|
vt/topo/test/faketopo
faketopo contains utitlities for tests that have to interact with a Vitess topology.
|
faketopo contains utitlities for tests that have to interact with a Vitess topology. |
|
vt/topotools
Package topotools contains high level functions based on vt/topo and vt/actionnode.
|
Package topotools contains high level functions based on vt/topo and vt/actionnode. |
|
vt/topotools/events
Package events defines the structures used for events dispatched from the wrangler package.
|
Package events defines the structures used for events dispatched from the wrangler package. |
|
vt/vtctl/gorpcproto
gorpcproto contains the Go RPC definitions of the structures used to execute remote vtctl commands.
|
gorpcproto contains the Go RPC definitions of the structures used to execute remote vtctl commands. |
|
vt/vtctl/gorpcvtctlclient
gorpcvtctlclient contains the go rpc version of the vtctl client protocol
|
gorpcvtctlclient contains the go rpc version of the vtctl client protocol |
|
vt/vtctl/gorpcvtctlserver
gorpcvtctlserver contains the Go RPC implementation of the server side of the remote execution of vtctl commands.
|
gorpcvtctlserver contains the Go RPC implementation of the server side of the remote execution of vtctl commands. |
|
vt/vtctl/vtctlclient
vtctlclient contains the generic client side of the remote vtctl protocol.
|
vtctlclient contains the generic client side of the remote vtctl protocol. |
|
vt/vtgate
Package vtgate provides query routing rpc services for vttablets.
|
Package vtgate provides query routing rpc services for vttablets. |
|
vt/vtgate/gorpcvtgateservice
Package gorpcvtgateservice provides to go rpc glue for vtgate
|
Package gorpcvtgateservice provides to go rpc glue for vtgate |
|
vt/worker
'worker' package contains the framework, utility methods and core functions for long running actions.
|
'worker' package contains the framework, utility methods and core functions for long running actions. |
|
vt/wrangler
wrangler contains the Wrangler object to manage complex topology actions.
|
wrangler contains the Wrangler object to manage complex topology actions. |
|
vt/wrangler/testlib
Package testlib contains utility methods to include in unit tests to deal with topology common tasks, liek fake tablets and action loops.
|
Package testlib contains utility methods to include in unit tests to deal with topology common tasks, liek fake tablets and action loops. |
|
zk
Emulate a "global" namespace across n zk quorums.
|
Emulate a "global" namespace across n zk quorums. |
|
zk/fakezk
Package fakezk is a pretty complete mock implementation of a Zookeper connection (see go/zk/zk.Conn).
|
Package fakezk is a pretty complete mock implementation of a Zookeper connection (see go/zk/zk.Conn). |
|
zk/zkns/pdns
To be used with PowerDNS (pdns) as a "pipe backend" CoProcess.
|
To be used with PowerDNS (pdns) as a "pipe backend" CoProcess. |
|
zk/zkocc
cache for zkocc
|
cache for zkocc |
|
java
|
|
|
test
|
|
|
third_party
|
|
|
go/launchpad.net/gozk/zookeeper
gozk - ZooKeeper support for the Go language
|
gozk - ZooKeeper support for the Go language |
Click to show internal directories.
Click to hide internal directories.