fakerealtime

README

Fake Real Time example

Summary

This example uses a log-style table in an approximation of the "fake real time" system used at Friendfeed. Two tables are used: a messages table stores the complete data for all messages organized by channel, and a global updates table stores metadata about recently-updated channels.

Running

Run against an existing cockroach node or cluster.

Insecure node or cluster

# Launch your node or cluster in insecure mode (with --insecure passed to cockroach).
# Find a reachable address: [mycockroach:26257].
# Run the example with:
./fakerealtime postgres://root@mycockroach:26257?sslmode=disable

Secure node or cluster

# Launch your node or cluster in secure mode with certificates in [mycertsdir]
# Find a reachable address:[mycockroach:26257].
# Run the example with:
./fakerealtime "postgres://root@mycockroach:26257?sslcert=mycertsdir/root.client.crt&sslkey=mycertsdir/root.client.key"

Documentation

Overview

This example uses a log-style table in an approximation of the "fake real time" system used at Friendfeed. Two tables are used: a `messages` table stores the complete data for all messages organized by channel, and a global `updates` table stores metadata about recently-updated channels.

The example currently runs a number of writers, which update both tables transactionally. Future work includes implementing a reader which will effectively tail the `updates` table to learn when it needs to query a channel's messages from the `messages` table.

The implementation currently guarantees that `update_ids` are strictly monotonic, which is extremely expensive under contention (two writer threads will fight with each other for 20 seconds or more before making progress).

One alternate solution is to relax the monotonicity constraint (by using a timestamp instead of a sequential ID), at the expense of making the reader more complicated (since it would need to re-scan records it had already read to see if anything "in the past" committed since it last read). We would still need to put some sort of bound on the reader's scans; any updates that went longer than this without being committed would go unnoticed.

Another alternative is to reduce contention by adding a small random number (in range (0, N)) to the beginning of the `updates` table primary key. This reduces contention but makes the reader do N queries to see all updates. This is effectively what Friendfeed did, since there was one `updates` table per MySQL shard.