README
¶
Overview
Gazette is infrastructure for building streaming platforms: platforms composed of loosely coupled services, built and operated by distinct teams, managing and serving large amounts of state, but all communicating continuously through a common catalog of streamed data. It features a lightweight container & cloud-native architecture, high availability, and integrates elegantly with existing batch workflows.
It has served production use cases for nearly five years, with deployments scaled to millions of streamed messages per second.
Gazette consists of a broker service serving journals, a byte-oriented and append-only resource resembling a file, and a consumers framework for building streaming applications in Go.
How It's Different
Gazette provides low-latency, durable, ordered publish/subscribe services, while also serving as the system-of-record for all historical data which has passed through those streams, no matter how old or large in volume.
It delegates storage and high-volume replay to elastic BLOB stores like S3. Brokers themselves are ephemeral and disposable. The cluster scales and recovers from faults in seconds. Its architecture obviates the common need for separate batch and real-time systems by providing the capabilities and advantages of both.
It features a rich framework for building scaled, available streaming applications in Go with exactly-once semantics. Applications may process against a remote database, or may use embedded stores such as RocksDB for fast and tunable storage of keys & values, or even SQLite for full SQL support. Details like the durable replication of embedded stores, provisioning of hot standbys, request routing, and fast fail-over are managed so that users can focus on their message-driven application behaviors.
Deploy applications which efficiently crunch through months or even years of historical data, then seamlessly transition to real-time tailing reads. Serve blazing fast APIs drawing from state stores embedded within applications. Apply Gazette's "batteries included" command-line tool to make quick work of integrating existing applications.
Brokers and applications are easily operated by container platforms like Kubernetes, and offer familiar primitives such as declarative YAML specifications and a powerful labels & selectors mechanism for tagging and querying over objects. Gazette seeks to be cost-efficient by best utilizing cloud pricing structures and being careful to minimize inter-zone data transfers.
Where to Start
Project Overview Slides discuss high-level capabilities and architecture (good for the impatient).
Brokers: A Tutorial Introduction is a walk-through of key broker features and concepts.
Design Goals (and Non-Goals) discusses rationale for the project, what Gazette strives to be great at, and what Gazette avoids trying to solve.
The project features several runnable example consumer applications:
- Finding Cycles in Bike-Share Streams.
- Serving up a Real-Time Language Model.
- SHA-summing over Multiplexed File Chunks.
Build and Test the project.
Architecture Briefs:
- Exactly-once Semantics in Gazette
- Operational Considerations
- (Others forthcoming).
Directories
¶
| Path | Synopsis |
|---|---|
|
Package allocator implements a distributed algorithm for assigning a number of "Items" across a number of "Members", where each Member runs an instance of the Allocator.
|
Package allocator implements a distributed algorithm for assigning a number of "Items" across a number of "Members", where each Member runs an instance of the Allocator. |
|
push_relabel
Package push_relabel implements a greedy variant of the push/relabel algorithm.
|
Package push_relabel implements a greedy variant of the push/relabel algorithm. |
|
Package broker implements the broker runtime and protocol.JournalServer APIs (Read, Append, Replicate, List, Apply).
|
Package broker implements the broker runtime and protocol.JournalServer APIs (Read, Append, Replicate, List, Apply). |
|
client
Package client implements a Go client for interacting with the gRPC Journal service of Gazette brokers.
|
Package client implements a Go client for interacting with the gRPC Journal service of Gazette brokers. |
|
fragment
Package fragment is a broker-only package concerned with the mapping of journal offsets to protocol.Fragments, and from there to corresponding local or remote journal content.
|
Package fragment is a broker-only package concerned with the mapping of journal offsets to protocol.Fragments, and from there to corresponding local or remote journal content. |
|
journalspace
Package journalspace provides mechanisms for mapping a collection of JournalSpecs into a minimally-described hierarchical structure, and for mapping back again.
|
Package journalspace provides mechanisms for mapping a collection of JournalSpecs into a minimally-described hierarchical structure, and for mapping back again. |
|
protocol
Package protocol defines the core broker datamodel, validation behaviors, and gRPC APIs which are shared across clients and broker servers.
|
Package protocol defines the core broker datamodel, validation behaviors, and gRPC APIs which are shared across clients and broker servers. |
|
Package brokertest provides utilities for testing components requiring a live Gazette broker.
|
Package brokertest provides utilities for testing components requiring a live Gazette broker. |
|
cmd
|
|
|
gazctl/editor
Package editor provides utilities for editing and re-editing text files in an editor until the changes are accepted or the user aborts the attempt.
|
Package editor provides utilities for editing and re-editing text files in an editor until the changes are accepted or the user aborts the attempt. |
|
Package consumer is a framework for distributed, stateful consumption of Gazette journals.
|
Package consumer is a framework for distributed, stateful consumption of Gazette journals. |
|
protocol
Package protocol defines the consumer datamodel, validation behaviors, and gRPC APIs which are shared across clients and consumer application servers.
|
Package protocol defines the consumer datamodel, validation behaviors, and gRPC APIs which are shared across clients and consumer application servers. |
|
recoverylog
Package recoverylog specifies a finite state machine for recording and replaying observed filesystem operations into a Gazette journal.
|
Package recoverylog specifies a finite state machine for recording and replaying observed filesystem operations into a Gazette journal. |
|
shardspace
Package shardspace provides mechanisms for mapping a collection of ShardSpecs into a minimally-described, semi hierarchical structure, and for mapping back again.
|
Package shardspace provides mechanisms for mapping a collection of ShardSpecs into a minimally-described, semi hierarchical structure, and for mapping back again. |
|
store-rocksdb
Package store_rocksdb implements the consumer.Store interface via an embedded RocksDB instance.
|
Package store_rocksdb implements the consumer.Store interface via an embedded RocksDB instance. |
|
store-sqlite
Package store_sqlite implements the consumer.Store interface via an embedded SQLite instance.
|
Package store_sqlite implements the consumer.Store interface via an embedded SQLite instance. |
|
Package consumertest provides utilities for in-process unit testing of Gazette consumer applications.
|
Package consumertest provides utilities for in-process unit testing of Gazette consumer applications. |
|
Package etcdtest provides test support for obtaining a client to an Etcd server.
|
Package etcdtest provides test support for obtaining a client to an Etcd server. |
|
Package examples parents packages which implement example Gazette consumer applications.
|
Package examples parents packages which implement example Gazette consumer applications. |
|
bike-share
Package bike_share implements a Gazette consumer application which processes and serves streaming Citi Bike system data.
|
Package bike_share implements a Gazette consumer application which processes and serves streaming Citi Bike system data. |
|
stream-sum
Package stream_sum is an example application consisting of three stages: 1) A `chunker` job randomly generates a number of unique "streams", with stream content emitted across a number of interleaved data chunks.
|
Package stream_sum is an example application consisting of three stages: 1) A `chunker` job randomly generates a number of unique "streams", with stream content emitted across a number of interleaved data chunks. |
|
stream-sum/summer
Package summer runs the stream_sum.Summer consumer.
|
Package summer runs the stream_sum.Summer consumer. |
|
word-count
Package word_count is an example application which provides a gRPC API for publishing texts and querying running counts of NGrams extracted from previously published texts.
|
Package word_count is an example application which provides a gRPC API for publishing texts and querying running counts of NGrams extracted from previously published texts. |
|
word-count/counter
Package counter runs the word_count.Counter consumer.
|
Package counter runs the word_count.Counter consumer. |
|
Package keyspace implements an efficient mechanism to mirror a decoded Etcd key/value space into a local KeySpace, which may be kept updated via a long- lived Watch operation.
|
Package keyspace implements an efficient mechanism to mirror a decoded Etcd key/value space into a local KeySpace, which may be kept updated via a long- lived Watch operation. |
|
Package labels defines well-known label names and values of Gazette.
|
Package labels defines well-known label names and values of Gazette. |
|
runconsumer
Package runconsumer extends consumer.Application with support for configuration and application initialization.
|
Package runconsumer extends consumer.Application with support for configuration and application initialization. |
|
Package message is a client-only library which implements exactly-once message semantics atop raw, at-least-once Journal byte-streams.
|
Package message is a client-only library which implements exactly-once message semantics atop raw, at-least-once Journal byte-streams. |