README
¶
Overview
Gazette is infrastructure for building streaming platforms: platforms composed of loosely coupled services, built and operated by distinct teams, managing and serving large amounts of state, but all communicating continuously through a common catalog of streamed data. It features a lightweight container & cloud-native architecture, high availability, and integrates elegantly with existing batch workflows.
Gazette consists of a broker service serving journals, a byte-oriented and append-only resource resembling a file, and a consumers library for building rich streaming applications in Go.
How It's Different
Gazette provides low-latency, durable, ordered publish/subscribe services, while also serving as the system-of-record for all historical data which has passed through those streams, no matter how old or large in volume.
It features powerful labels and selectors mechanisms for tagging and querying over event streams, similar to those of Kubernetes.
It minimizes operational risk by delegating both storage and high-volume replay to elastic BLOB stores like S3. Brokers themselves are ephemeral and disposable. The cluster scales and recovers from faults in seconds.
It seeks to be cost-efficient by best utilizing cloud pricing structures and being careful to minimize inter-zone data transfers.
It features a rich library for building fault-tolerant, highly available, highly stateful streaming applications in Go, as well as a "batteries included" command- line tool which makes quick work of integrating existing applications.
Where to Start
- Brokers: A Tutorial Introduction is a walk-through of key broker features and concepts.
- Design Goals (and Non-Goals)
Coming soon:
- Broker architecture briefs
- Introduction to the consumers library and using it to build applications.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package allocator implements a distributed algorithm for assigning a number of "Items" across a number of "Members", where each Member runs an instance of the Allocator.
|
Package allocator implements a distributed algorithm for assigning a number of "Items" across a number of "Members", where each Member runs an instance of the Allocator. |
|
push_relabel
Package push_relabel implements a greedy variant of the push/relabel algorithm.
|
Package push_relabel implements a greedy variant of the push/relabel algorithm. |
|
Package broker implements the broker runtime and protocol.JournalServer APIs (Read, Append, Replicate, List, Apply).
|
Package broker implements the broker runtime and protocol.JournalServer APIs (Read, Append, Replicate, List, Apply). |
|
client
Package client provides implementation for clients of the Journal API.
|
Package client provides implementation for clients of the Journal API. |
|
fragment
Package fragment is concerned with the mapping of journal offsets to protocol.Fragments, to corresponding local or remote journal content.
|
Package fragment is concerned with the mapping of journal offsets to protocol.Fragments, to corresponding local or remote journal content. |
|
journalspace
Package journalspace provides mechanisms for mapping a collection of JournalSpecs into a minimally-described hierarchical structure, and for mapping back again.
|
Package journalspace provides mechanisms for mapping a collection of JournalSpecs into a minimally-described hierarchical structure, and for mapping back again. |
|
protocol
Package protocol defines the core broker datamodel, validation behaviors, and gRPC APIs which are shared across clients and broker servers.
|
Package protocol defines the core broker datamodel, validation behaviors, and gRPC APIs which are shared across clients and broker servers. |
|
Package brokertest provides utilities for testing components requiring a live Gazette broker.
|
Package brokertest provides utilities for testing components requiring a live Gazette broker. |
|
cmd
|
|
|
gazctl/editor
Package editor provides utilities for editing and re-editing text files in an editor until the changes are accepted or the user aborts the attempt.
|
Package editor provides utilities for editing and re-editing text files in an editor until the changes are accepted or the user aborts the attempt. |
|
Package consumer is a library for distributed, stateful consumption of Gazette journals.
|
Package consumer is a library for distributed, stateful consumption of Gazette journals. |
|
protocol
Package protocol defines the consumer datamodel, validation behaviors, and gRPC APIs which are shared across clients and consumer application servers.
|
Package protocol defines the consumer datamodel, validation behaviors, and gRPC APIs which are shared across clients and consumer application servers. |
|
recoverylog
Package recoverylog specifies a finite state machine for recording and replaying observed filesystem operations into a Gazette journal.
|
Package recoverylog specifies a finite state machine for recording and replaying observed filesystem operations into a Gazette journal. |
|
shardspace
Package shardspace provides mechanisms for mapping a collection of ShardSpecs into a minimally-described, semi hierarchical structure, and for mapping back again.
|
Package shardspace provides mechanisms for mapping a collection of ShardSpecs into a minimally-described, semi hierarchical structure, and for mapping back again. |
|
store-rocksdb
Package store_rocksdb implements the consumer.Store interface via an embedded RocksDB instance.
|
Package store_rocksdb implements the consumer.Store interface via an embedded RocksDB instance. |
|
Package consumertest provides utilities for in-process unit testing of Gazette consumer applications.
|
Package consumertest provides utilities for in-process unit testing of Gazette consumer applications. |
|
Package etcdtest provides test support for obtaining a client to an Etcd server.
|
Package etcdtest provides test support for obtaining a client to an Etcd server. |
|
examples
|
|
|
stream-sum
Package stream_sum is an example application consisting of three stages: 1) A `chunker` job randomly generates a number of unique "streams", with stream content emitted across a number of interleaved data chunks.
|
Package stream_sum is an example application consisting of three stages: 1) A `chunker` job randomly generates a number of unique "streams", with stream content emitted across a number of interleaved data chunks. |
|
stream-sum/summer
Package summer runs the stream_sum.Summer consumer.
|
Package summer runs the stream_sum.Summer consumer. |
|
word-count
Package word_count is an example application which provides a gRPC API for publishing texts and querying running counts of NGrams extracted from previously published texts.
|
Package word_count is an example application which provides a gRPC API for publishing texts and querying running counts of NGrams extracted from previously published texts. |
|
word-count/counter
Package counter runs the word_count.Counter consumer.
|
Package counter runs the word_count.Counter consumer. |
|
Package keyspace implements an efficient mechanism to mirror a decoded Etcd key/value space into a local KeySpace, which may be kept updated via a long- lived Watch operation.
|
Package keyspace implements an efficient mechanism to mirror a decoded Etcd key/value space into a local KeySpace, which may be kept updated via a long- lived Watch operation. |
|
Package labels defines well-known label names and values of Gazette.
|
Package labels defines well-known label names and values of Gazette. |
|
runconsumer
Package runconsumer extends consumer.Application with support for configuration and application initialization.
|
Package runconsumer extends consumer.Application with support for configuration and application initialization. |
|
Package message defines a Message interface and Envelope type, and provides a Framing interface and implementations.
|
Package message defines a Message interface and Envelope type, and provides a Framing interface and implementations. |
Click to show internal directories.
Click to hide internal directories.