gazette

package
v2.0.209+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 5, 2019 License: MIT Imports: 40 Imported by: 30

Documentation

Overview

Package gazette contains server components tied to the service lifetime,
and clients.

Code generated by mockery v1.0.0

Index

Examples

Constants

View Source
const (
	PersisterLocksPrefix = "persister_locks/"
	PersisterLocksRoot   = ServiceRoot + "/" + PersisterLocksPrefix
)
View Source
const (
	CommitDeltaHeader          = "X-Commit-Delta"
	FragmentLastModifiedHeader = "X-Fragment-Last-Modified"
	FragmentLocationHeader     = "X-Fragment-Location"
	FragmentNameHeader         = "X-Fragment-Name"
	RouteTokenHeader           = "X-Route-Token"
	WriteHeadHeader            = "X-Write-Head"

	ReplicateClientIdlePoolSize = 6
)
View Source
const ServiceRoot = "/gazette/cluster"

Variables

This section is empty.

Functions

func MakeHttpTransport

func MakeHttpTransport() *http.Transport

If you want to use your own |http.Transport| with Gazette, start with this one.

Types

type AppendOpHandler

type AppendOpHandler interface {
	Append(journal.AppendOp)
}

type CachedURL

type CachedURL struct {
	// Accessible, absolute URL of this endpoint.
	Base string
	// contains filtered or unexported fields
}

A parsing DNS caching layer on a string URL.

func (*CachedURL) InvalidateResolution

func (ep *CachedURL) InvalidateResolution()

func (*CachedURL) NewHTTPRequest

func (ep *CachedURL) NewHTTPRequest(method, uri string, body io.Reader) (
	*http.Request, error)

func (*CachedURL) ResolveURL

func (ep *CachedURL) ResolveURL() (*url.URL, error)

func (*CachedURL) URL

func (ep *CachedURL) URL() (*url.URL, error)

type Client

type Client struct {
	// contains filtered or unexported fields
}

func NewClient

func NewClient(endpoint string) (*Client, error)

NewClient returns a new Client. To export metrics, register the prometheus.Collector instances in metrics.GazetteClientCollectors().

Example
package main

import (
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
	"github.com/sirupsen/logrus"

	"github.com/LiveRamp/gazette/pkg/gazette"
	"github.com/LiveRamp/gazette/pkg/journal"
	"github.com/LiveRamp/gazette/pkg/metrics"
)

func main() {
	// Register collectors and attach handler to expose metrics to Prometheus.
	prometheus.MustRegister(metrics.GazetteClientCollectors()...)
	http.Handle("/metrics", promhttp.Handler())

	// Get a client.
	client, err := gazette.NewClient("192.168.0.1:8081")
	if err != nil {
		logrus.WithField("err", err).Fatal("Failed to connect to gazette")
	}

	// Use client.
	client.Get(journal.ReadArgs{Journal: "a/journal", Offset: 1234})
}
Output:

func NewClientWithHttpClient

func NewClientWithHttpClient(endpoint string, hc *http.Client) (*Client, error)

func (*Client) Create

func (c *Client) Create(name journal.Name) error

Creates the Journal of the given name.

func (*Client) Do

func (c *Client) Do(request *http.Request) (*http.Response, error)

Thin layer upon http.Do(), which manages re-writes from and update to the Client.locationCache. Specifically, request.Path is mapped into a previously- stored Location re-write. If none is available, the request is re-written to reference the default endpoint. Cache entries are updated on successful redirect or response with a Location: header. On error, cache entries are expunged (eg, future requests are performed against the default endpoint).

func (*Client) FragmentBeforeTime

func (c *Client) FragmentBeforeTime(name journal.Name, t time.Time) (journal.Fragment, error)

Returns the |Fragment| whose Modified time is closest to but prior to the given |t|. Can return a zeroed Fragment structure, if no fragment matches.

func (*Client) FragmentsInRange

func (c *Client) FragmentsInRange(name journal.Name, minOff, maxOff int64) ([]journal.Fragment, error)

Returns a list of |Fragment|s that service the given offset range in |journal|.

func (*Client) Get

func (*Client) GetDirect

func (c *Client) GetDirect(args journal.ReadArgs) (journal.ReadResult, io.ReadCloser)

func (*Client) Head

func (c *Client) Head(args journal.ReadArgs) (journal.ReadResult, *url.URL)

func (*Client) Put

Performs a Gazette PUT operation, which appends content to the named journal. Put panics if |args.Content| does not implement io.ReadSeeker.

type CreateAPI

type CreateAPI struct {
	// contains filtered or unexported fields
}

API for creation of a new Journal. In particular, CreateAPI creates an Etcd item directory for the Journal under Gazette's consensus.Allocator root and responds to the client when the Journal is ready for transactions.

func NewCreateAPI

func NewCreateAPI(cfs cloudstore.FileSystem, keysAPI etcd.KeysAPI,
	requiredReplicas int) *CreateAPI

func (*CreateAPI) Create

func (h *CreateAPI) Create(w http.ResponseWriter, r *http.Request)

func (*CreateAPI) Register

func (h *CreateAPI) Register(router *mux.Router)

type JournalReplica

type JournalReplica interface {
	AppendOpHandler
	ReadOpHandler
	ReplicateOpHandler
	Shutdown()
	StartBrokeringWithPeers(journal.RouteToken, []journal.Replicator)
	StartReplicating(journal.RouteToken)
}

See journal.Replica.

type Persister

type Persister struct {
	// contains filtered or unexported fields
}

func NewPersister

func NewPersister(directory string, cfs cloudstore.FileSystem,
	keysAPI etcd.KeysAPI, routeKey string) *Persister

func (*Persister) IsShuttingDown

func (p *Persister) IsShuttingDown() bool

func (*Persister) Persist

func (p *Persister) Persist(fragment journal.Fragment)

func (*Persister) StartPersisting

func (p *Persister) StartPersisting() *Persister

func (*Persister) Stop

func (p *Persister) Stop()

func (*Persister) String

func (p *Persister) String() string

Note: This String() implementation is primarily for the benefit of expvar, which expects the string to be a serialized JSON object.

type ReadAPI

type ReadAPI struct {
	// contains filtered or unexported fields
}

func NewReadAPI

func NewReadAPI(handler ReadOpHandler, cfs cloudstore.FileSystem) *ReadAPI

func (*ReadAPI) Head

func (h *ReadAPI) Head(w http.ResponseWriter, r *http.Request)

func (*ReadAPI) Read

func (h *ReadAPI) Read(w http.ResponseWriter, r *http.Request)

func (*ReadAPI) Register

func (h *ReadAPI) Register(router *mux.Router)

type ReadOpHandler

type ReadOpHandler interface {
	Read(journal.ReadOp)
}

type ReplicaFactory

type ReplicaFactory func(journal.Name) JournalReplica

Builds a JournalReplica instance with the given journal.Name.

type ReplicateAPI

type ReplicateAPI struct {
	// contains filtered or unexported fields
}

func NewReplicateAPI

func NewReplicateAPI(handler ReplicateOpHandler) *ReplicateAPI

func (*ReplicateAPI) Register

func (h *ReplicateAPI) Register(router *mux.Router)

func (*ReplicateAPI) Replicate

func (h *ReplicateAPI) Replicate(w http.ResponseWriter, r *http.Request)

type ReplicateClient

type ReplicateClient struct {
	// contains filtered or unexported fields
}

func NewReplicateClient

func NewReplicateClient(ep *CachedURL) ReplicateClient

func (ReplicateClient) Replicate

func (c ReplicateClient) Replicate(op journal.ReplicateOp)

type ReplicateOpHandler

type ReplicateOpHandler interface {
	Replicate(journal.ReplicateOp)
}

type Router

type Router struct {
	// contains filtered or unexported fields
}

Routes and dispatches Read, Append, and Replicate operations to a collection of responsible JournalReplicas.

func NewRouter

func NewRouter(factory ReplicaFactory) *Router

func (*Router) Append

func (r *Router) Append(op journal.AppendOp)

func (*Router) BrokeredJournals

func (r *Router) BrokeredJournals() []journal.Name

Returns the set of Journals which are brokered by this Router.

func (*Router) HasServedAppend

func (r *Router) HasServedAppend(name journal.Name) bool

Returns whether |name| is both locally brokered and has successfully served an Append operation under the current route topology. This is an important indicator for consistency, as a successful Append ensures that all replicas reached agreement on the route token & write head during the transaction.

func (*Router) Read

func (r *Router) Read(op journal.ReadOp)

func (*Router) Replicate

func (r *Router) Replicate(op journal.ReplicateOp)

func (*Router) ReplicatedJournals

func (r *Router) ReplicatedJournals() []journal.Name

Returns the set of Journals which are replicated by this Router.

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

func NewRunner

func NewRunner(client etcd.Client, localRouteKey string, replicaCount int, router *Router) *Runner

func (*Runner) FixedItems

func (r *Runner) FixedItems() []string

consumer.Allocator implementation.

func (*Runner) InstanceKey

func (r *Runner) InstanceKey() string

func (*Runner) ItemIsReadyForPromotion

func (r *Runner) ItemIsReadyForPromotion(item, state string) bool

func (*Runner) ItemRoute

func (r *Runner) ItemRoute(item string, route consensus.Route, index int, tree *etcd.Node)

func (*Runner) ItemState

func (r *Runner) ItemState(item string) string

func (*Runner) KeysAPI

func (r *Runner) KeysAPI() etcd.KeysAPI

func (*Runner) PathRoot

func (r *Runner) PathRoot() string

func (*Runner) Replicas

func (r *Runner) Replicas() int

func (*Runner) Run

func (r *Runner) Run() error

type WriteAPI

type WriteAPI struct {
	// contains filtered or unexported fields
}

func NewWriteAPI

func NewWriteAPI(handler AppendOpHandler) *WriteAPI

func (*WriteAPI) Register

func (h *WriteAPI) Register(router *mux.Router)

func (*WriteAPI) Write

func (h *WriteAPI) Write(w http.ResponseWriter, r *http.Request)

type WriteService

type WriteService struct {
	// contains filtered or unexported fields
}

WriteService wraps a Client to provide asynchronous batching and automatic retries of writes to Gazette journals. Writes to each journal are spooled to local disk (and never memory), so back-pressure from slow or down brokers does not affect busy writers (at least, until disk runs out). Writes are retried indefinitely, until aknowledged by a broker.

func NewWriteService

func NewWriteService(client *Client) *WriteService

func (*WriteService) ReadFrom

func (c *WriteService) ReadFrom(name journal.Name, r io.Reader) (*journal.AsyncAppend, error)

Appends |r|'s content to |journal|, by reading until io.EOF. Either all of |r| is written, or none of it is. Returns an AsyncAppend which is resolved when the write has been fully committed.

func (*WriteService) SetConcurrency

func (c *WriteService) SetConcurrency(concurrency int)

func (*WriteService) Start

func (c *WriteService) Start()

Begins the write service loop. Be sure to invoke Stop() prior to process exit, to ensure that all pending writes have been flushed.

func (*WriteService) Stop

func (c *WriteService) Stop()

Stops the write service loop. Returns only after all writes have completed.

func (*WriteService) Write

func (c *WriteService) Write(name journal.Name, buf []byte) (*journal.AsyncAppend, error)

Appends |buffer| to |journal|. Either all of |buffer| is written, or none of it is. Returns a AsyncAppendwhich is resolved when the write has been fully committed.

func (*WriteService) WriterFor

func (c *WriteService) WriterFor(name journal.Name, sync bool) io.Writer

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL