gazette

package
v2.0.211+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 15, 2019 License: MIT Imports: 40 Imported by: 30

Documentation

Overview 

Package gazette contains server components tied to the service lifetime,
and clients.

Code generated by mockery v1.0.0

Index

Examples

Constants

View Source 
const (
	PersisterLocksPrefix = "persister_locks/"
	PersisterLocksRoot   = ServiceRoot + "/" + PersisterLocksPrefix
)
View Source 
const (
	CommitDeltaHeader          = "X-Commit-Delta"
	FragmentLastModifiedHeader = "X-Fragment-Last-Modified"
	FragmentLocationHeader     = "X-Fragment-Location"
	FragmentNameHeader         = "X-Fragment-Name"
	RouteTokenHeader           = "X-Route-Token"
	WriteHeadHeader            = "X-Write-Head"

	ReplicateClientIdlePoolSize = 6
)
View Source 
const ServiceRoot = "/gazette/cluster"

Variables

This section is empty.

Functions

func MakeHttpTransport 

func MakeHttpTransport() *http.Transport

If you want to use your own |http.Transport| with Gazette, start with this one.

Types

type AppendOpHandler 

type AppendOpHandler interface {
	Append(journal.AppendOp)
}

type CachedURL 

type CachedURL struct {
	// Accessible, absolute URL of this endpoint.
	Base string
	// contains filtered or unexported fields
}

A parsing DNS caching layer on a string URL.

func (*CachedURL) InvalidateResolution 

func (ep *CachedURL) InvalidateResolution()

func (*CachedURL) NewHTTPRequest 

func (ep *CachedURL) NewHTTPRequest(method, uri string, body io.Reader) (
	*http.Request, error)

func (*CachedURL) ResolveURL 

func (ep *CachedURL) ResolveURL() (*url.URL, error)

func (*CachedURL) URL 

func (ep *CachedURL) URL() (*url.URL, error)

type Client 

type Client struct {
	// contains filtered or unexported fields
}

func NewClient 

func NewClient(endpoint string) (*Client, error)

NewClient returns a new Client. To export metrics, register the prometheus.Collector instances in metrics.GazetteClientCollectors().

Example 
package main

import (
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
	"github.com/sirupsen/logrus"

	"github.com/LiveRamp/gazette/pkg/gazette"
	"github.com/LiveRamp/gazette/pkg/journal"
	"github.com/LiveRamp/gazette/pkg/metrics"
)

func main() {
	// Register collectors and attach handler to expose metrics to Prometheus.
	prometheus.MustRegister(metrics.GazetteClientCollectors()...)
	http.Handle("/metrics", promhttp.Handler())

	// Get a client.
	client, err := gazette.NewClient("192.168.0.1:8081")
	if err != nil {
		logrus.WithField("err", err).Fatal("Failed to connect to gazette")
	}

	// Use client.
	client.Get(journal.ReadArgs{Journal: "a/journal", Offset: 1234})
}

Output:

func NewClientWithHttpClient 

func NewClientWithHttpClient(endpoint string, hc *http.Client) (*Client, error)

func (*Client) Create 

func (c *Client) Create(name journal.Name) error

Creates the Journal of the given name.

func (*Client) Do 

func (c *Client) Do(request *http.Request) (*http.Response, error)

Thin layer upon http.Do(), which manages re-writes from and update to the Client.locationCache. Specifically, request.Path is mapped into a previously- stored Location re-write. If none is available, the request is re-written to reference the default endpoint. Cache entries are updated on successful redirect or response with a Location: header. On error, cache entries are expunged (eg, future requests are performed against the default endpoint).

func (*Client) FragmentBeforeTime 

func (c *Client) FragmentBeforeTime(name journal.Name, t time.Time) (journal.Fragment, error)

Returns the |Fragment| whose Modified time is closest to but prior to the given |t|. Can return a zeroed Fragment structure, if no fragment matches.

func (*Client) FragmentsInRange 

func (c *Client) FragmentsInRange(name journal.Name, minOff, maxOff int64) ([]journal.Fragment, error)

Returns a list of |Fragment|s that service the given offset range in |journal|.

func (*Client) Get 

func (c *Client) Get(args journal.ReadArgs) (journal.ReadResult, io.ReadCloser)

func (*Client) GetDirect 

func (c *Client) GetDirect(args journal.ReadArgs) (journal.ReadResult, io.ReadCloser)

func (*Client) Head 

func (c *Client) Head(args journal.ReadArgs) (journal.ReadResult, *url.URL)

func (*Client) Put 

func (c *Client) Put(args journal.AppendArgs) journal.AppendResult

Performs a Gazette PUT operation, which appends content to the named journal. Put panics if |args.Content| does not implement io.ReadSeeker.

type CreateAPI 

type CreateAPI struct {
	// contains filtered or unexported fields
}

API for creation of a new Journal. In particular, CreateAPI creates an Etcd item directory for the Journal under Gazette's consensus.Allocator root and responds to the client when the Journal is ready for transactions.

func NewCreateAPI 

func NewCreateAPI(cfs cloudstore.FileSystem, keysAPI etcd.KeysAPI,
	requiredReplicas int) *CreateAPI

func (*CreateAPI) Create 

func (h *CreateAPI) Create(w http.ResponseWriter, r *http.Request)

func (*CreateAPI) Register 

func (h *CreateAPI) Register(router *mux.Router)

type JournalReplica 

type JournalReplica interface {
	AppendOpHandler
	ReadOpHandler
	ReplicateOpHandler
	Shutdown()
	StartBrokeringWithPeers(journal.RouteToken, []journal.Replicator)
	StartReplicating(journal.RouteToken)
}

See journal.Replica.

type Persister 

type Persister struct {
	// contains filtered or unexported fields
}

func NewPersister 

func NewPersister(directory string, cfs cloudstore.FileSystem,
	keysAPI etcd.KeysAPI, routeKey string) *Persister

func (*Persister) IsShuttingDown 

func (p *Persister) IsShuttingDown() bool

func (*Persister) Persist 

func (p *Persister) Persist(fragment journal.Fragment)

func (*Persister) StartPersisting 

func (p *Persister) StartPersisting() *Persister

func (*Persister) Stop 

func (p *Persister) Stop()

func (*Persister) String 

func (p *Persister) String() string

Note: This String() implementation is primarily for the benefit of expvar, which expects the string to be a serialized JSON object.

type ReadAPI 

type ReadAPI struct {
	// contains filtered or unexported fields
}

func NewReadAPI 

func NewReadAPI(handler ReadOpHandler, cfs cloudstore.FileSystem) *ReadAPI

func (*ReadAPI) Head 

func (h *ReadAPI) Head(w http.ResponseWriter, r *http.Request)

func (*ReadAPI) Read 

func (h *ReadAPI) Read(w http.ResponseWriter, r *http.Request)

func (*ReadAPI) Register 

func (h *ReadAPI) Register(router *mux.Router)

type ReadOpHandler 

type ReadOpHandler interface {
	Read(journal.ReadOp)
}

type ReplicaFactory 

type ReplicaFactory func(journal.Name) JournalReplica

Builds a JournalReplica instance with the given journal.Name.

type ReplicateAPI 

type ReplicateAPI struct {
	// contains filtered or unexported fields
}

func NewReplicateAPI 

func NewReplicateAPI(handler ReplicateOpHandler) *ReplicateAPI

func (*ReplicateAPI) Register 

func (h *ReplicateAPI) Register(router *mux.Router)

func (*ReplicateAPI) Replicate 

func (h *ReplicateAPI) Replicate(w http.ResponseWriter, r *http.Request)

type ReplicateClient 

type ReplicateClient struct {
	// contains filtered or unexported fields
}

func NewReplicateClient 

func NewReplicateClient(ep *CachedURL) ReplicateClient

func (ReplicateClient) Replicate 

func (c ReplicateClient) Replicate(op journal.ReplicateOp)

type ReplicateOpHandler 

type ReplicateOpHandler interface {
	Replicate(journal.ReplicateOp)
}

type Router 

type Router struct {
	// contains filtered or unexported fields
}

Routes and dispatches Read, Append, and Replicate operations to a collection of responsible JournalReplicas.

func NewRouter 

func NewRouter(factory ReplicaFactory) *Router

func (*Router) Append 

func (r *Router) Append(op journal.AppendOp)

func (*Router) BrokeredJournals 

func (r *Router) BrokeredJournals() []journal.Name

Returns the set of Journals which are brokered by this Router.

func (*Router) HasServedAppend 

func (r *Router) HasServedAppend(name journal.Name) bool

Returns whether |name| is both locally brokered and has successfully served an Append operation under the current route topology. This is an important indicator for consistency, as a successful Append ensures that all replicas reached agreement on the route token & write head during the transaction.

func (*Router) Read 

func (r *Router) Read(op journal.ReadOp)

func (*Router) Replicate 

func (r *Router) Replicate(op journal.ReplicateOp)

func (*Router) ReplicatedJournals 

func (r *Router) ReplicatedJournals() []journal.Name

Returns the set of Journals which are replicated by this Router.

type Runner 

type Runner struct {
	// contains filtered or unexported fields
}

func NewRunner 

func NewRunner(client etcd.Client, localRouteKey string, replicaCount int, router *Router) *Runner

func (*Runner) FixedItems 

func (r *Runner) FixedItems() []string

consumer.Allocator implementation.

func (*Runner) InstanceKey 

func (r *Runner) InstanceKey() string

func (*Runner) ItemIsReadyForPromotion 

func (r *Runner) ItemIsReadyForPromotion(item, state string) bool

func (*Runner) ItemRoute 

func (r *Runner) ItemRoute(item string, route consensus.Route, index int, tree *etcd.Node)

func (*Runner) ItemState 

func (r *Runner) ItemState(item string) string

func (*Runner) KeysAPI 

func (r *Runner) KeysAPI() etcd.KeysAPI

func (*Runner) PathRoot 

func (r *Runner) PathRoot() string

func (*Runner) Replicas 

func (r *Runner) Replicas() int

func (*Runner) Run 

func (r *Runner) Run() error

type WriteAPI 

type WriteAPI struct {
	// contains filtered or unexported fields
}

func NewWriteAPI 

func NewWriteAPI(handler AppendOpHandler) *WriteAPI

func (*WriteAPI) Register 

func (h *WriteAPI) Register(router *mux.Router)

func (*WriteAPI) Write 

func (h *WriteAPI) Write(w http.ResponseWriter, r *http.Request)

type WriteService 

type WriteService struct {
	// contains filtered or unexported fields
}

WriteService wraps a Client to provide asynchronous batching and automatic retries of writes to Gazette journals. Writes to each journal are spooled to local disk (and never memory), so back-pressure from slow or down brokers does not affect busy writers (at least, until disk runs out). Writes are retried indefinitely, until aknowledged by a broker.

func NewWriteService 

func NewWriteService(client *Client) *WriteService

func (*WriteService) ReadFrom 

func (c *WriteService) ReadFrom(name journal.Name, r io.Reader) (*journal.AsyncAppend, error)

Appends |r|'s content to |journal|, by reading until io.EOF. Either all of |r| is written, or none of it is. Returns an AsyncAppend which is resolved when the write has been fully committed.

func (*WriteService) SetConcurrency 

func (c *WriteService) SetConcurrency(concurrency int)

func (*WriteService) Start 

func (c *WriteService) Start()

Begins the write service loop. Be sure to invoke Stop() prior to process exit, to ensure that all pending writes have been flushed.

func (*WriteService) Stop 

func (c *WriteService) Stop()

Stops the write service loop. Returns only after all writes have completed.

func (*WriteService) Write 

func (c *WriteService) Write(name journal.Name, buf []byte) (*journal.AsyncAppend, error)

Appends |buffer| to |journal|. Either all of |buffer| is written, or none of it is. Returns a AsyncAppendwhich is resolved when the write has been fully committed.

func (*WriteService) WriterFor 

func (c *WriteService) WriterFor(name journal.Name, sync bool) io.Writer

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.