numaflow

package module

v0.6.1 Latest Latest Go to latest Published: Oct 27, 2022 License: Apache-2.0 Imports: 2 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/numaproj/numaflow

Links

Open Source Insights

README ¶

Numaflow

Summary

Numaflow is a Kubernetes-native tool for running massively parallel stream processing. A Numaflow Pipeline is implemented as a Kubernetes custom resource and consists of one or more source, data processing, and sink vertices.

Numaflow installs in a few minutes and is easier and cheaper to use for simple data processing applications than a full-featured stream processing platforms.

Key Features

Kubernetes-native: If you know Kubernetes, you already know how to use Numaflow.
Language agnostic: Use your favorite programming language.
Exactly-Once semantics: No input element is duplicated or lost even as pods are rescheduled or restarted.
Auto-scaling with back-pressure: Each vertex automatically scales from zero to whatever is needed.

Data Integrity Guarantees:

Minimally provide at-least-once semantics
Provide exactly-once semantics for unbounded and near real-time data sources
Preserving order is not required

Roadmap

Data aggregation (e.g. group-by)

Resources

Documentation ¶

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Version ¶

type Version struct {
	Version      string
	BuildDate    string
	GitCommit    string
	GitTag       string
	GitTreeState string
	GoVersion    string
	Compiler     string
	Platform     string
}

Version contains version information

func GetVersion ¶

func GetVersion() Version

GetVersion returns the version information

func (Version) String ¶

func (v Version) String() string

String outputs the version as a string

Source Files ¶

View all Source files

version.go

Directories ¶

Path	Synopsis
cmd
commands
hack
crdgen
gen-openapi-spec
json-schema
pkg
apis/numaflow/v1alpha1 +kubebuilder:object:generate=true +groupName=numaflow.numaproj.io +k8s:deepcopy-gen=package,register +k8s:openapi-gen=true	+kubebuilder:object:generate=true +groupName=numaflow.numaproj.io +k8s:deepcopy-gen=package,register +k8s:openapi-gen=true
apis/proto/daemon Package daemon is a reverse proxy.	Package daemon is a reverse proxy.
client/clientset/versioned This package has the automatically generated clientset.	This package has the automatically generated clientset.
client/clientset/versioned/fake This package has the automatically generated fake clientset.	This package has the automatically generated fake clientset.
client/clientset/versioned/scheme This package contains the scheme of the automatically generated clientset.	This package contains the scheme of the automatically generated clientset.
client/clientset/versioned/typed/numaflow/v1alpha1 This package has the automatically generated typed clients.	This package has the automatically generated typed clients.
client/clientset/versioned/typed/numaflow/v1alpha1/fake Package fake has the automatically generated clients.	Package fake has the automatically generated clients.
client/informers/externalversions
client/informers/externalversions/internalinterfaces
client/informers/externalversions/numaflow
client/informers/externalversions/numaflow/v1alpha1
client/listers/numaflow/v1alpha1
daemon/client
daemon/server
daemon/server/service Package service is built for querying metadata and to expose it over daemon service.	Package service is built for querying metadata and to expose it over daemon service.
isb
isb/forward Package forward does the Read (fromBuffer) -> Process (UDF) -> Forward (toBuffers) -> Ack (fromBuffer) loop.	Package forward does the Read (fromBuffer) -> Process (UDF) -> Forward (toBuffers) -> Ack (fromBuffer) loop.
isb/stores/jetstream
isb/stores/redis
isb/stores/simplebuffer
isb/testutils
isbsvc
messages
metrics
pbq
pbq/partition
pbq/store
pbq/store/memory
pbq/store/noop
reconciler Package reconciler defines implementations of the Reconciler interface defined at sigs.k8s.io/controller-runtime/pkg/reconcile.Reconciler.	Package reconciler defines implementations of the Reconciler interface defined at sigs.k8s.io/controller-runtime/pkg/reconcile.Reconciler.
reconciler/cmd
reconciler/isbsvc
reconciler/isbsvc/installer
reconciler/pipeline
reconciler/vertex
reconciler/vertex/scaling Package scaling provides the autoscaling capability for Numaflow.	Package scaling provides the autoscaling capability for Numaflow.
reduce Package reduce reads messages from isb attaches watermark to read messages invoke the read-loop with the read messages	Package reduce reads messages from isb attaches watermark to read messages invoke the read-loop with the read messages
reduce/pnf Package pnf processes and then forwards messages belonging to a window.	Package pnf processes and then forwards messages belonging to a window.
reduce/readloop Package readloop is responsible for the first part of reduce subsystem.	Package readloop is responsible for the first part of reduce subsystem.
shared/clients/jetstream Package jetstream provides interface and two implementations to connect Nats JetStream.	Package jetstream provides interface and two implementations to connect Nats JetStream.
shared/clients/redis
shared/expr
shared/logging
shared/queue
shared/tls
shared/util
shuffle
sinks
sinks/kafka
sinks/logger
sinks/udsink
sources
sources/generator Package generator contains an implementation of a in memory generator that generates payloads in json format.	Package generator contains an implementation of a in memory generator that generates payloads in json format.
sources/http
sources/kafka
udf
udf/applier
udf/builtin
udf/builtin/cat
udf/builtin/filter
udf/function Package funcmock is a generated GoMock package.	Package funcmock is a generated GoMock package.
udf/reducer
watermark/fetch
watermark/generic
watermark/generic/jetstream
watermark/processor Package processor is the smallest processor entity for which the watermark will strictly monotonically increase.	Package processor is the smallest processor entity for which the watermark will strictly monotonically increase.
watermark/publish
watermark/store
watermark/store/inmem Package inmem package implements the watermark progression using in mem store as the KV store.	Package inmem package implements the watermark progression using in mem store as the KV store.
watermark/store/jetstream Package jetstream package implements the watermark progression using Jetstream as the KV store.	Package jetstream package implements the watermark progression using Jetstream as the KV store.
watermark/store/noop
window Package window implements windowing constructs.	Package window implements windowing constructs.
window/keyed Package keyed ....	Package keyed ....
window/strategy/fixed Package fixed implements Fixed windows.	Package fixed implements Fixed windows.
server
apis
apis/v1
cmd
routes
test
e2e-api
fixtures

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL