numaflow

package module

v0.7.0 Latest Latest Go to latest Published: Jan 13, 2023 License: Apache-2.0 Imports: 2 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/numaproj/numaflow

Links

Open Source Insights

README ¶

Numaflow

Summary

Numaflow is a Kubernetes-native tool for running massively parallel stream processing. A Numaflow Pipeline is implemented as a Kubernetes custom resource and consists of one or more source, data processing, and sink vertices.

Numaflow installs in a few minutes and is easier and cheaper to use for simple data processing applications than a full-featured stream processing platforms.

Use Cases

Real-time data analytics applications.
Event driven applications such as anomaly detection, monitoring and alerting.
Streaming applications such as data instrumentation and data movement.
Workflows running in a streaming manner.

Key Features

Kubernetes-native: If you know Kubernetes, you already know how to use Numaflow.
Language agnostic: Use your favorite programming language.
Exactly-Once semantics: No input element is duplicated or lost even as pods are rescheduled or restarted.
Auto-scaling with back-pressure: Each vertex automatically scales from zero to whatever is needed.

Data Integrity Guarantees:

Minimally provide at-least-once semantics
Provide exactly-once semantics for unbounded and near real-time data sources
Preserving order is not required

Roadmap

User Defined Transformer at Source for better deserialization and filter for cost reduction (v0.8)
Multi partitioned edges for higher throughput (v0.9)

Resources

Documentation ¶

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Version ¶

type Version struct {
	Version      string
	BuildDate    string
	GitCommit    string
	GitTag       string
	GitTreeState string
	GoVersion    string
	Compiler     string
	Platform     string
}

Version contains version information

func GetVersion ¶

func GetVersion() Version

GetVersion returns the version information

func (Version) String ¶

func (v Version) String() string

String outputs the version as a string

Source Files ¶

View all Source files

version.go

Directories ¶

Path	Synopsis
cmd
commands
hack
crdgen
gen-openapi-spec
json-schema
pkg
apis/numaflow/v1alpha1 +kubebuilder:object:generate=true +groupName=numaflow.numaproj.io +k8s:deepcopy-gen=package,register +k8s:openapi-gen=true	+kubebuilder:object:generate=true +groupName=numaflow.numaproj.io +k8s:deepcopy-gen=package,register +k8s:openapi-gen=true
apis/proto/daemon Package daemon is a reverse proxy.	Package daemon is a reverse proxy.
client/clientset/versioned This package has the automatically generated clientset.	This package has the automatically generated clientset.
client/clientset/versioned/fake This package has the automatically generated fake clientset.	This package has the automatically generated fake clientset.
client/clientset/versioned/scheme This package contains the scheme of the automatically generated clientset.	This package contains the scheme of the automatically generated clientset.
client/clientset/versioned/typed/numaflow/v1alpha1 This package has the automatically generated typed clients.	This package has the automatically generated typed clients.
client/clientset/versioned/typed/numaflow/v1alpha1/fake Package fake has the automatically generated clients.	Package fake has the automatically generated clients.
client/informers/externalversions
client/informers/externalversions/internalinterfaces
client/informers/externalversions/numaflow
client/informers/externalversions/numaflow/v1alpha1
client/listers/numaflow/v1alpha1
daemon/client
daemon/server
daemon/server/service Package service is built for querying metadata and to expose it over daemon service.	Package service is built for querying metadata and to expose it over daemon service.
forward Package forward does the Read (fromBuffer) -> Process (UDF) -> Forward (toBuffers) -> Ack (fromBuffer) loop.	Package forward does the Read (fromBuffer) -> Process (UDF) -> Forward (toBuffers) -> Ack (fromBuffer) loop.
isb
isb/stores/jetstream
isb/stores/redis
isb/stores/simplebuffer
isb/testutils
isbsvc
metrics
reconciler Package reconciler defines implementations of the Reconciler interface defined at sigs.k8s.io/controller-runtime/pkg/reconcile.Reconciler.	Package reconciler defines implementations of the Reconciler interface defined at sigs.k8s.io/controller-runtime/pkg/reconcile.Reconciler.
reconciler/cmd
reconciler/isbsvc
reconciler/isbsvc/installer
reconciler/pipeline
reconciler/vertex
reconciler/vertex/scaling Package scaling provides the autoscaling capability for Numaflow.	Package scaling provides the autoscaling capability for Numaflow.
reduce Package reduce reads messages from isb attaches watermark to read messages invoke the read-loop with the read messages	Package reduce reads messages from isb attaches watermark to read messages invoke the read-loop with the read messages
reduce/pbq
reduce/pbq/partition
reduce/pbq/store
reduce/pbq/store/memory
reduce/pbq/store/noop
reduce/pbq/store/wal Package wal implements write-ahead-log.	Package wal implements write-ahead-log.
reduce/pnf Package pnf processes and then forwards messages belonging to a window.	Package pnf processes and then forwards messages belonging to a window.
reduce/readloop Package readloop is responsible for the first part of reduce subsystem.	Package readloop is responsible for the first part of reduce subsystem.
shared/clients/nats
shared/clients/nats/test
shared/clients/redis
shared/expr
shared/logging
shared/queue
shared/tls
shared/util
shuffle
sinks
sinks/blackhole
sinks/kafka
sinks/logger
sinks/udsink
sources
sources/generator Package generator contains an implementation of a in memory generator that generates payloads in json format.	Package generator contains an implementation of a in memory generator that generates payloads in json format.
sources/http
sources/kafka
sources/nats
udf
udf/applier
udf/builtin
udf/builtin/cat
udf/builtin/filter
udf/function Package funcmock is a generated GoMock package.	Package funcmock is a generated GoMock package.
watermark/fetch
watermark/generic
watermark/generic/jetstream
watermark/ot Package ot represents the offset-timeline pair and its corresponding encoder and decoder.	Package ot represents the offset-timeline pair and its corresponding encoder and decoder.
watermark/processor Package processor is the smallest processor entity for which the watermark will strictly monotonically increase.	Package processor is the smallest processor entity for which the watermark will strictly monotonically increase.
watermark/publish
watermark/store
watermark/store/inmem Package inmem package implements the watermark progression using in mem store as the KV store.	Package inmem package implements the watermark progression using in mem store as the KV store.
watermark/store/jetstream Package jetstream package implements the watermark progression using Jetstream as the KV store.	Package jetstream package implements the watermark progression using Jetstream as the KV store.
watermark/store/noop
window Package window implements windowing constructs.	Package window implements windowing constructs.
window/keyed Package keyed implements KeyedWindows.	Package keyed implements KeyedWindows.
window/strategy/fixed Package fixed implements Fixed windows.	Package fixed implements Fixed windows.
window/strategy/sliding Package sliding implements Sliding windows.	Package sliding implements Sliding windows.
server
apis
apis/v1
cmd
routes
test
e2e-api
fixtures

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL