argo-dataflow

README

Argo Dataflow

Summary

Argo Dataflow is intended as a cloud-native and language-agnostic platform for executing large parallel data-processing pipelines composed of many steps which are often small and homogenic.

Use Cases

• Real-time "click" analytics
• Anomaly detection
• Fraud detection
• Operational (including IoT) analytics

Screenshot

Example

apiVersion: dataflow.argoproj.io/v1alpha1
kind: Pipeline
metadata:
  name: example
  annotations:
    dataflow.argoproj.io/name: "Example pipeline"
spec:
  steps:
    - name: find-cats
      sources:
        - kafka:
            topic: pets
      filter: 'object(msg).type == "cat"'
      sinks:
        - stan:
            subject: cats

    - name: hi-cats
      sources:
        - stan:
            subject: cats
      map: '"hello " + object(msg).name'
      replicas:
        min: 2
      sinks:
        - kafka:
            topic: hello-to-cats

Documentation

• Examples
• Configuration
• Handlers
• Git usage
• Command line
• Expression syntax
• Reading material
• Contributing

Architecture Diagram