argo-dataflow

README

Argo Dataflow

Summary

Argo Dataflow is intended as a cloud-native and language-agnostic platform for executing large parallel data-processing pipelines composed of many steps which are often small and homogenic.

Use Cases

• Real-time "click" analytics
• Anomaly detection
• Fraud detection
• Operational (including IoT) analytics

Screenshot

Example

pip install git+https://github.com/argoproj-labs/argo-dataflow#subdirectory=dsls/python

from argo_dataflow import cron, pipeline

if __name__ == '__main__':
    (pipeline('hello')
     .namespace('argo-dataflow-system')
     .step(
        (cron('*/3 * * * * *')
         .cat('main')
         .log())
    )
     .run())

Documentation

Read in order:

Beginner:

• Quick start
• Examples
• Steps
• Sources and Sinks
• Garbage collection

Intermediate:

• Handlers
• Git usage
• Reliability
• Scaling
• Command line
• Expression syntax
• Workflow interop

Advanced

• Configuration
• Features
• Metrics
• Image contract
• Reading material
• Security
• Contributing

Architecture Diagram