README
¶
Maintainable automation.
🚧 Status: alpha 🚧
Early development
Please expect breaking changes. Feel free to file an issue if you
- find a bug
- need a feature added or tweaked for your use case
- have any other suggestions
PipeDream is an open-source, general-purpose automation tool.
It is an alternative to shell scripts - just as powerful, but more portable, composable, testable and reliable.
How it works
Define a pipeline in a yaml file. Using a simple, but extensible syntax, you can call, chain, merge, and split pipelines, invoke shell commands, handle errors, write tests, create mocks, and much more.
Getting started
Have a look at the Quick Start Guide, the Documentation or just run the program. It will offer to create a sample pipeline file in your current folder.
Installation
From npm
npm i -g @l9/pipedream
Using homebrew (Mac OS/linux)
brew install layer9berlin/tap/pipedream
From source (requires Go installed)
git clone https://github.com/Layer9Berlin/PipeDream
cd PipeDream
go build
pipedream
then select Install when promted.
This will compile PipeDream and run it locally, allowing you to execute the install.pipe file (very meta), which in turn installs everything you need to run PipeDream anywhere on your machine. If you make changes to the code, just run PipeDream and execute the installation.pipe again to update your installed version. You can skip the installation step entirely if you only want to run PipeDream in the project folder and don't need any of the utility tools.
If you use PipeDream a lot, it might be convenient to set up an alias like p that is faster to type.
Uninstall
Installed via npm
npm uninstall -g @l9/pipedream
Installed via homebrew
brew uninstall pipedream
Compiled from source
Simply run
pipedream
in the repo source directory and select Uninstall.
Not sure
If you don't recall how you installed PipeDream, run
pipedream version
and check the via field.
Why wouldn't I just use a shell script?
Shell scripting is an art that takes years to master. Tools like shellcheck and bats can help you improve the code quality of your scripts, but they don't solve the core problems:
- There are many traps for beginners to fall into
- Each command has its own syntax (not to mention subtle differences between shells)
- Error handling is so difficult it's rarely done well
- Even simple tasks often require contorted solutions that break for edge cases
- A lack of tests and documentation means you generally don't want to adapt an existing script
If you have used a modern CI/CD, orchestration or containerization tool, you are probably already familiar with a different way to write a script: defining tasks in yaml format. PipeDream is the natural extension of these tools to your localhost. Automate anything, anywhere, using a simple syntax that follows the principle of least astonishment.
"It's like Ansible for localhost!"
Anonymous
We have created PipeDream to simplify maintenance and dependency management tasks that used to involve a large number of steps, combining many different tools running both locally and on remote servers.
To illustrate its potential, we have used PipeDream to define a universal dependency manager pipe. It offers a single, consistent interface pulling together results from different package managers.
Documentation
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Implements the `pipedream` command line tool
|
Implements the `pipedream` command line tool |
|
Package src contains the implementation of the `pipedream` functionalities
|
Package src contains the implementation of the `pipedream` functionalities |
|
custom
Extensions and customizations of native/third party libraries
|
Extensions and customizations of native/third party libraries |
|
custom/evaluate
Package evaluate provides custom functions extending the `govaluate` package
|
Package evaluate provides custom functions extending the `govaluate` package |
|
custom/filepath
Custom functions extending the native `filepath` package
|
Custom functions extending the native `filepath` package |
|
custom/io
Custom functions extending the native `io` package
|
Custom functions extending the native `io` package |
|
custom/math
Custom functions extending the native `math` package
|
Custom functions extending the native `math` package |
|
custom/stringmap
Convenience functions for maps with string keys
|
Convenience functions for maps with string keys |
|
custom/strings
Custom functions extending the native `strings` package
|
Custom functions extending the native `strings` package |
|
datastream
Provides a composable data stream, which pipes multiple inputs to multiple outputs according to the defined compositions
|
Provides a composable data stream, which pipes multiple inputs to multiple outputs according to the defined compositions |
|
logging
Package logging provides custom logging and activity indication
|
Package logging provides custom logging and activity indication |
|
logging/fields
Provides custom log entry fields
|
Provides custom log entry fields |
|
middleware
Package middleware slices the implementation into middlewares, each performing side effects and adapting the run based on provided arguments
|
Package middleware slices the implementation into middlewares, each performing side effects and adapting the run based on provided arguments |
|
middleware/catch
Package catch provides a middleware for graceful handling of stderr output
|
Package catch provides a middleware for graceful handling of stderr output |
|
middleware/catcheach
Package catcheach provides a middleware for graceful handling of stderr output based on regex patterns
|
Package catcheach provides a middleware for graceful handling of stderr output based on regex patterns |
|
middleware/dir
Package dir provides a middleware for changing the current working directory
|
Package dir provides a middleware for changing the current working directory |
|
middleware/docker
Package docker provides a middleware for execution within a Docker (Compose) container
|
Package docker provides a middleware for execution within a Docker (Compose) container |
|
middleware/each
Package each provides a middleware that copies some input into several child pipes running simultaneously
|
Package each provides a middleware that copies some input into several child pipes running simultaneously |
|
middleware/env
Package env provides a middleware handling environment variables
|
Package env provides a middleware handling environment variables |
|
middleware/inherit
Package inherit provides a middleware that passes arguments from a parent to its children
|
Package inherit provides a middleware that passes arguments from a parent to its children |
|
middleware/input
Package input provides a middleware to overwrite a pipe's input directly
|
Package input provides a middleware to overwrite a pipe's input directly |
|
middleware/interpolate
Package interpolate provides a middleware to substitute arguments or inputs into other arguments
|
Package interpolate provides a middleware to substitute arguments or inputs into other arguments |
|
middleware/output
Package output provides a middleware to overwrite a pipe's output directly
|
Package output provides a middleware to overwrite a pipe's output directly |
|
middleware/pipe
Package pipe provides a middleware to execute several commands in sequence
|
Package pipe provides a middleware to execute several commands in sequence |
|
middleware/select
Package select provides a middleware that shows a selection prompt to the user
|
Package select provides a middleware that shows a selection prompt to the user |
|
middleware/shell
Package shell provides a middleware to execute commands in a shell
|
Package shell provides a middleware to execute commands in a shell |
|
middleware/ssh
Package ssh provides a middleware enabling remote command execution via SSH
|
Package ssh provides a middleware enabling remote command execution via SSH |
|
middleware/stack
Package stack defines a list of middleware items to be executed when a pipeline is run
|
Package stack defines a list of middleware items to be executed when a pipeline is run |
|
middleware/sync
Package sync provides a middleware to defer execution until a condition is fulfilled
|
Package sync provides a middleware to defer execution until a condition is fulfilled |
|
middleware/timer
Package timer provides a middleware that records execution time
|
Package timer provides a middleware that records execution time |
|
middleware/when
Package when provides a middleware that enables conditional execution
|
Package when provides a middleware that enables conditional execution |
|
middleware/with
Package with provides a middleware that extracts and processes input patterns
|
Package with provides a middleware that extracts and processes input patterns |
|
parsing
Package parsing provides a parser for pipeline yaml files
|
Package parsing provides a parser for pipeline yaml files |
|
pipeline
Provides a model representing a pipeline definition that defines default arguments for a pipeline identifier Provides a representation of a yaml pipeline file as a Go struct Provides models to represent pipeline files and runs Provides a model representing a pipeline run
|
Provides a model representing a pipeline definition that defines default arguments for a pipeline identifier Provides a representation of a yaml pipeline file as a Go struct Provides models to represent pipeline files and runs Provides a model representing a pipeline run |
|
run
Package run contains the implementation of the default `pipedream` shell command, selecting, parsing and executing pipelines
|
Package run contains the implementation of the default `pipedream` shell command, selecting, parsing and executing pipelines |
|
version
Package version provides the implementation of the version command, providing information about the current `pipedream` command line tool installation
|
Package version provides the implementation of the version command, providing information about the current `pipedream` command line tool installation |
|
Additional command line tools to handle specific, specialized tasks not covered by pipes or middleware
|
Additional command line tools to handle specific, specialized tasks not covered by pipes or middleware |
|
semver-compare
Parse and compare two or more semantic versions
|
Parse and compare two or more semantic versions |
|
traffic-light
Augment output with a green, amber or red status indicator
|
Augment output with a green, amber or red status indicator |