pachyderm

README

Pachyderm: A Containerized Data Lake

• News
• Getting Started
• What is Pachyderm?
• What's new about Pachyderm? (How is it different from Hadoop?)
• Our Vision
• Contributing
• Usage Metrics

News

WE'RE HIRING! Love Docker, Go and distributed systems? Learn more about our team and email us at jobs@pachyderm.io.

Getting Started

Already got a kubernetes cluster:

$ kubectl create -f https://pachyderm.io/manifest.json

Otherwise, check out our setup instructions.

If you've never used Pachyderm before you should look at the fruit stand example.

Pachyderm has a CLI called pachctl, you can install it with make install or using Homebrew:

$ brew tap pachyderm/tap && brew install pachctl

Docs for pachctl can be found here.

Pachyderm also supports a Go client library, Godocs are here.

What is Pachyderm?

Pachyderm is a Data Lake -- a place to dump and process gigantic data sets. Pachyderm is inspired by the Hadoop ecosystem but shares no code with it. Instead, we leverage the container ecosystem to provide the broad functionality of Hadoop with the ease of use of Docker.

Pachyderm offers the following core functionality:

• Virtually limitless storage for any data.
• Virtually limitless processing power using any tools.
• Tracking of data history, provenance and ownership. (Version Control for data).
• Automatic processing on new data as it’s ingested. (Streaming).
• Chaining processes together. (Pipelining)

What's new about Pachyderm? (How is it different from Hadoop?)

There are two bold new ideas in Pachyderm:

• Containers as the core processing primitive
• Version Control for data

These ideas lead directly to a system that's much more powerful, flexible and easy to use.

To process data, you simply create a containerized program which reads and writes to the local filesystem. You can use any tools you want because it's all just going in a container! Pachyderm will take your container and inject data into it by way of a FUSE volume. We'll then automatically replicate your container, showing each copy a different chunk of data. With this technique, Pachyderm can scale any code you write to process up to petabytes of data (Example: distributed grep).

Pachyderm also version controls all data using a commit-based distributed filesystem (PFS), similar to what git does with code. Version control for data has far reaching consequences in a distributed filesystem. You get the full history of your data, it's much easier to collaborate with teammates, and if anything goes wrong you can revert the entire cluster with one click!

Version control is also very synergistic with our containerized processing engine. Pachyderm understands how your data changes and thus, as new data is ingested, can run your workload on the diff of the data rather than the whole thing. This means that there's no difference between a batched job and a streaming job, the same code will work for both!

Our Vision

Containers are a revolutionary new technology with a compelling application to big data. Our goal is to fully realize that use case. Hadoop has spawned a sprawling ecosystem of tools but with each new tool the complexity of your cluster grows until maintaining it becomes a full-time job. Containers are the perfect antidote to this problem. What if adding a new tool to your data infrastructure was as easy as installing an app? Thanks to the magic of containers in Pachyderm, it really is!

The most exciting thing about this vision though is what comes next. Pachyderm can do big data with anything that runs on Linux. And anything you build can be easily shared with the rest of the community, afterall it's just a container so it's completely reusable and will run the same every time. We have some ideas of our own about what the best starting building blocks will be, but it's just the tip of the iceburg -- we expect our users will have many more interesting ideas. We can't wait to see what they are!

Contributing

Deploying Pachyderm.

To get started, sign the Contributor License Agreement.

Send us PRs, we would love to see what you do!

Usage Metrics

Pachyderm automatically reports anonymized usage metrics. These metrics help us understand how people are using Pachyderm and make it better. They can be disabled by setting the env variable METRICS to false in the pachd container.

Documentation

Index

• func Asset(name string) ([]byte, error)
• func AssetDir(name string) ([]string, error)
• func AssetInfo(name string) (os.FileInfo, error)
• func AssetNames() []string
• func MustAsset(name string) []byte
• func RestoreAsset(dir, name string) error
• func RestoreAssets(dir, name string) error

Functions

func Asset

func Asset(name string) ([]byte, error)

Asset loads and returns the asset for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetDir

func AssetDir(name string) ([]string, error)

AssetDir returns the file names below a certain directory embedded in the file by go-bindata. For example if you run go-bindata on data/... and data contains the following hierarchy:

data/
  foo.txt
  img/
    a.png
    b.png

then AssetDir("data") would return []string{"foo.txt", "img"} AssetDir("data/img") would return []string{"a.png", "b.png"} AssetDir("foo.txt") and AssetDir("notexist") would return an error AssetDir("") will return []string{"data"}.

func AssetInfo

func AssetInfo(name string) (os.FileInfo, error)

AssetInfo loads and returns the asset info for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetNames

func AssetNames() []string

AssetNames returns the names of the assets.

func MustAsset

func MustAsset(name string) []byte

MustAsset is like Asset but panics when Asset would return an error. It simplifies safe initialization of global variables.

func RestoreAsset

func RestoreAsset(dir, name string) error

RestoreAsset restores an asset under the given directory

func RestoreAssets

func RestoreAssets(dir, name string) error

RestoreAssets restores an asset under the given directory recursively