README
¶
Pachyderm
- News
- What is Pachyderm?
- What's new about Pachyderm? (How is it different from Hadoop?)
- Our Vision
- Contributing
News
WE'RE HIRING! Love Docker, Go and distributed systems? Learn more about our team and email us at jobs@pachyderm.io.
Getting Started
Already got a kube cluster:
$ kubectl create -f http://pachyderm.io/manifest.json
Get up and running with the fruit stand example.
What is Pachyderm?
Pachyderm is a Data Lake -- a place to dump and process gigantic data sets. Pachyderm is inspired by the Hadoop ecosystem but shares no code with it. Instead, we leverage the container ecosystem to provide the broad functionality of Hadoop with the ease of use of Docker.
Pachyderm offers the following core functionality:
- Virtually limitless storage for any data.
- Virtually limitless processing power using any tools.
- Tracking of data history, provenance and ownership. (Version Control for data).
- Automatic processing on new data as it’s ingested. (Streaming).
- Chaining processes together. (Pipelining)
What's new about Pachyderm? (How is it different from Hadoop?)
There are two bold new ideas in Pachyderm:
- Containers as the core processing primitive
- Version Control for data
These ideas lead directly to a system that's much more powerful, flexible and easy to use.
To process data, you simply create a containerized program which reads and writes to the local filesystem. You can use any tools you want because it's all just going in a container! Pachyderm will take your container and inject data into it by way of a FUSE volume. We'll then automatically replicate your container, showing each copy a different chunk of data. With this technique, Pachyderm can scale any code you write to process up to petabytes of data (Example: distributed grep).
Pachyderm also version controls all data using a commit-based distributed filesystem (PFS), similar to what git does with code. Version control for data has far reaching consequences in a distributed filesystem. You get the full history of your data, it's much easier to collaborate with teammates, and if anything goes wrong you can revert the entire cluster with one click!
Version control is also very synergistic with our containerized processing engine. Pachyderm understands how your data changes and thus, as new data is ingested, can run your workload on the diff of the data rather than the whole thing. This means that there's no difference between a batched job and a streaming job, the same code will work for both!
Our Vision
Containers are a revolutionary new technology with a compelling application to big data. Our goal is to fully realize that use case. Hadoop has spawned a sprawling ecosystem of tools but with each new tool the complexity of your cluster grows until maintaining it becomes a full-time job. Containers are the perfect antidote to this problem. What if adding a new tool to your data infrastructure was as easy as installing an app? Thanks to the magic of containers in Pachyderm, it really is!
The most exciting thing about this vision though is what comes next. Pachyderm can do big data with anything that runs on Linux. And anything you build can be easily shared with the rest of the community, afterall it's just a container so it's completely reusable and will run the same every time. We have some ideas of our own about what the best starting building blocks will be, but it's just the tip of the iceburg -- we expect our users will have many more interesting ideas. We can't wait to see what they are!
Contributing
To get started, sign the Contributor License Agreement.
Send us PRs, we would love to see what you do!
Documentation
¶
Overview ¶
Package pfs is a reverse proxy.
It translates gRPC into RESTful JSON APIs.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RegisterAPIHandler ¶
RegisterAPIHandler registers the http handlers for service API to "mux". The handlers forward requests to the grpc endpoint over "conn".
func RegisterAPIHandlerFromEndpoint ¶
func RegisterAPIHandlerFromEndpoint(ctx context.Context, mux *runtime.ServeMux, endpoint string, opts []grpc.DialOption) (err error)
RegisterAPIHandlerFromEndpoint is same as RegisterAPIHandler but automatically dials to "endpoint" and closes the connection when "ctx" gets done.
Types ¶
This section is empty.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
examples
|
|
|
src
|
|
|
client/pfs
Package pfs is a generated protocol buffer package.
|
Package pfs is a generated protocol buffer package. |
|
client/pkg/shard
Package shard is a generated protocol buffer package.
|
Package shard is a generated protocol buffer package. |
|
client/pps
Package pps is a generated protocol buffer package.
|
Package pps is a generated protocol buffer package. |
|
server/pfs/drive
Package drive provides the definitions for the low-level pfs storage drivers.
|
Package drive provides the definitions for the low-level pfs storage drivers. |
|
server/pfs/fuse
Package fuse is a generated protocol buffer package.
|
Package fuse is a generated protocol buffer package. |
|
server/pkg/container
Package container provides functionality to interact with containers.
|
Package container provides functionality to interact with containers. |
|
server/pkg/deploy
Package deploy is a generated protocol buffer package.
|
Package deploy is a generated protocol buffer package. |
|
server/pkg/metrics
Package metrics is a generated protocol buffer package.
|
Package metrics is a generated protocol buffer package. |
|
server/pps
Package pps is a generated protocol buffer package.
|
Package pps is a generated protocol buffer package. |
|
server/pps/persist
Package persist is a generated protocol buffer package.
|
Package persist is a generated protocol buffer package. |