bacalhau

command module
v1.5.0-dev16 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 3, 2024 License: Apache-2.0 Imports: 11 Imported by: 0

README ยถ

Bacalhau

The Distributed Computation Framework โšก
Compute Over Data (CoD)


total download Bacalhau contributors Bacalhau website follow on Twitter

Bacalhau is a platform for fast, cost efficient, and secure computation by running jobs where the data is generated and stored. With Bacalhau you can streamline your existing workflows without the need of extensive rewriting by running arbitrary Docker containers, WebAssembly (wasm) images, or arbitrary binaries as tasks.

Table of Contents

Why Bacalhau?

  • โšก Fast job processing: Jobs in Bacalhau are processed where the data was created and all jobs are parallel by default.
  • ๐Ÿ’ฐ Low cost: Reduce (or eliminate) ingress/egress costs since jobs are processed closer to the source. Take advantage of as well idle computation capabilities at the edge.
  • ๐Ÿ”’ Secure: Data scrubbing and security can happen before migration to reduce the chance of leaking private information, and with a far more granular, code-based permission model.
  • ๐Ÿš› Large-scale data: Bacalhau operates on a network of open compute resources made available to serve any data processing workload. With Bacalhau, you can batch process petabytes (quadrillion bytes) of data.

Getting started - Bacalhau in 1 minute

Go to the folder directory that you want to store your job results

Install the bacalhau client

curl -sL https://get.bacalhau.org/install.sh | bash

Submit a "Hello World" job

bacalhau docker run ubuntu echo Hello World

Download your result

bacalhau get 63d08ff0..... # make sure to use the right job id from the docker run command

For a more detailed tutorial, check out our Getting Started tutorial.

Learn more

Documentation

๐Ÿ“š Read the Bacalhau docs guide here! ๐Ÿ“š

The Bacalhau docs is the best starting point as it contains all the information to ensure that everyone who uses Bacalhau is doing so efficiently.

Developers guide

Running Bacalhau locally

Developers can spin up bacalhau and run a local demo using the devstack command.

Please see running_locally.md for instructions. Also, see debugging_locally.md for some useful tricks for debugging.

Notes for Dev contributors

Bacalhau's CI pipeline performs a variety of linting and formatting checks on new pull requests. To have these checks run locally when you make a new commit, you can use the precommit hook in ./githooks:

make install-pre-commit

# check if pre-commit works
make precommit

If you want to run the linter manually:

curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sudo sh -s -- -b /usr/local/go/bin
golangci-lint --version
make lint

The config lives in .golangci.yml

OpenAPI

OpenAPI v2 annotations sit by the endpoints in pkg/publicapi; these are built using swag, a Go converter for Swagger documentation. Find more details about the Swag annotations in their docs. The swagger specification is built automatically by the CI pipeline (see the build_swagger workflow) but you can trigger a local build with make swagger-docs.

The build parses the OpenAPI annotations as well as the markdown files in docs/swagger/ (containing long-form descriptions of the API endpoints), and generates the following swagger specification files:

  • docs/docs.go
  • docs/swagger.json
  • docs/swagger.yaml

Note that the CI pipeline will open and automatically merge a pull request titled [circleci] Build swagger reference - this is an automatic commit ... containing only the updated spec files.

Python Libraries

We ship two Python Bacalhau libraries:

  • bacalhau-apiclient wraps only the API endpoint calls and request/response models. It's autogenerated from the OpenAPI specification (see paragraph above). Read more about it in its readme.
  • bacalhau-sdk is a high-level Bacalhau SDK that ships all the client-side logic (e.g. signing requests) needed to query the endpoints. Its examples folder contains code snippets to create, list and inspect jobs. Under the hood, it uses the bacalhau-apiclient to call the API. Please use this library in your projects. Read more about it in its readme.

Issues, feature requests, and questions

We are excited to hear your feedback!

  • For issues and feature requests, please open a GitHub issue.
  • For questions, give feedback or answer questions that will help other user product please use GitHub Discussions.
  • To engage with other members in the community, join us in our slack community #bacalhau channel ๐Ÿ™‹

Ways to contribute

All manner of contributions are more than welcome!

We have highlighted the different ways you can contribute in our contributing guide. You can be part of community discussions, development, and more.

Open Source

This repository contains the Bacalhau software, covered under the Apache-2.0, except where noted (any Bacalhau logos or trademarks are not covered under the Apache License, and should be explicitly noted by a LICENSE file.)

Bacalhau is a product produced from this open source software, exclusively by Expanso, Inc. It is distributed under our commercial terms.

Others are allowed to make their own distribution of the software, but they cannot use any of the Bacalhau trademarks, cloud services, etc.

We explicitly grant permission for you to make a build that includes our trademarks while developing Bacalhau software itself. You may not publish or share the build, and you may not use that build to run Bacalhau software for any other purpose.

We have borrowed the above Open Source clause from the excellent System Initiative

Documentation ยถ

The Go Gopher

There is no documentation for this package.

Directories ยถ

Path Synopsis
apps
cmd
cli
util/printer
Package printer provides functionality for printing job events and progress.
Package printer provides functionality for printing job events and progress.
pkg
bacerrors
Package bacerrors provides a rich error type for detailed error handling in Go applications.
Package bacerrors provides a rich error type for detailed error handling in Go applications.
bidstrategy
Package bidstrategy is a generated GoMock package.
Package bidstrategy is a generated GoMock package.
compute
Package compute is a generated GoMock package.
Package compute is a generated GoMock package.
compute/store
Package store is a generated GoMock package.
Package store is a generated GoMock package.
eventhandler/mock_eventhandler
Package mock_system is a generated GoMock package.
Package mock_system is a generated GoMock package.
job
jobstore
Package jobstore is a generated GoMock package.
Package jobstore is a generated GoMock package.
lib/backoff
Package backoff is a generated GoMock package.
Package backoff is a generated GoMock package.
lib/watcher
Package watcher is a generated GoMock package.
Package watcher is a generated GoMock package.
logger
DataFrame wraps the byte framing used by docker to multiplex output from a docker container when there is no TTY in use.
DataFrame wraps the byte framing used by docker to multiplex output from a docker container when there is no TTY in use.
nats/stream
Package stream provides a NATS client for streaming records between clients with asynchronous response handling.
Package stream provides a NATS client for streaming records between clients with asynchronous response handling.
orchestrator
Package orchestrator is a generated GoMock package.
Package orchestrator is a generated GoMock package.
s3
storage/inline
Package inline provides a storage abstraction that stores data for use by Bacalhau jobs within the storage spec itself, without needing any connection to an external storage provider.
Package inline provides a storage abstraction that stores data for use by Bacalhau jobs within the storage spec itself, without needing any connection to an external storage provider.
swagger
Code generated by swaggo/swag.
Code generated by swaggo/swag.
test/scenario
Package scenario provides a high-level testing framework for running Bacalhau jobs in different configurations and making assertions against the results.
Package scenario provides a high-level testing framework for running Bacalhau jobs in different configurations and making assertions against the results.
test/teststack
Package testutils collects common test utilities.
Package testutils collects common test utilities.
version
Package version provides information about what Bacalhau was built from.
Package version provides information about what Bacalhau was built from.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL