bruin

README

Bruin

Bruin is a command-line tool for validating and running data transformations on SQL, similar to dbt. On top, bruin can also run Python assets within the same pipeline.

• ✨ run SQL transformations on BigQuery/Snowflake

• 🐍 run Python in isolated environments

• 💅 built-in data quality checks

• 🚀 Jinja templating language to avoid repetition

• ✅ validate data pipelines end-to-end to catch issues early on via dry-run on live

• 📐 table/view materialization

• ➕ incremental tables

• 💻 mix different technologies + databases in a single pipeline, e.g. SQL and Python in the same pipeline

• ⚡ blazing fast pipeline execution: bruin is written in Golang and uses concurrency at every opportunity

Installation

You need to have Golang installed in the first place, then you can run the following command:

go install github.com/bruin-data/bruin@latest

Please make sure to add GOPATH to your executable path.

Getting Started

All you need is a simple pipeline.yml in your Git repo:

name: bruin-example
schedule: "daily"
start_date: "2023-03-01"

default_connections:
  google_cloud_platform: "gcp"

create a new folder called assets and create your first asset there assets/bruin-test.sql:

-- @bruin.name: dataset.bruin_test
-- @bruin.type: bq.sql
-- @bruin.materialization.type: table

SELECT 1 as result

bruin will take this result, and will create a dataset.bruin_test table on BigQuery. You can also use view materialization type instead of table to create a view instead.

Snowflake assets If you'd like to run the asset on Snowflake, simply replace the bq.sql with sf.sql, and define snowflake as a connection instead of google_cloud_platform.

Then let's create a Python asset assets/hello.py:

# @bruin.name: hello
# @bruin.type: python
# @bruin.depends: dataset.bruin_test

print("Hello, world!")

Once you are done, run the following command to validate your pipeline:

bruin validate .

You should get an output that looks like this:

Pipeline: bruin-example (.)
  No issues found

✓ Successfully validated 2 tasks across 1 pipeline, all good.

If you have defined your credentials, bruin will automatically detect them and validate all of your queries using dry-run.

Environments

bruin allows you to run your pipelines / assets against different environments, such as development or production. The environments are managed in the .bruin.yml file.

The following is an example configuration that defines two environments called default and production:

environments:
  default:
    connections:
      google_cloud_platform:
        - name: "gcp"
          service_account_file: "/path/to/my/key.json"
          project_id: "my-project-dev"
      snowflake:
        - name: "snowflake"
          username: "my-user"
          password: "my-password"
          account: "my-account"
          database: "my-database"
          warehouse: "my-warehouse"
          schema: "my-dev-schema"
  production:
    connections:
      google_cloud_platform:
        - name: "gcp"
          service_account_file: "/path/to/my/prod-key.json"
          project_id: "my-project-prod"
      snowflake:
        - name: "snowflake"
          username: "my-user"
          password: "my-password"
          account: "my-account"
          database: "my-database"
          warehouse: "my-warehouse"
          schema: "my-prod-schema" 

You can simply switch the environment using the --environment flag, e.g.:

bruin validate --environment production . 

Running the pipeline

bruin CLI can run the whole pipeline or any task with the downstreams:

bruin run .

Starting the pipeline execution...

[2023-03-16T18:25:14Z] [worker-0] Running: dashboard.bruin-test
[2023-03-16T18:25:16Z] [worker-0] Completed: dashboard.bruin-test (1.681s)
[2023-03-16T18:25:16Z] [worker-4] Running: hello
[2023-03-16T18:25:16Z] [worker-4] [hello] >> Hello, world!
[2023-03-16T18:25:16Z] [worker-4] Completed: hello (116ms)

Executed 2 tasks in 1.798s

You can also run a single task:

bruin run assets/hello.py                            

Starting the pipeline execution...

[2023-03-16T18:25:59Z] [worker-0] Running: hello
[2023-03-16T18:26:00Z] [worker-0] [hello] >> Hello, world!
[2023-03-16T18:26:00Z] [worker-0] Completed: hello (103ms)


Executed 1 tasks in 103ms

You can optionally pass a --downstream flag to run the task with all of its downstreams.

Upcoming Features

• Secrets for Python assets
• More databases: Postgres, Redshift, MySQL, and more

Disclaimer

bruin is still in its early stages, so please use it with caution. We are working on improving the documentation and adding more features.