bruin

README

Bruin is a command-line tool for validating and running data transformations on SQL, similar to dbt. On top, bruin can also run Python assets within the same pipeline.

Bruin is built to make your life easier when it comes to data transformations:

• ✨ run SQL transformations on BigQuery/Snowflake
• 🐍 run Python in isolated environments in the same pipeline
• 💅 built-in data quality checks
• 🚀 Jinja templating to avoid repetition
• ✅ validate data pipelines end-to-end to catch issues early on via dry-run on live
• 📐 table/view materialization
• ➕ incremental tables
• ⚡ blazing fast pipeline execution: bruin is written in Golang and uses concurrency at every opportunity
• 🔒 secrets injection via environment variables
• 📦 easy to install and use

Installation

macOS

brew tap bruin-data/tap
brew install bruin

Linux

Binaries are available on the releases page.

via Golang installer

You need to have Golang installed in the first place, then you can run the following command:

go install github.com/bruin-data/bruin@latest

[!IMPORTANT] Please make sure to add GOPATH to your executable path.

Getting Started

All you need is a simple pipeline.yml in your Git repo:

name: bruin-example
schedule: "daily"
start_date: "2023-03-01"

default_connections:
  google_cloud_platform: "gcp"

create a new folder called assets and create your first asset there assets/bruin-test.sql:

-- @bruin.name: dataset.bruin_test
-- @bruin.type: bq.sql
-- @bruin.materialization.type: table

SELECT 1 as result

bruin will take this result, and will create a dataset.bruin_test table on BigQuery. You can also use view materialization type instead of table to create a view instead.

Snowflake assets If you'd like to run the asset on Snowflake, simply replace the bq.sql with sf.sql, and define snowflake as a connection instead of google_cloud_platform.

Then let's create a Python asset assets/hello.py:

# @bruin.name: hello
# @bruin.depends: dataset.bruin_test

print("Hello, world!")

Once you are done, run the following command to validate your pipeline:

bruin validate .

You should get an output that looks like this:

Pipeline: bruin-example (.)
  No issues found

✓ Successfully validated 2 tasks across 1 pipeline, all good.

If you have defined your credentials, bruin will automatically detect them and validate all of your queries using dry-run.

Environments

bruin allows you to run your pipelines / assets against different environments, such as development or production. The environments are managed in the .bruin.yml file.

The following is an example configuration that defines two environments called default and production:

environments:
  default:
    connections:
      google_cloud_platform:
        - name: "gcp"
          service_account_file: "/path/to/my/key.json"
          project_id: "my-project-dev"
      snowflake:
        - name: "snowflake"
          username: "my-user"
          password: "my-password"
          account: "my-account"
          database: "my-database"
          warehouse: "my-warehouse"
          schema: "my-dev-schema"
      generic:
        - name: KEY1
          value: value1
  production:
    connections:
      google_cloud_platform:
        - name: "gcp"
          service_account_file: "/path/to/my/prod-key.json"
          project_id: "my-project-prod"
      snowflake:
        - name: "snowflake"
          username: "my-user"
          password: "my-password"
          account: "my-account"
          database: "my-database"
          warehouse: "my-warehouse"
          schema: "my-prod-schema" 
      generic:
        - name: KEY1
          value: value1

You can simply switch the environment using the --environment flag, e.g.:

bruin validate --environment production . 

Running the pipeline

bruin CLI can run the whole pipeline or any task with the downstreams:

bruin run .

Starting the pipeline execution...

[2023-03-16T18:25:14Z] [worker-0] Running: dashboard.bruin-test
[2023-03-16T18:25:16Z] [worker-0] Completed: dashboard.bruin-test (1.681s)
[2023-03-16T18:25:16Z] [worker-4] Running: hello
[2023-03-16T18:25:16Z] [worker-4] [hello] >> Hello, world!
[2023-03-16T18:25:16Z] [worker-4] Completed: hello (116ms)

Executed 2 tasks in 1.798s

You can also run a single task:

bruin run assets/hello.py                            

Starting the pipeline execution...

[2023-03-16T18:25:59Z] [worker-0] Running: hello
[2023-03-16T18:26:00Z] [worker-0] [hello] >> Hello, world!
[2023-03-16T18:26:00Z] [worker-0] Completed: hello (103ms)


Executed 1 tasks in 103ms

You can optionally pass a --downstream flag to run the task with all of its downstreams.