bigquery

README

---
title: "BigQuery"
lang: "en-US"
draft: false
description: "Learn about how to set up a VDP BigQuery component https://github.com/instill-ai/instill-core"
---

The BigQuery component is a data component that allows users to insert data to BigQuery tables.
It can carry out the following tasks:

- [Insert](#insert)
- [Read](#read)



## Release Stage

`Alpha`



## Configuration

The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/bigquery/v0/config/definition.json).




## Setup


| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| JSON Key File contents (required) | `json-key` | string | Contents of the JSON key file with access to the bucket. |
| BigQuery Project ID (required) | `project-id` | string | Fill in your BigQuery Project ID. |
| BigQuery Dataset ID (required) | `dataset-id` | string | Fill in your BigQuery Dataset ID. |
| BigQuery Table Name (required) | `table-name` | string | Fill in your BigQuery Table Name. |




## Supported Tasks

### Insert

Insert data to BigQuery.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_INSERT` |
| Data | `data` | object | The data to be inserted to BigQuery |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Status of the upload operation |






### Read

Read data from BigQuery.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_READ` |
| Filtering | `filtering` | string | The filter to be applied to the data with SQL syntax, which starts with WHERE clause |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Data | `data` | array[object] | The data to be read from BigQuery |

Documentation

Index

• func Init(bc base.Component) *component
• func NewClient(jsonKey, projectID string) (*bigquery.Client, error)
• type Column
• type DataSaver
  • func (v DataSaver) Save() (row map[string]bigquery.Value, insertID string, err error)
• type ReadInput
• type ReadOutput
• type TableColumns

Functions

func Init

func Init(bc base.Component) *component

func NewClient

func NewClient(jsonKey, projectID string) (*bigquery.Client, error)

Types

type Column

type Column struct {
	Name string
	Type string
}

type DataSaver

type DataSaver struct {
	Schema  bigquery.Schema
	DataMap map[string]bigquery.Value
}

func (DataSaver) Save

func (v DataSaver) Save() (row map[string]bigquery.Value, insertID string, err error)

type ReadInput

type ReadInput struct {
	ProjectID string
	DatasetID string
	TableName string
	Client    *bigquery.Client
	Filtering string
}

type ReadOutput

type ReadOutput struct {
	Data []map[string]any `json:"data"`
}

type TableColumns

type TableColumns struct {
	TableName string
	Columns   []Column
}