googlecloudstorage

package
v0.29.0-beta Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 30, 2024 License: MIT Imports: 13 Imported by: 0

README 

---
title: "Google Cloud Storage"
lang: "en-US"
draft: false
description: "Learn about how to set up a VDP Google Cloud Storage component https://github.com/instill-ai/instill-core"
---

The Google Cloud Storage component is a data component that allows users to upload data to Google's Cloud Storage.
It can carry out the following tasks:
- [Upload](#upload)
- [Read Objects](#read-objects)
- [Create Bucket](#create-bucket)

## Release Stage

`Alpha`

## Configuration

The component definition and tasks are defined in the [definition.json](https://github.com/instill-ai/component/blob/main/data/googlecloudstorage/v0/config/definition.json) and [tasks.json](https://github.com/instill-ai/component/blob/main/data/googlecloudstorage/v0/config/tasks.json) files respectively.

## Setup


In order to communicate with Google, the following connection details need to be
provided. You may specify them directly in a pipeline recipe as key-value pairs
within the component's `setup` block, or you can create a **Connection** from
the [**Integration Settings**](https://www.instill.tech/docs/vdp/integration)
page and reference the whole `setup` as `setup:
${connection.<my-connection-id>}`.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| JSON Key File contents (required) | `json-key` | string | Contents of the JSON key file with access to the bucket.  |

</div>




## Supported Tasks

### Upload

Upload data to Google Cloud Storage.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_UPLOAD` |
| Bucket Name (required) | `bucket-name` | string | Name of the bucket to be used for object storage |
| Object Name (required) | `object-name` | string | The name of the object to be created |
| Data (required) | `data` | string | The data to be saved in the object |
</div>






<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Authenticated URL (optional) | `authenticated-url` | string | Only users granted permission can access the object with this link |
| Gsutil URI (optional) | `gsutil-uri` | string | File path to this resource in Cloud Storage |
| Public Access (optional) | `public-access` | boolean | Whether the object is publicly accessible |
| Public URL (optional) | `public-url` | string | Anyone with this link can access the object on the public Internet |
| Upload Status | `status` | string | Status of the upload operation |
</div>

### Read Objects

Read objects from Google Cloud Storage.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_READ_OBJECTS` |
| Bucket Name (required) | `bucket-name` | string | Name of the bucket to be used for object storage |
| Delimiter | `delimiter` | string | The delimiter to use when listing objects |
| Prefix | `prefix` | string | The prefix to use when listing objects |
| Versions | `versions` | boolean | Whether to list all versions of an object |
| Start Offset | `start-offset` | string | The offset to start listing objects from |
| End Offset | `end-offset` | string | The offset to stop listing objects at |
| Include Trailing Delimiter | `include-trailing-delimiter` | boolean | Whether to include the delimiter in the object name |
| Match Glob | `match-glob` | string | Whether to match objects using a glob pattern |
| Include Folders As Prefixes | `include-folders-as-prefixes` | boolean | Whether to include folders as prefixes |
</div>






<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| [Text Objects](#read-objects-text-objects) (optional) | `text-objects` | array[object] | The text objects in the bucket |
| [Image Objects](#read-objects-image-objects) (optional) | `image-objects` | array[object] | The image objects in the bucket |
| [Document Objects](#read-objects-document-objects) (optional) | `document-objects` | array[object] | The document objects in the bucket |
| [Audio Objects](#read-objects-audio-objects) (optional) | `audio-objects` | array[object] | The audio objects in the bucket |
| [Video Objects](#read-objects-video-objects) (optional) | `video-objects` | array[object] | The video objects in the bucket |
</div>

<details>
<summary> Output Objects in Read Objects</summary>

<h4 id="read-objects-text-objects">Text Objects</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Attributes](#read-objects-attributes) | `attributes` | object | The attributes of the object |
| Data | `data` | string | The data of the object |
</div>

<h4 id="read-objects-attributes">Attributes</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Content Disposition | `content-disposition` | string | The content disposition of the object |
| Content Encoding | `content-encoding` | string | The content encoding of the object |
| Content Language | `content-language` | string | The content language of the object |
| Content Type | `content-type` | string | The content type of the object |
| Media Link | `media-link` | string | The media link of the object |
| Metadata | `metadata` | object | The object metadata |
| Object Name | `name` | string | The name of the object |
| Owner | `owner` | string | The owner of the object |
| Size | `size` | integer | The size of the object |
| Storage Class | `storage-class` | string | The storage class of the object |
</div>

<h4 id="read-objects-image-objects">Image Objects</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Attributes](#read-objects-attributes) | `attributes` | object | The attributes of the object |
| Data | `data` | string | The data of the object |
</div>

<h4 id="read-objects-attributes">Attributes</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Content Disposition | `content-disposition` | string | The content disposition of the object |
| Content Encoding | `content-encoding` | string | The content encoding of the object |
| Content Language | `content-language` | string | The content language of the object |
| Content Type | `content-type` | string | The content type of the object |
| Media Link | `media-link` | string | The media link of the object |
| Metadata | `metadata` | object | The object metadata |
| Object Name | `name` | string | The name of the object |
| Owner | `owner` | string | The owner of the object |
| Size | `size` | integer | The size of the object |
| Storage Class | `storage-class` | string | The storage class of the object |
</div>

<h4 id="read-objects-document-objects">Document Objects</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Attributes](#read-objects-attributes) | `attributes` | object | The attributes of the object |
| Data | `data` | string | The data of the object |
</div>

<h4 id="read-objects-attributes">Attributes</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Content Disposition | `content-disposition` | string | The content disposition of the object |
| Content Encoding | `content-encoding` | string | The content encoding of the object |
| Content Language | `content-language` | string | The content language of the object |
| Content Type | `content-type` | string | The content type of the object |
| Media Link | `media-link` | string | The media link of the object |
| Metadata | `metadata` | object | The object metadata |
| Object Name | `name` | string | The name of the object |
| Owner | `owner` | string | The owner of the object |
| Size | `size` | integer | The size of the object |
| Storage Class | `storage-class` | string | The storage class of the object |
</div>

<h4 id="read-objects-audio-objects">Audio Objects</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Attributes](#read-objects-attributes) | `attributes` | object | The attributes of the object |
| Data | `data` | string | The data of the object |
</div>

<h4 id="read-objects-attributes">Attributes</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Content Disposition | `content-disposition` | string | The content disposition of the object |
| Content Encoding | `content-encoding` | string | The content encoding of the object |
| Content Language | `content-language` | string | The content language of the object |
| Content Type | `content-type` | string | The content type of the object |
| Media Link | `media-link` | string | The media link of the object |
| Metadata | `metadata` | object | The object metadata |
| Object Name | `name` | string | The name of the object |
| Owner | `owner` | string | The owner of the object |
| Size | `size` | integer | The size of the object |
| Storage Class | `storage-class` | string | The storage class of the object |
</div>

<h4 id="read-objects-video-objects">Video Objects</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Attributes](#read-objects-attributes) | `attributes` | object | The attributes of the object |
| Data | `data` | string | The data of the object |
</div>

<h4 id="read-objects-attributes">Attributes</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Content Disposition | `content-disposition` | string | The content disposition of the object |
| Content Encoding | `content-encoding` | string | The content encoding of the object |
| Content Language | `content-language` | string | The content language of the object |
| Content Type | `content-type` | string | The content type of the object |
| Media Link | `media-link` | string | The media link of the object |
| Metadata | `metadata` | object | The object metadata |
| Object Name | `name` | string | The name of the object |
| Owner | `owner` | string | The owner of the object |
| Size | `size` | integer | The size of the object |
| Storage Class | `storage-class` | string | The storage class of the object |
</div>
</details>

### Create Bucket

Read objects from Google Cloud Storage.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_CREATE_BUCKET` |
| Bucket Name (required) | `bucket-name` | string | Name of the bucket to be used for object storage |
| Project ID (required) | `project-id` | string | The project ID to create the bucket in |
| Location | `location` | string | The location to create the bucket in |
</div>






<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Result (optional) | `result` | string | The result of the operation |
| Bucket Name (optional) | `name` | string | The name of the bucket |
| Location (optional) | `location` | string | The location of the bucket |
</div>

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Init 

func Init(bc base.Component) *component

func NewClient 

func NewClient(jsonKey string) (*storage.Client, error)

Types

type Attributes 

type Attributes struct {
	Name               string            `json:"name"`
	ContentType        string            `json:"content-type"`
	ContentLanguage    string            `json:"content-language"`
	Owner              string            `json:"owner"`
	Size               int64             `json:"size"`
	ContentEncoding    string            `json:"content-encoding"`
	ContentDisposition string            `json:"content-disposition"`
	MD5                []byte            `json:"md5"`
	MediaLink          string            `json:"media-link"`
	Metadata           map[string]string `json:"metadata"`
	StorageClass       string            `json:"storage-class"`
}

type AudioObject 

type AudioObject struct {
	Data       string     `json:"data"`
	Attributes Attributes `json:"attributes"`
}

type CreateBucketInput 

type CreateBucketInput struct {
	BucketName string `json:"bucket-name"`
	ProjectID  string `json:"project-id"`
	Location   string `json:"location"`
}

type CreateBucketOutput 

type CreateBucketOutput struct {
	Result   string `json:"result"`
	Name     string `json:"name"`
	Location string `json:"location"`
}

type DocumentObject 

type DocumentObject struct {
	Data       string     `json:"data"`
	Attributes Attributes `json:"attributes"`
}

type ImageObject 

type ImageObject struct {
	Data       string     `json:"data"`
	Attributes Attributes `json:"attributes"`
}

type ReadInput 

type ReadInput struct {
	BucketName               string
	Delimiter                string
	Prefix                   string
	Versions                 bool
	StartOffset              string
	EndOffset                string
	IncludeTrailingDelimiter bool
	MatchGlob                string
	IncludeFoldersAsPrefixes bool
}

type ReadOutput 

type ReadOutput struct {
	TextObjects     []TextObject     `json:"text-objects"`
	ImageObjects    []ImageObject    `json:"image-objects"`
	DocumentObjects []DocumentObject `json:"document-objects"`
	AudioObjects    []AudioObject    `json:"audio-objects"`
	VideoObjects    []VideoObject    `json:"video-objects"`
}

type TextObject 

type TextObject struct {
	Data       string     `json:"data"`
	Attributes Attributes `json:"attributes"`
}

type VideoObject 

type VideoObject struct {
	Data       string     `json:"data"`
	Attributes Attributes `json:"attributes"`
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.