Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func New ¶
func New(env *Options) destination.Destination
New returns a valid Blacksmith destination.Destination for a blob storage.
Types ¶
type Blob ¶
type Blob struct {
// contains filtered or unexported fields
}
Blob implements the Blacksmith destination.Destination interface for working with blob storages.
func (*Blob) Actions ¶
func (d *Blob) Actions() map[string]destination.Action
Actions return a list of actions the destination Blob is able to handle.
func (*Blob) Init ¶
func (d *Blob) Init(tk *destination.Toolkit) error
Init is part of the destination.WithHooks interface. It allows to properly open the connection with the bucket. It is called when starting the scheduler service.
func (*Blob) Options ¶
func (d *Blob) Options() *destination.Options
Options returns common destination options for a blob storage. They will be shared across every actions of this destination, except when overridden.
type Driver ¶
type Driver string
Driver is a custom type allowing the user to only pass supported drivers when creating the destination.
var DriverAWSS3 Driver = "aws/s3"
DriverAWSS3 is used to leverage AWS S3 as the destination's driver.
Environment variables:
- AWS_ACCESS_KEY_ID (required)
- AWS_SECRET_ACCESS_KEY (required)
- AWS_REGION
var DriverAzureBlob Driver = "azure/blob"
DriverAzureBlob is used to leverage Azure Blob Storage as the destination's driver.
Environment variables:
- AZURE_STORAGE_ACCOUNT (required)
- AZURE_STORAGE_KEY || AZURE_STORAGE_SAS_TOKEN (required)
var DriverGoogleStorage Driver = "google/storage"
DriverGoogleStorage is used to leverage Google Cloud Storage as the destination's driver.
Environment variables:
- GOOGLE_APPLICATION_CREDENTIALS (required)
type Options ¶
type Options struct {
// Realtime indicates if the pubsub adapter of the Blacksmith application shall
// be used to load events to the destination in realtime or not. When false, the
// Interval will be used.
Realtime bool
// Interval represents an interval or a CRON string at which a job shall be
// loaded to the destination. It is used as the time-lapse between retries in
// case of a job failure.
//
// Defaults to "@every 1h".
Interval string
// MaxRetries indicates the maximum number of retries per job the scheduler will
// attempt to execute before it succeed. When the limit is reached, the job is
// marked as "discarded".
//
// Defaults to 72.
MaxRetries uint16
// Name indicates the identifier of the bucket to use in Blacksmith. The computed
// name is "blob(<name>)". This does not have any consequences on the bucket's
// name used in the cloud provider.
//
// Example: "mybucket"
// Required.
Name string
// Driver is the driver to leverage for using this destination.
//
// Required.
Driver Driver
// Connection is the driver's specific connection string to use for writing
// into the bucket.
//
// Format for AWS S3: "<bucket>"
// Format for Azure Blob Storage: "<container>"
// Format for Google Cloud Storage: "<bucket>"
Connection string
// Params can be used to add specific configuration per driver.
//
// Supported fields for AWS S3:
// url.Values{
// "region": {"<region>"}, // Required if environment variable 'AWS_REGION' is not set.
// }
Params url.Values
}
Options is the options the destination can take as an input to be configured.
type Write ¶
type Write struct {
Filename string `json:"filename"`
Content []byte `json:"content"`
// contains filtered or unexported fields
}
Write implements the Blacksmith destination.Action interface for the action "write". It holds the complete job's structure to load into the destination.
func (Write) Load ¶
func (a Write) Load(tk *destination.Toolkit, queue *store.Queue, then chan<- destination.Then)
Load is the function being run by the scheduler to load the data into the destination. It is in charge of the "L" in the ETL process.
func (Write) Marshal ¶
func (a Write) Marshal(tk *destination.Toolkit) (*destination.Job, error)
Marshal is the function being run when the action receives data into the Write receiver. It allows to transform and enrich the data before saving it in the store adapter.
func (Write) Schedule ¶
func (a Write) Schedule() *destination.Schedule
Schedule allows the action to override the schedule options of its destination. Do not override.
Source Files