destination

Documentation

Overview

Package destination provides the development kit for working with third-party services that will receive events from Blacksmith. This is used for analytics, marketing, advertising, data warehousing, security scanning, etc.

Index

• Variables
• type Action
• type Actions
• type Destination
• type Job
• type Options
• type Schedule
• type Then
• type Toolkit
• type WithHooks

Variables

var Defaults = &Options{
	DefaultSchedule: &Schedule{
		Realtime:   false,
		Interval:   "@every 1h",
		MaxRetries: 72,
	},
}

Defaults are the defaults options set for a destination. When not set, these values will automatically be applied.

We set a hourly interval for 3 days so it give time to teams to be aware of the failures and debug the destination if needed.

var InterfaceAction = "destination/action"

InterfaceAction is the string representation for the destination's action interface.

var InterfaceDestination = "destination"

InterfaceDestination is the string representation for the destination interface.

Types

type Action

type Action interface {

	// String returns the string representation of the destination's action.
	//
	// Example: "identify"
	String() string

	// Schedule represents a schedule at which an action should run. When returning
	// nil, the parent destination's schedule is applied.
	Schedule() *Schedule

	// Marshal is in charge of marshalling the received data for the action. It
	// can be in charge of the "T" in the ETL process if needed: it can Transform
	// the data of the pointer receiver originally passed by sources' triggers or
	// destinations' actions. It must return a job including the context and data
	// as JSON marshaled values.
	//
	// If the context in the returned job is nil, the one from the event will
	// automatically be applied.
	//
	// If the function returns an error, the event can not be considered as transformed.
	// Therefore, no jobs will be created and the action will never run.
	Marshal(*Toolkit) (*Job, error)

	// Load is in charge of the "L" in the ETL process: it Loads the data to the
	// destination's endpoint. It is executed either on a schedule basis or in realtime
	// when applicable.
	//
	// The queue only includes received events triggering the action. The jobs inside
	// each event are therefore specific to this action only.
	//
	// When desired, the function can return a list of destinations' actions (in
	// Then) to run depending on on the status of the current job. Every jobs will
	// then be processed by the scheduler, respecting the scheduling options of
	// each one.
	Load(*Toolkit, *store.Queue, chan<- Then)
}

Action represents a specific action to run against a destination.

A new action can be generated using the Blacksmith CLI:

$ blacksmith generate action --name <name> [--path <path>] [--migrations]

type Actions

type Actions map[string][]Action

Actions is used to return a slice of Action grouped by their destination name. This is used by the package flow when creating flow to distribute data from triggers to actions.

type Destination

type Destination interface {

	// String returns the string representation of the destination.
	//
	// Example: "zendesk"
	String() string

	// Options returns the options originally passed to the Options struct. This
	// can be used to validate and override user's options if necessary.
	Options() *Options

	// Actions returns a list of actions the destination can handle. Destinations'
	// actions are run from sources' triggers and can also be triggered by other
	// destinations' actions. When a destination's action is called, it is
	// represented as a "job" in the platform.
	Actions() map[string]Action
}

Destination is the interface used to load events to third-party services. Those can be of any kind, such as APIs or databases.

A new destination can be generated using the Blacksmith CLI:

$ blacksmith generate destination --name <name> [--path <path>] [--migrations]

type Job

type Job struct {

	// Version is the version number of the destination used by a flow when executed.
	//
	// Examples: "v1.0", "2020-10-01"
	Version string `json:"version,omitempty"`

	// Context is a dictionary of information that provides useful context about an
	// event. The context should be used inside every events for consistency.
	//
	// Note: It must be a valid JSON since it will be used using encoding/json Marshal
	// and Unmarshal functions.
	Context []byte `json:"context"`

	// Data is the byte representation of the data sent by the event.
	//
	// Note: It must be a valid JSON since it will be used using encoding/json Marshal
	// and Unmarshal functions.
	Data []byte `json:"data"`

	// SentAt allows you to keep track of the timestamp when the event was originally
	// sent.
	SentAt *time.Time `json:"sent_at,omitempty"`
}

Job represents the fields an action must fill when being loaded into the destination.

type Options

type Options struct {

	// Versions is a collection of supported versions for a destination. The value
	// of each version is its deprecation date. It must be set to an empty time.Time
	// when the version is still maintained.
	//
	// When nil or empty, versioning is disabled for the destination.
	//
	// Note: Feature only available in Blacksmith Enterprise Edition.
	Versions map[string]time.Time `json:"versions,omitempty"`

	// DefaultVersion is the default version to apply if the version is not set by
	// a flow when executing an action. It must be one of the versions supported in
	// Versions.
	//
	// Note: Feature only available in Blacksmith Enterprise Edition.
	DefaultVersion string `json:"default_version,omitempty"`

	// DefaultSchedule represents a schedule at which a destination's action should
	// run. This value can be overridden by the underlying destination if necessary
	// so the user does not make any scheduling mistake. This value can also be
	// overridden by each destination action to benefit a per action basis schedule.
	DefaultSchedule *Schedule `json:"schedule"`
}

Options is the options a user can pass to use a destination.

type Schedule

type Schedule struct {

	// Realtime indicates if the pubsub adapter of the Blacksmith application shall
	// be used to load events to the destination in realtime or not. When false, the
	// Interval will be used.
	Realtime bool `json:"realtime"`

	// Interval represents an interval or a CRON string at which a job shall be
	// loaded to the destination. It is used as the time-lapse between retries in
	// case of a job failure.
	Interval string `json:"interval"`

	// MaxRetries indicates the maximum number of retries per job the scheduler will
	// attempt to execute for each job. When the limit is reached, the job is marked
	// as "discarded".
	MaxRetries uint16 `json:"max_retries"`
}

Schedule represents a schedule at which a destination's action should run. SaaS APIs could be used in realtime whereas data warehouses shall be used only a few times per day.

type Then

type Then struct {

	// Jobs is the list of the job IDs being processed. It informs the scheduler
	// of the status of the desired jobs.
	//
	// When nil or empty, all jobs from the queue will be affected by the result.
	// This allows to either load the data entry-per-entry or in batch if the
	// destination allows it. If a job ID is not returned, the scheduler will not
	// be aware of its status and will mark it as "unknown".
	Jobs []string `json:"jobs"`

	// Error is the error encountered when loading the data into the destination's
	// action.
	//
	// When not nil the related jobs will be either marked as "failed" or "discarded"
	// given the max retries of the action. OnFailed or OnDiscarded will automatically
	// be applied by the scheduler as jobs to be executed. When Error is nil the jobs
	// are marked as "succeeded" and OnSucceeded will be applied.
	Error error `json:"error"`

	// ForceDiscard manually marks a job as discarded. It is useful if you know it
	// is impossible for the job to succeed even after multiple retries.
	//
	// When set, Error must not be nil.
	ForceDiscard bool `json:"is_force_discarded"`

	// List of destinations actions to run in case the job has succeeded.
	OnSucceeded []Action `json:"on_succeeded,omitempty"`

	// List of destinations actions to run in case the job has failed.
	OnFailed []Action `json:"on_failed,omitempty"`

	// List of destinations actions to run in case the job has been discarded.
	OnDiscarded []Action `json:"on_discarded,omitempty"`
}

Then allows to inform the scheduler of job status and to execute other actions from the same destination depending on the job status.

type Toolkit

type Toolkit struct {

	// Logger gives access to the logrus Logger passed in options when creating the
	// Blacksmith application.
	Logger *logrus.Logger

	// Service represents the instance of the service registered in the supervisor
	// and currently processing the action. It is an instance of the gateway service
	// when Marshaling the action, and an instance of the scheduler service when
	// Loading the action into the destination.
	//
	// Note: This is nil when there is no supervisor adapter configured.
	Service *supervisor.Service

	// EventID is the unique identifier of the event generated by the gateway and
	// that is related to the job being marshaled.
	//
	// Note: This is not applicable when using the Load function, since it receives
	// a queue of events and related jobs to execute.
	//
	// Example: "1UYc8EebLqCAFMOSkbYZdJwNLAJ"
	EventID string

	// JobID is the unique identifier of the job generated by the scheduler and
	// that is being marshaled.
	//
	// Note: This is not applicable when using the Load function, since it receives
	// a queue of events and related jobs to execute.
	//
	// Example: "1UYc8EebLqCAFMOSkbYZdJwNLAJ"
	JobID string
}

Toolkit contains a suite of utilities and data to help the user successfully run the destination functions.

type WithHooks

type WithHooks interface {

	// Init lets you initialize a destination or an action. It is useful when
	// initialization is necessary, such as opening a connection pool with the
	// destination.
	//
	// Init is called when starting the scheduler service or before running migrations.
	// If an error is returned, the running process will try to gracefully shutdown.
	//
	// The Init function of a destination will always be executed before the ones
	// of its actions. Therefore, the Init function of an action will always be
	// executed after the one of its destination.
	//
	// Note: EventID and JobID in Toolkit will always be empty.
	Init(*Toolkit) error

	// Shutdown lets you gracefully shutdown a destination or an action. It is
	// useful when shutting down is necessary, such as closing a connection pool
	// with the destination.
	//
	// Shutdown is called when shutting down the scheduler service or after running
	// migrations. If an error is returned, it will only be logged and the running
	// process will continue its shutdown.
	//
	// The Shutdown function of a destination will always be executed after the ones
	// of its actions. Therefore, the Shutdown function of an action will always be
	// executed before the one of its destination.
	//
	// Note: EventID and JobID in Toolkit will always be empty.
	Shutdown(*Toolkit) error
}

WithHooks can be implemented by destinations and actions to add custom logic when the scheduler service is starting and shutting down, or before and after running migrations.