README
¶
This repository only contains the public Go APIs used by the Blacksmith platform.
Blacksmith
After a few years we decided to crystallize all of our data engineering best practices into a product, on top of which organizations can layer cloud-assisted data solutions.
Blacksmith is a platform specifically designed for data engineering teams. It allows you to build reliable data pipelines in a consistent way. Data extraction can happen on HTTP requests, CRON schedules, CDC notifications, and Pub / Sub messages. Then, jobs can be created against any kind of destination, such as HTTP APIs or data warehouses.
The goal of Blacksmith is to address as many pain points as possible engineering teams encounter while working on data solutions.
Any team that is building — or think about building — a complete data pipeline knows the tremendous amount of work needed to properly accomplish this mission. Think of Blacksmith as the central piece of your data engineering workflow, leading you to save months of customized and professional ETL work.
Product offerings
Blacksmith is available in two editions:
- Blacksmith Standard Edition addresses the technical complexity of data engineering. It is and will always be free.
- Blacksmith Enterprise Edition addresses the complexity of collaboration and governance across multi-team and multi-scope data solutions.
- Compare Editions
Links
- Learn more on Nunchi website
- Sample application using Blacksmith on GitHub
- API reference on Go developer portal
Professional services
Along consulting and training, we provide different product offerings as well as different levels of support.
License
Repository licensed under the Apache License, Version 2.0.
By downloading, installing, and using Blacksmith, you agree to the Blacksmith Terms and Conditions.
Documentation
¶
Overview ¶
Package blacksmith is the package to create applications on top of the Blacksmith platform.
Blacksmith is an on-premise platform specifically designed for data engineering teams. It allows you to build reliable data pipelines in a consistent way. Whether you are collecting data from HTTP APIs, CRON tasks, or CDC notifications and whether you are loading this data in realtime to third-party services or using a specific schedule to data warehouses.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Options ¶
type Options struct {
// Logger allows you to use a logrus Logger across all Blacksmith adapters and
// the application built on top of it.
Logger *logrus.Logger
// Supervisor is the options passed to use the supervisor adapter.
// The supervisor is optional.
Supervisor *supervisor.Options
// Wanderer is the options passed to use the wanderer adapter.
// The wanderer is optional.
Wanderer *wanderer.Options
// Store is the options passed to use the store adapter.
Store *store.Options
// PubSub is the options passed to use the pubsub adapter.
// The pusub is optional.
PubSub *pubsub.Options
// Gateway is the options passed to use the gateway service.
Gateway *service.Options
// Scheduler is the options passed to use the scheduler service.
Scheduler *service.Options
// Sources is a slice of options passed to create sources.
Sources []*source.Options
// Destinations is a slice of options passed to create destinations.
Destinations []*destination.Options
}
Options is the options a user can pass to create a new application.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
adapter
|
|
|
pubsub
Package pubsub provides the development kit for working with Publish / Subscribe mechanism between the gateway and scheduler services.
|
Package pubsub provides the development kit for working with Publish / Subscribe mechanism between the gateway and scheduler services. |
|
store
Package store provides the development kit for working with a database-as-a-queue, so the jobs queue can be persisted in a datastore.
|
Package store provides the development kit for working with a database-as-a-queue, so the jobs queue can be persisted in a datastore. |
|
supervisor
Package supervisor provides the development kit for running Blacksmith applications in a distributed environment.
|
Package supervisor provides the development kit for running Blacksmith applications in a distributed environment. |
|
wanderer
Package wanderer provides the development kit for working with database migrations.
|
Package wanderer provides the development kit for working with database migrations. |
|
Package flow is a middleman between sources and destinations.
|
Package flow is a middleman between sources and destinations. |
|
destination
Package destination provides the development kit for working with third-party services that will receive events from Blacksmith.
|
Package destination provides the development kit for working with third-party services that will receive events from Blacksmith. |
|
source
Package source provides the development kit for working with Blacksmith SDKs, cloud services, databases, or any kind of application able to send data or push notifications.
|
Package source provides the development kit for working with Blacksmith SDKs, cloud services, databases, or any kind of application able to send data or push notifications. |
|
helper
|
|
|
errors
Package errors provides the error interface used by the Blacksmith SDK and by the adapters.
|
Package errors provides the error interface used by the Blacksmith SDK and by the adapters. |
|
logger
Package logger contains tools such as defaults and hooks for working with logrus (github.com/sirupsen/logrus) correctly inside Blacksmith.
|
Package logger contains tools such as defaults and hooks for working with logrus (github.com/sirupsen/logrus) correctly inside Blacksmith. |
|
rest
Package rest provides utilities shared across the gateway and scheduler adapters to build REST APIs.
|
Package rest provides utilities shared across the gateway and scheduler adapters to build REST APIs. |
|
sqlike
Package sqlike provides a suite of utilities to work with SQL-like datastores.
|
Package sqlike provides a suite of utilities to work with SQL-like datastores. |
|
gateway
Package gateway provides the development kit for working with incoming events.
|
Package gateway provides the development kit for working with incoming events. |
|
scheduler
Package scheduler provides the development kit for working with a scheduler that receives events from the gateway and pubsub packages and, used in tandem with the store package, is in charge of the reliability of the event delivery to destinations.
|
Package scheduler provides the development kit for working with a scheduler that receives events from the gateway and pubsub packages and, used in tandem with the store package, is in charge of the reliability of the event delivery to destinations. |
|
Package version provides the version details of the Blacksmith SDK alongside the Go version used to compile the CLI and the application.
|
Package version provides the version details of the Blacksmith SDK alongside the Go version used to compile the CLI and the application. |