Documentation
¶
Overview ¶
Package gcp implements streaming connectors for Google Cloud Platform.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type StorageObject ¶
type StorageObject struct {
// Key is the object name including any subdirectories.
// For example, "directory/file.json".
Key string
// Data is an [io.ReadCloser] representing the binary content of the object.
Data io.ReadCloser
}
StorageObject contains details of the GCP Storage object.
type StorageSink ¶
type StorageSink struct {
// contains filtered or unexported fields
}
StorageSink represents the Google Cloud Storage sink connector.
func NewStorageSink ¶
func NewStorageSink(ctx context.Context, client *storage.Client, config *StorageSinkConfig, logger *slog.Logger) *StorageSink
NewStorageSink returns a new StorageSink. Incoming elements are expected to be of the StorageObject type. These will be uploaded to the configured bucket using their key field as the path.
func (*StorageSink) In ¶
func (s *StorageSink) In() chan<- any
In returns the input channel of the StorageSink connector.
type StorageSinkConfig ¶
type StorageSinkConfig struct {
// The name of the GCP Storage bucket to write to.
Bucket string
// The number of concurrent workers to use when writing data to GCP Storage.
// The default is 1.
Parallelism int
}
StorageSinkConfig represents the configuration options for the GCP Storage sink connector.
type StorageSource ¶
type StorageSource struct {
// contains filtered or unexported fields
}
StorageSource represents the Google Cloud Storage source connector.
func NewStorageSource ¶
func NewStorageSource(ctx context.Context, client *storage.Client, config *StorageSourceConfig, logger *slog.Logger) *StorageSource
NewStorageSource returns a new StorageSource. The connector reads all objects within the configured path and transmits them as an StorageObject through the output channel.
func (*StorageSource) Out ¶
func (s *StorageSource) Out() <-chan any
Out returns the output channel of the StorageSource connector.
func (*StorageSource) Via ¶
func (s *StorageSource) Via(operator streams.Flow) streams.Flow
Via streams data to a specified operator and returns it.
type StorageSourceConfig ¶
type StorageSourceConfig struct {
// The name of the storage bucket to read from.
Bucket string
// The path within the bucket to use. If empty, the root of the
// bucket will be used.
Prefix string
// Delimiter can be used to restrict the results to only the objects in
// the given "directory". Without the delimiter, the entire tree under
// the prefix is returned.
Delimiter string
}
StorageSourceConfig represents the configuration options for the GCP Storage source connector.
Source Files