Documentation
¶
Overview ¶
Package aws implements streaming connectors for Amazon Web Services.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type S3Object ¶
type S3Object struct {
// Key is the object name including any subdirectories.
// For example, "directory/file.json".
Key string
// Data is an [io.Reader] representing the binary content of the object.
// This can be a file, a buffer, or any other type that implements the
// io.Reader interface.
Data io.Reader
}
S3Object contains details of the S3 object.
type S3Sink ¶
type S3Sink struct {
// contains filtered or unexported fields
}
S3Sink represents the AWS S3 sink connector.
func NewS3Sink ¶
func NewS3Sink(ctx context.Context, client *s3.Client, config *S3SinkConfig, logger *slog.Logger) *S3Sink
NewS3Sink returns a new S3Sink. Incoming elements are expected to be of the S3Object type. These will be uploaded to the configured bucket using their key field as the path.
func (*S3Sink) AwaitCompletion ¶
func (s *S3Sink) AwaitCompletion()
AwaitCompletion blocks until the S3Sink connector has processed all the received data.
type S3SinkConfig ¶
type S3SinkConfig struct {
// The name of the S3 bucket to write to.
Bucket string
// The number of concurrent workers to use when writing data to S3.
// The default is 1.
Parallelism int
}
S3SinkConfig represents the configuration options for the S3 sink connector.
type S3Source ¶
type S3Source struct {
// contains filtered or unexported fields
}
S3Source represents the AWS S3 source connector.
func NewS3Source ¶
func NewS3Source(ctx context.Context, client *s3.Client, config *S3SourceConfig, logger *slog.Logger) *S3Source
NewS3Source returns a new S3Source. The connector reads all objects within the configured path and transmits them as an S3Object through the output channel.
type S3SourceConfig ¶
type S3SourceConfig struct {
// The name of the S3 bucket to read from.
Bucket string
// The path within the bucket to use. If empty, the root of the
// bucket will be used.
Path string
// The number of concurrent workers to use when reading data from S3.
// The default is 1.
Parallelism int
// The size of chunks in bytes to use when reading data from S3.
// The default is 5 MB.
ChunkSize int
}
S3SourceConfig represents the configuration options for the S3 source connector.
Source Files