schemaio

Documentation

Overview

Package schemaio contains utilities for constructing cross-language IO wrappers meant to interface with the Java SDK's Schema IOs. Schema IO is an interface for any IO that operates on Beam schema supported elements. Various IOs are implemented via Schema IO, and each implementation requires its own IO wrapper in the Go SDK (for example, JDBC IO or BigQuery IO), and those IO wrappers can make use of these utilities.

For implementation details of Schema IO see https://s.apache.org/schemaio-development-guide.

Index

• func EncodePayload(location string, config any, dataSchema reflect.Type) ([]byte, error)
• func MustEncodePayload(location string, config any, dataSchema reflect.Type) []byte
• type Payload

Functions

func EncodePayload

func EncodePayload(location string, config any, dataSchema reflect.Type) ([]byte, error)

EncodePayload encodes a SchemaIO payload. It takes a location for the SchemaIO's data, an IO-specific configuration struct, and an optional struct representing the Beam schema for the data.

func MustEncodePayload

func MustEncodePayload(location string, config any, dataSchema reflect.Type) []byte

MustEncodePayload encodes a SchemaIO payload. It takes a location for the SchemaIO's data, an IO-specific configuration struct, and an optional struct representing the Beam schema for the data. Unlike EncodePayload, this panics if an error occurs.

Types

type Payload

type Payload struct {
	// Location specifies the location to find the data (for example, a URL to a database).
	Location string `beam:"location"`

	// Config is a Beam schema encoded struct containing configuration details specific to the
	// underlying IO implementation.
	Config []byte `beam:"config"`

	// DataSchema is an optional Beam schema encoded struct representing the schema for data being
	// read or written.
	DataSchema *[]byte `beam:"dataSchema"`
}

Payload is a struct matching the expected cross-language payload of a Schema IO.

This documentation describes the expected usage of each field, but individual IO implementations are free to use this payload differently. For implementation details of those IOs, refer to SchemaIOProvider implementations in the Java SDK.