debeziumio

Documentation

Overview

Package debeziumio contains cross-language functionality for using Debezium (http://kafka.apache.org/). These transforms only work on runners that support cross-language transforms.

Setup

Transforms specified here are cross-language transforms implemented in a different SDK (listed below). During pipeline construction, the Go SDK will need to connect to an expansion service containing information on these transforms in their native SDK.

To use an expansion service, it must be run as a separate process accessible during pipeline construction. The address of that process must be passed to the transforms in this package.

The version of the expansion service should match the version of the Beam SDK being used. For numbered releases of Beam, these expansions services are released to the Maven repository as modules. For development versions of Beam, it is recommended to build and run it from source using Gradle.

Current supported SDKs including expansion service modules

Java:

• Vendored Module: beam-sdks-java-io-debezium-expansion-service
• Run via Gradle: ./gradlew :sdks:java:io:debezium:expansion-service:shadowJar java -jar <path-to-debezium-jar> <port>
• Reference Class: org.apache.beam.io.debezium.DebeziumIO

Index

• func ConnectionProperties(cp []string) readOption
• func ExpansionAddr(expansionAddr string) readOption
• func MaxRecord(r int64) readOption
• func Read(s beam.Scope, username, password, host, port string, ...) beam.PCollection
• type DriverClassName

Functions

func ConnectionProperties

func ConnectionProperties(cp []string) readOption

ConnectionProperties specifies properties of the debezium connection passed as a string with format [propertyName=property;]*

func ExpansionAddr

func ExpansionAddr(expansionAddr string) readOption

ExpansionAddr sets the expansion service address to use for DebeziumIO cross-langauage transform.

func MaxRecord

func MaxRecord(r int64) readOption

MaxRecord specifies maximum number of records to be fetched before stop.

func Read

func Read(s beam.Scope, username, password, host, port string, connectorClass DriverClassName, t reflect.Type, opts ...readOption) beam.PCollection

Read is an external PTransform which reads from Debezium and returns a JSON string. It requires the address of an expansion service for Debezium IO. If both the host and port address are provided as "", an appropriate expansion service will be automatically started; however this is slower than having a persistent expansion service running.

Example:

username := "debezium"
password := "dbz"
host := "localhost"
port := "5432"
connectorClass := debeziumIO.POSTGRESQL
maxrecords := 1
debeziumio.Read(s.Scope("Read from debezium"), expansionAddr, username, password, host, port, connectorClass,
                reflectx.String, debeziumio.MaxRecord(maxrecords), debeziumio.ExpansionAddr("localhost:9000"))

Types

type DriverClassName

type DriverClassName string

DriverClassName is the type for valid and supported Database connectors for Debezium IO.

const (
	// MySQL connector for Debezium
	MySQL DriverClassName = "MySQL"
	// PostgreSQL connector for Debezium
	PostgreSQL = "PostgreSQL"
)