http

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 16, 2024 License: Apache-2.0 Imports: 22 Imported by: 0

README

Conduit Connector for HTTP

The HTTP connector is a Conduit plugin. It provides both, a source and a destination HTTP connectors.

How to build?

Run make build to build the connector's binary.

Testing

Run make test to run all the unit tests.

Source

The HTTP source connector pulls data from the HTTP URL every pollingPeriod, the source adds the params and headers to the request, and sends it to the URL with the specified method from the Configuration. The returned data is used to create an openCDC record and return it.

Note: when using the OPTIONS method, the resulted options will be added to the record's metadata.

Configuration
name description required default value example
url HTTP URL to send requests to. true https://example.com/api/v1
method HTTP method to use in the request, supported methods are (GET,HEAD,OPTIONS). false GET POST
headers HTTP headers to use in the request, comma separated list of : separated pairs. false Authorization:Bearer TOKEN_VALUE,Content-Type:application/xml
params.* parameters to use in the request, use params.* as the config key and specify its value, ex: set "params.id" as "1". false params.query="foobar"
pollingperiod how often the connector will get data from the url, formatted as a time.Duration. false "5m" "5m"
script.parseResponse

The path to a .js file containing the code to parse the response.

The signature of the function needs to be:

function parseResponse(bytes)

where bytes is the original response's raw bytes (i.e. unparsed).

The function needs to return a Response object.

 false /path/to/get_request_data.js

An example script can be found in test/get_request_data.js
script.getRequestData

The path to a .js file containing the code to prepare the request data.

The signature of the function needs to be:

function getRequestData(cfg, previousResponse, position)

where:

  • cfg (a map) is the connector configuration
  • previousResponse (a map) contains data from the previous response (if any), returned by parseResponse
  • position (a byte array) contains the starting position of the connector.

The function needs to return a Request object.

 false /path/to/parse_response.js

An example script can be found in test/parse_response.js

Destination

The HTTP destination connector pushes data from upstream resources to an HTTP URL via Conduit. the destination adds the params and headers to the request, and sends it to the URL with the specified method from the Configuration.

Note: The request Body that will be sent is the value under record.Payload.After, if you want to change the format of that or manipulate the field in any way, please check our Builtin Processors Docs , or check Standalone Processors Docs if you'd like to build your own processor .

Configuration
name description required default value
url Is a Go template expression for the URL used in the HTTP request, using Go templates. The value provided to the template is opencdc.Record, so the template has access to all its fields (e.g. .Position, .Key, .Metadata, and so on). We also inject all template functions provided by sprig to make it easier to write templates. true
method Http method to use in the request, supported methods are (POST,PUT,DELETE,PATCH). false POST
headers Http headers to use in the request, comma separated list of : separated pairs. false
params.* parameters to use in the request, use params.* as the config key and specify its value, ex: set "params.id" as "1". false

Documentation

Overview

Package http is a generated GoMock package.

Package http is a generated GoMock package.

Index

Constants

View Source 
const (
	DestinationConfigHeaders = "headers"
	DestinationConfigMethod  = "method"
	DestinationConfigParams  = "params.*"
	DestinationConfigUrl     = "url"
)
View Source 
const (
	SourceConfigHeaders              = "headers"
	SourceConfigMethod               = "method"
	SourceConfigParams               = "params.*"
	SourceConfigPollingPeriod        = "pollingPeriod"
	SourceConfigScriptGetRequestData = "script.getRequestData"
	SourceConfigScriptParseResponse  = "script.parseResponse"
	SourceConfigUrl                  = "url"
)

Variables

View Source 
var Connector = sdk.Connector{
	NewSpecification: Specification,
	NewSource:        NewSource,
	NewDestination:   NewDestination,
}

Connector combines all constructors for each plugin in one struct.

Functions

func NewDestination 

func NewDestination() sdk.Destination

func NewSource 

func NewSource() sdk.Source

func Specification 

func Specification() sdk.Specification

Specification returns the connector's specification.

Types

type Config 

type Config struct {
	// Http headers to use in the request, comma separated list of : separated pairs
	Headers []string
	// parameters to use in the request, use params.* as the config key and specify its value, ex: set "params.id" as "1".
	Params map[string]string
}

type Destination 

type Destination struct {
	sdk.UnimplementedDestination
	// contains filtered or unexported fields
}

func (*Destination) Configure 

func (d *Destination) Configure(ctx context.Context, cfg config.Config) error

func (*Destination) EvaluateURL added in v0.2.0 

func (d *Destination) EvaluateURL(rec opencdc.Record) (string, error)

func (*Destination) Open 

func (d *Destination) Open(ctx context.Context) error

func (*Destination) Parameters 

func (d *Destination) Parameters() config.Parameters

func (*Destination) Teardown 

func (d *Destination) Teardown(ctx context.Context) error

func (*Destination) Write 

func (d *Destination) Write(ctx context.Context, records []opencdc.Record) (int, error)

type DestinationConfig 

type DestinationConfig struct {
	Config
	// URL is a Go template expression for the URL used in the HTTP request, using Go [templates](https://pkg.go.dev/text/template).
	// The value provided to the template is [opencdc.Record](https://github.com/ConduitIO/conduit-connector-sdk/blob/bfc1d83eb75460564fde8cb4f8f96318f30bd1b4/record.go#L81),
	// so the template has access to all its fields (e.g. .Position, .Key, .Metadata, and so on). We also inject all template functions provided by [sprig](https://masterminds.github.io/sprig/)
	// to make it easier to write templates.
	URL string `json:"url" validate:"required"`
	// Http method to use in the request
	Method string `default:"POST" validate:"inclusion=POST|PUT|DELETE|PATCH"`
}

func (DestinationConfig) Parameters 

func (DestinationConfig) Parameters() map[string]config.Parameter

type MockRequestBuilder added in v0.2.0 

type MockRequestBuilder struct {
	// contains filtered or unexported fields
}

MockRequestBuilder is a mock of requestBuilder interface.

func NewMockRequestBuilder added in v0.2.0 

func NewMockRequestBuilder(ctrl *gomock.Controller) *MockRequestBuilder

NewMockRequestBuilder creates a new mock instance.

func (*MockRequestBuilder) EXPECT added in v0.2.0 

func (m *MockRequestBuilder) EXPECT() *MockRequestBuilderMockRecorder

EXPECT returns an object that allows the caller to indicate expected use.

type MockRequestBuilderMockRecorder added in v0.2.0 

type MockRequestBuilderMockRecorder struct {
	// contains filtered or unexported fields
}

MockRequestBuilderMockRecorder is the mock recorder for MockRequestBuilder.

type MockResponseParser added in v0.2.0 

type MockResponseParser struct {
	// contains filtered or unexported fields
}

MockResponseParser is a mock of responseParser interface.

func NewMockResponseParser added in v0.2.0 

func NewMockResponseParser(ctrl *gomock.Controller) *MockResponseParser

NewMockResponseParser creates a new mock instance.

func (*MockResponseParser) EXPECT added in v0.2.0 

func (m *MockResponseParser) EXPECT() *MockResponseParserMockRecorder

EXPECT returns an object that allows the caller to indicate expected use.

type MockResponseParserMockRecorder added in v0.2.0 

type MockResponseParserMockRecorder struct {
	// contains filtered or unexported fields
}

MockResponseParserMockRecorder is the mock recorder for MockResponseParser.

type MockrequestBuilder added in v0.2.0 

type MockrequestBuilder struct {
	// contains filtered or unexported fields
}

MockrequestBuilder is a mock of requestBuilder interface.

func NewMockrequestBuilder added in v0.2.0 

func NewMockrequestBuilder(ctrl *gomock.Controller) *MockrequestBuilder

NewMockrequestBuilder creates a new mock instance.

func (*MockrequestBuilder) EXPECT added in v0.2.0 

func (m *MockrequestBuilder) EXPECT() *MockrequestBuilderMockRecorder

EXPECT returns an object that allows the caller to indicate expected use.

type MockrequestBuilderMockRecorder added in v0.2.0 

type MockrequestBuilderMockRecorder struct {
	// contains filtered or unexported fields
}

MockrequestBuilderMockRecorder is the mock recorder for MockrequestBuilder.

type MockresponseParser added in v0.2.0 

type MockresponseParser struct {
	// contains filtered or unexported fields
}

MockresponseParser is a mock of responseParser interface.

func NewMockresponseParser added in v0.2.0 

func NewMockresponseParser(ctrl *gomock.Controller) *MockresponseParser

NewMockresponseParser creates a new mock instance.

func (*MockresponseParser) EXPECT added in v0.2.0 

func (m *MockresponseParser) EXPECT() *MockresponseParserMockRecorder

EXPECT returns an object that allows the caller to indicate expected use.

type MockresponseParserMockRecorder added in v0.2.0 

type MockresponseParserMockRecorder struct {
	// contains filtered or unexported fields
}

MockresponseParserMockRecorder is the mock recorder for MockresponseParser.

type Request added in v0.2.0 

type Request struct {
	URL string
}

type Response added in v0.2.0 

type Response struct {
	CustomData map[string]any
	Records    []*jsRecord
}

type Source 

type Source struct {
	sdk.UnimplementedSource
	// contains filtered or unexported fields
}

func (*Source) Ack 

func (s *Source) Ack(ctx context.Context, position opencdc.Position) error

func (*Source) Configure 

func (s *Source) Configure(ctx context.Context, cfg config.Config) error

func (*Source) Open 

func (s *Source) Open(ctx context.Context, pos opencdc.Position) error

func (*Source) Parameters 

func (s *Source) Parameters() config.Parameters

func (*Source) Read 

func (s *Source) Read(ctx context.Context) (opencdc.Record, error)

func (*Source) Teardown 

func (s *Source) Teardown(context.Context) error

type SourceConfig 

type SourceConfig struct {
	Config
	// Http url to send requests to
	URL string `json:"url" validate:"required"`
	// how often the connector will get data from the url
	PollingPeriod time.Duration `json:"pollingPeriod" default:"5m"`
	// Http method to use in the request
	Method string `default:"GET" validate:"inclusion=GET|HEAD|OPTIONS"`

	// The path to a .js file containing the code to prepare the request data.
	// The signature of the function needs to be:
	// `function getRequestData(cfg, previousResponse, position)` where:
	// * `cfg` (a map) is the connector configuration
	// * `previousResponse` (a map) contains data from the previous response (if any), returned by `parseResponse`
	// * `position` (a byte array) contains the starting position of the connector.
	// The function needs to return a Request object.
	GetRequestDataScript string `json:"script.getRequestData"`
	// The path to a .js file containing the code to parse the response.
	// The signature of the function needs to be:
	// `function parseResponse(bytes)` where
	// `bytes` are the original response's raw bytes (i.e. unparsed).
	// The response should be a Response object.
	ParseResponseScript string `json:"script.parseResponse"`
}

func (SourceConfig) Parameters 

func (SourceConfig) Parameters() map[string]config.Parameter

Source Files

View all Source files

Directories

Path Synopsis
cmd
connector

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.