file

README

Overview

The Conduit File plugin provides both source and destination connectors, allowing for a file to be either a source or a destination in a Conduit pipeline.

How it works

The Source connector listens for changes appended to the source file and sends records with the changes. The Destination connector receives records and writes them to a file.

Source

The Source connector only cares to have a valid path, even if the file doesn't exist, it will still run and wait until a file with the configured name is there, then it will start listening to changes and sending records.

Destination

The Destination connector will create the file if it doesn't exist, and records with changes will be appended to the destination file when received.

How to build it

Run make build-file-plugin.

Configuration

name	part of	description	required	default value
path	destination, source	The path to the file	true

Limitations

• The Source connector only detects appended changes to the file, so it doesn't detect deletes or edits.
• The connectors can only access local files on the machine where Conduit is running. So, running Conduit on a server means it can't access a file on your local machine.
• Currently, only works reliably with text files (may work with non-text files, but not guaranteed)

Documentation

Index

• Constants
• type Destination
  • func (c *Destination) Open(ctx context.Context, config plugins.Config) error
  • func (c *Destination) Teardown() error
  • func (c *Destination) Validate(cfg plugins.Config) error
  • func (c *Destination) Write(ctx context.Context, r record.Record) (record.Position, error)
• type Source
  • func (c *Source) Ack(ctx context.Context, position record.Position) error
  • func (c *Source) Open(ctx context.Context, config plugins.Config) error
  • func (c *Source) Read(ctx context.Context, p record.Position) (record.Record, error)
  • func (c *Source) Teardown() error
  • func (c *Source) Validate(cfg plugins.Config) error
• type Spec
  • func (s Spec) Specify() (plugins.Specification, error)

Constants

const (
	// ConfigPath is the config name for the path to the file.
	ConfigPath = "path"
)

Types

type Destination

type Destination struct {
	Scanner *bufio.Scanner
	File    *os.File
	Config  map[string]string
}

Destination connector

func (*Destination) Open

func (c *Destination) Open(ctx context.Context, config plugins.Config) error

func (*Destination) Teardown

func (c *Destination) Teardown() error

func (*Destination) Validate

func (c *Destination) Validate(cfg plugins.Config) error

func (*Destination) Write

func (c *Destination) Write(ctx context.Context, r record.Record) (record.Position, error)

type Source

type Source struct {
	Tail   *tail.Tail
	Config map[string]string
	// contains filtered or unexported fields
}

Source connector

func (*Source) Ack

func (c *Source) Ack(ctx context.Context, position record.Position) error

func (*Source) Open

func (c *Source) Open(ctx context.Context, config plugins.Config) error

func (*Source) Read

func (c *Source) Read(ctx context.Context, p record.Position) (record.Record, error)

func (*Source) Teardown

func (c *Source) Teardown() error

func (*Source) Validate

func (c *Source) Validate(cfg plugins.Config) error

type Spec

type Spec struct{}

func (Spec) Specify

func (s Spec) Specify() (plugins.Specification, error)

Specify returns the Plugin's Specification