deilephila

README

Deilephila

A go service that sync MongoDB to Neo4j (but can easily adapt to other databases).

How it Works

Deilephila implements the infrastructure for online sync databases fast, without assuming anything about the databases' scheme or the mapping function.
Each driver supplies basic functionality such as Init and Read.
In the beginning we start listening to events from the src driver. Upon a new event, we send it to one of the parser workers, this worker responsible to handle this event.

In order to run the application write your custom configurations in configTemplate.yml. Then, write your custom Map function that receives a src event type and handle it.

| Src DB | --> | listening to events | -> | parser workers | -> | Dst DB |

Assumption: Order of events does not matter. We prefer performance over consistency, also note you can handle this in your transform function.

Quick start guide

1. Set configs following the template: configTemplate.yml
2. Write your custom callback: just implement the function Map in plugin/pluginTemplate.go
3. Pull docker image: docker pull ofirbirka/deilephila
4. Run: docker run -v /local/path/to/config:/etc/config -v /local/path/to/plugin.go:/usr/src/deilephila/plugin/pluginTemplate.go --network=host deilephila

Tests

See here

Contributing

Add a new driver

1. Implement a new driver class under drivers folder that implement the Driver interface
2. Add a test driver in test/drivers/ and tests under test/integration
3. Compile and run tests

Push new image

docker build -t deilephila .
docker tag deilephila ofirbirka/deilephila:v1
docker push ofirbirka/deilephila:v1

Documentation

Index

• type Config
  • func ReadConfig(configPath string) *Config
• type Deilephila
  • func New(config *Config) *Deilephila
  • func (t *Deilephila) Run(ctx context.Context)
• type Driver
• type Mapper
• type SyncOptions

Types

type Config

type Config struct {
	Src  interface{} `yaml:"src"`
	Dst  interface{} `yaml:"dst"`
	Sync SyncOptions `yaml:"sync"`
}

func ReadConfig

func ReadConfig(configPath string) *Config

Parse yaml file to Config struct

type Deilephila

type Deilephila struct {
	// contains filtered or unexported fields
}

func New

func New(config *Config) *Deilephila

func (*Deilephila) Run

func (t *Deilephila) Run(ctx context.Context)

type Driver

type Driver interface {
	Init(config []byte)
	Read(context.Context, chan mongodb.ChangeEvent) error
}

type Mapper

type Mapper func(event mongodb.ChangeEvent, srcDriver Driver, dstDriver Driver) error

type SyncOptions

type SyncOptions struct {
	Workers     int `yaml:"workers"`
	MapFunction Mapper
}