mongoimport

README

Mongoimport

CLI and go library for importing data from CSV, JSON or XML files into MongoDB.

go run github.com/romnn/mongoimport/cmd/mongoimport --db-user=root --db-password=example csv <path-to-csv-files>

You can also download pre built binaries from the releases page.

For a list of options, run

go run github.com/romnn/mongoimport/cmd/mongoimport csv --help

Usage as a library

Using the tool as a standalone CLI tool is great for quick loading of a few files. However, you might need more fine-grained control over what files are imported into which collection or perform additional pre/post processing (e.g. parsing timestamps). For this use case, we offer a very extensivle and modular API for configuring your imports.

import "github.com/romnn/mongoimport"

// example t.b.a

For more examples, see examples/.

Development

All commits are automatically built and tested via github actions. In order to pass the required checks it is strongly recommended to install the repositories pre commit hooks (assuming you are in the repositories root):

pip install pre-commit invoke bump2version
pre-commit install

The pre commit hooks will run a number of go tools. Try to run pre-commit run --all-files and check for missing tools. You might need to install some of those:

go get -u golang.org/x/lint/golint
go get -u github.com/fzipp/gocyclo

In order to be found, make sure to include $GOPATH/bin in your $PATH ($GOPATH has to be set of course like export GOPATH="$HOME/go for example).

Before committing, pre-commit is run to make sure all checks pass!

Documentation

Index

• Constants
• type Datasource
• type Import
  • func (i *Import) Start() (ImportResult, error)
• type ImportJob
• type ImportResult
  • func (ir ImportResult) Summary() string
• type LoggableResult
• type MongoConnection
  • func (c *MongoConnection) Client() (*mongo.Client, error)
• type Options
• type PartialResult
  • func (ir PartialResult) Summary() string
• type PostLoadHook
• type PreDumpHook
• type SourceResult
  • func (ir SourceResult) Summary() string
• type UpdateFilterHook

Constants

const Version = "0.1.12"

Version is incremented using bump2version

Types

type Datasource

type Datasource struct {
	Options
	Disabled     bool
	Description  string
	FileProvider files.FileProvider
	// contains filtered or unexported fields
}

Datasource ...

type Import

type Import struct {
	Options
	Connection     *MongoConnection
	Sources        []*Datasource
	MaxParallelism int
	// contains filtered or unexported fields
}

Import ...

func (*Import) Start

func (i *Import) Start() (ImportResult, error)

Start ...

type ImportJob

type ImportJob struct {
	Source             *Datasource
	Loader             *loaders.Loader
	File               string
	InsertionBatchSize int
	IgnoreErrors       bool
	Collection         *mongo.Collection
}

ImportJob ...

type ImportResult

type ImportResult struct {
	TotalFiles     int
	TotalSources   int
	Description    string
	Succeeded      int
	Failed         int
	Elapsed        time.Duration
	PartialResults []SourceResult
}

ImportResult ...

func (ImportResult) Summary

func (ir ImportResult) Summary() string

Summary ...

type LoggableResult

type LoggableResult interface {
	Summary() string
}

LoggableResult ...

type MongoConnection

type MongoConnection struct {
	DatabaseName     string
	AuthDatabaseName string
	User             string
	Password         string
	Host             string
	Port             uint
}

MongoConnection ...

func (*MongoConnection) Client

func (c *MongoConnection) Client() (*mongo.Client, error)

Client ...

type Options

type Options struct {
	DatabaseName       string
	Collection         string
	Loader             loaders.Loader
	PostLoad           PostLoadHook
	PreDump            PreDumpHook
	UpdateFilter       UpdateFilterHook
	EmptyCollection    *opt.Flag
	Sanitize           *opt.Flag
	FailOnErrors       *opt.Flag
	CollectErrors      *opt.Flag
	IndividualProgress *opt.Flag
	ShowCurrentFile    *opt.Flag
	InsertionBatchSize int
}

Options ...

type PartialResult

type PartialResult struct {
	File       string
	Collection string
	Source     *Datasource
	Succeeded  int
	Failed     int
	Elapsed    time.Duration
	Errors     []error
}

PartialResult ...

func (PartialResult) Summary

func (ir PartialResult) Summary() string

Summary ...

type PostLoadHook

type PostLoadHook func(loaded map[string]interface{}) ([]interface{}, error)

PostLoadHook ...

type PreDumpHook

type PreDumpHook func(loaded interface{}) ([]interface{}, error)

PreDumpHook ...

type SourceResult

type SourceResult struct {
	TotalFiles     int
	Collection     string
	Description    string
	Succeeded      int
	Failed         int
	Elapsed        time.Duration
	PartialResults []PartialResult
}

SourceResult ...

func (SourceResult) Summary

func (ir SourceResult) Summary() string

Summary ...

type UpdateFilterHook

type UpdateFilterHook func(loaded interface{}) ([]interface{}, error)

UpdateFilterHook ...