mongoimport

package
v0.0.0-...-e563b53 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 10, 2020 License: Apache-2.0 Imports: 31 Imported by: 1

Documentation

Overview

Package mongoimport allows importing content from a JSON, CSV, or TSV into a MongoDB instance.

Index

Constants

View Source 
const (
	CSV  = "csv"
	TSV  = "tsv"
	JSON = "json"
)

Input format types accepted by mongoimport.

Variables

View Source 
var (
	// ErrNoOpeningBracket means that the input source did not contain any
	// opening brace - returned only if --jsonArray is passed in.
	ErrNoOpeningBracket = errors.New("bad JSON array format - found no " +
		"opening bracket '[' in input source")

	// ErrNoClosingBracket means that the input source did not contain any
	// closing brace - returned only if --jsonArray is passed in.
	ErrNoClosingBracket = errors.New("bad JSON array format - found no " +
		"closing bracket ']' in input source")
)
View Source 
var (
	UTF8_BOM = []byte{0xEF, 0xBB, 0xBF}
)
View Source 
var Usage = `` /* 203-byte string literal not displayed */

Functions

func ColumnNames 

func ColumnNames(fs []ColumnSpec) (s []string)

ColumnNames maps a ColumnSpec slice to their associated names

Types

type CSVConverter 

type CSVConverter struct {
	// contains filtered or unexported fields
}

CSVConverter implements the Converter interface for CSV input.

func (CSVConverter) Convert 

func (c CSVConverter) Convert() (b bson.D, err error)

Convert implements the Converter interface for CSV input. It converts a CSVConverter struct to a BSON document.

func (CSVConverter) Print 

func (c CSVConverter) Print()

type CSVInputReader 

type CSVInputReader struct {
	// contains filtered or unexported fields
}

CSVInputReader implements the InputReader interface for CSV input types.

func NewCSVInputReader 

func NewCSVInputReader(colSpecs []ColumnSpec, in io.Reader, rejects io.Writer, numDecoders int, ignoreBlanks bool, useArrayIndexFields bool) *CSVInputReader

NewCSVInputReader returns a CSVInputReader configured to read data from the given io.Reader, extracting only the specified columns using exactly "numDecoders" goroutines.

func (*CSVInputReader) ReadAndValidateHeader 

func (r *CSVInputReader) ReadAndValidateHeader() (err error)

ReadAndValidateHeader reads the header from the underlying reader and validates the header fields. It sets err if the read/validation fails.

func (*CSVInputReader) ReadAndValidateTypedHeader 

func (r *CSVInputReader) ReadAndValidateTypedHeader(parseGrace ParseGrace) (err error)

ReadAndValidateHeader reads the header from the underlying reader and validates the header fields. It sets err if the read/validation fails.

func (*CSVInputReader) StreamDocument 

func (r *CSVInputReader) StreamDocument(ordered bool, readDocs chan bson.D) (retErr error)

StreamDocument takes a boolean indicating if the documents should be streamed in read order and a channel on which to stream the documents processed from the underlying reader. Returns a non-nil error if streaming fails.

type ColumnSpec 

type ColumnSpec struct {
	Name       string
	Parser     FieldParser
	ParseGrace ParseGrace
	TypeName   string
	NameParts  []string
}

ColumnSpec keeps information for each 'column' of import.

func ParseAutoHeaders 

func ParseAutoHeaders(headers []string) (fs []ColumnSpec)

ParseAutoHeaders converts a list of header items to ColumnSpec objects, with automatic parsers.

func ParseTypedHeader 

func ParseTypedHeader(header string, parseGrace ParseGrace) (f ColumnSpec, err error)

ParseTypedHeader produces a ColumnSpec from a header item, extracting type information from the it. The parseGrace is passed along to the new ColumnSpec.

func ParseTypedHeaders 

func ParseTypedHeaders(headers []string, parseGrace ParseGrace) (fs []ColumnSpec, err error)

ParseTypedHeaders performs ParseTypedHeader on each item, returning an error if any single one fails.

type Converter 

type Converter interface {
	Convert() (document bson.D, err error)
}

Converter is an interface that adds the basic Convert method which returns a valid BSON document that has been converted by the underlying implementation. If conversion fails, err will be set.

type FieldAutoParser 

type FieldAutoParser struct{}

func (*FieldAutoParser) Parse 

func (ap *FieldAutoParser) Parse(in string) (interface{}, error)

type FieldBinaryParser 

type FieldBinaryParser struct {
	// contains filtered or unexported fields
}

func NewFieldBinaryParser 

func NewFieldBinaryParser(arg string) (*FieldBinaryParser, error)

func (*FieldBinaryParser) Parse 

func (bp *FieldBinaryParser) Parse(in string) (interface{}, error)

type FieldBooleanParser 

type FieldBooleanParser struct{}

func (*FieldBooleanParser) Parse 

func (bp *FieldBooleanParser) Parse(in string) (interface{}, error)

type FieldDateParser 

type FieldDateParser struct {
	// contains filtered or unexported fields
}

func (*FieldDateParser) Parse 

func (dp *FieldDateParser) Parse(in string) (interface{}, error)

type FieldDecimalParser 

type FieldDecimalParser struct{}

func (*FieldDecimalParser) Parse 

func (ip *FieldDecimalParser) Parse(in string) (interface{}, error)

type FieldDoubleParser 

type FieldDoubleParser struct{}

func (*FieldDoubleParser) Parse 

func (dp *FieldDoubleParser) Parse(in string) (interface{}, error)

type FieldInfo 

type FieldInfo struct {
	// contains filtered or unexported fields
}

FieldInfo contains information about field names. It is used in validateFields.

type FieldInt32Parser 

type FieldInt32Parser struct{}

func (*FieldInt32Parser) Parse 

func (ip *FieldInt32Parser) Parse(in string) (interface{}, error)

type FieldInt64Parser 

type FieldInt64Parser struct{}

func (*FieldInt64Parser) Parse 

func (ip *FieldInt64Parser) Parse(in string) (interface{}, error)

type FieldParser 

type FieldParser interface {
	Parse(in string) (interface{}, error)
}

FieldParser is the interface for any parser of a field item.

func NewFieldParser 

func NewFieldParser(t columnType, arg string) (parser FieldParser, err error)

NewFieldParser yields a FieldParser corresponding to the given columnType. arg is passed along to the specific type's parser, if it permits an argument. An error will be raised if arg is not valid for the type's parser.

type FieldStringParser 

type FieldStringParser struct{}

func (*FieldStringParser) Parse 

func (sp *FieldStringParser) Parse(in string) (interface{}, error)

type IngestOptions 

type IngestOptions struct {
	// Drops target collection before importing.
	Drop bool `long:"drop" description:"drop collection before inserting documents"`

	// Ignores fields with empty values in CSV and TSV imports.
	IgnoreBlanks bool `long:"ignoreBlanks" description:"ignore fields with empty values in CSV and TSV"`

	// Indicates that documents will be inserted in the order of their appearance in the input source.
	MaintainInsertionOrder bool `` /* 286-byte string literal not displayed */

	// Sets the number of insertion routines to use
	NumInsertionWorkers int `` /* 149-byte string literal not displayed */

	// Forces mongoimport to halt the import operation at the first insert or upsert error.
	StopOnError bool `` /* 411-byte string literal not displayed */

	// Modify the import process.
	// For existing documents (match --upsertFields) in the database:
	// "insert": Insert only, skip existing documents.
	// "upsert": Insert new documents or replace existing ones.
	// "merge": Insert new documents or modify existing ones; Preserve values in the database that are not overwritten.
	// "delete": Skip new documents or delete existing ones that match --upsertFields.
	// We don't set `default: insert` here since we need to be able to set mode to upsert if --mode isn't set and --upsertFields is set.
	Mode string `` /* 389-byte string literal not displayed */

	Upsert bool `long:"upsert" hidden:"true" description:"(deprecated; same as --mode=upsert) insert or update objects that already exist"`

	// Specifies a list of fields for the query portion of the upsert; defaults to _id field.
	UpsertFields string `` /* 145-byte string literal not displayed */

	// Sets write concern level for write operations.
	// By default mongoimport uses a write concern of 'majority'.
	// Cannot be used simultaneously with write concern options in a URI.
	WriteConcern string `` /* 202-byte string literal not displayed */

	// Indicates that the server should bypass document validation on import.
	BypassDocumentValidation bool `long:"bypassDocumentValidation" description:"bypass document validation"`

	// Specifies the number of threads to use in processing data read from the input source
	NumDecodingWorkers int `long:"numDecodingWorkers" default:"0" hidden:"true"`

	BulkBufferSize int `long:"batchSize" default:"1000" hidden:"true"`
}

IngestOptions defines the set of options for storing data.

func (*IngestOptions) Name 

func (_ *IngestOptions) Name() string

Name returns a description of the IngestOptions struct.

type InputOptions 

type InputOptions struct {
	// Fields is an option to directly specify comma-separated fields to import to CSV.
	Fields *string `long:"fields" value-name:"<field>[,<field>]*" short:"f" description:"comma separated list of fields, e.g. -f name,age"`

	// FieldFile is a filename that refers to a list of fields to import, 1 per line.
	FieldFile *string `long:"fieldFile" value-name:"<filename>" description:"file with field names - 1 per line"`

	// Specifies the location and name of a file containing the data to import.
	File string `long:"file" value-name:"<filename>" description:"file to import from; if not specified, stdin is used"`

	// Treats the input source's first line as field list (csv and tsv only).
	HeaderLine bool `long:"headerline" description:"use first line in input source as the field list (CSV and TSV only)"`

	// Indicates that the underlying input source contains a single JSON array with the documents to import.
	JSONArray bool `long:"jsonArray" description:"treat input source as a JSON array"`

	// Indicates how to handle type coercion failures
	ParseGrace string `` /* 155-byte string literal not displayed */

	// Specifies the file type to import. The default format is JSON, but it’s possible to import CSV and TSV files.
	Type string `long:"type" value-name:"<type>" default:"json" default-mask:"-" description:"input format to import: json, csv, or tsv"`

	// Indicates that field names include type descriptions
	ColumnsHaveTypes bool `` /* 573-byte string literal not displayed */

	// Indicates that the legacy extended JSON format should be used to parse JSON documents. Defaults to false.
	Legacy bool `long:"legacy" description:"use the legacy extended JSON format"`

	UseArrayIndexFields bool `` /* 245-byte string literal not displayed */
}

InputOptions defines the set of options for reading input data.

func (*InputOptions) Name 

func (_ *InputOptions) Name() string

Name returns a description of the InputOptions struct.

type InputReader 

type InputReader interface {
	// StreamDocument takes a boolean indicating if the documents should be streamed
	// in read order and a channel on which to stream the documents processed from
	// the underlying reader.  Returns a non-nil error if encountered.
	StreamDocument(ordered bool, read chan bson.D) error

	// ReadAndValidateHeader reads the header line from the InputReader and returns
	// a non-nil error if the fields from the header line are invalid; returns
	// nil otherwise. No-op for JSON input readers.
	ReadAndValidateHeader() error

	// ReadAndValidateTypedHeader is the same as ReadAndValidateHeader,
	// except it also parses types from the fields of the header. Parse errors
	// will be handled according parseGrace.
	ReadAndValidateTypedHeader(parseGrace ParseGrace) error
	// contains filtered or unexported methods
}

type JSONConverter 

type JSONConverter struct {
	// contains filtered or unexported fields
}

JSONConverter implements the Converter interface for JSON input.

func (JSONConverter) Convert 

func (c JSONConverter) Convert() (bson.D, error)

Convert implements the Converter interface for JSON input. It converts a JSONConverter struct to a BSON document.

type JSONInputReader 

type JSONInputReader struct {
	// contains filtered or unexported fields
}

JSONInputReader is an implementation of InputReader that reads documents in JSON format.

func NewJSONInputReader 

func NewJSONInputReader(isArray bool, legacyExtJSON bool, in io.Reader, numDecoders int) *JSONInputReader

NewJSONInputReader creates a new JSONInputReader in array mode if specified, configured to read data to the given io.Reader.

func (*JSONInputReader) ReadAndValidateHeader 

func (r *JSONInputReader) ReadAndValidateHeader() error

ReadAndValidateHeader is a no-op for JSON imports; always returns nil.

func (*JSONInputReader) ReadAndValidateTypedHeader 

func (r *JSONInputReader) ReadAndValidateTypedHeader(parseGrace ParseGrace) error

ReadAndValidateTypedHeader is a no-op for JSON imports; always returns nil.

func (*JSONInputReader) StreamDocument 

func (r *JSONInputReader) StreamDocument(ordered bool, readChan chan bson.D) (retErr error)

StreamDocument takes a boolean indicating if the documents should be streamed in read order and a channel on which to stream the documents processed from the underlying reader. Returns a non-nil error if encountered

type MongoImport 

type MongoImport struct {

	// generic mongo tool options
	ToolOptions *options.ToolOptions

	// InputOptions defines options used to read data to be ingested
	InputOptions *InputOptions

	// IngestOptions defines options used to ingest data into MongoDB
	IngestOptions *IngestOptions

	// SessionProvider is used for connecting to the database
	SessionProvider *db.SessionProvider

	// the tomb is used to synchronize ingestion goroutines and causes
	// other sibling goroutines to terminate immediately if one errors out
	tomb.Tomb
	// contains filtered or unexported fields
}

MongoImport is a container for the user-specified options and internal state used for running mongoimport.

func New 

func New(opts Options) (*MongoImport, error)

New constructs a new MongoImport instance from the provided options. This will fail if the options are invalid or if it cannot establish a new connection to the server.

func (*MongoImport) Close 

func (imp *MongoImport) Close()

Close disconnects the server.

func (*MongoImport) ImportDocuments 

func (imp *MongoImport) ImportDocuments() (uint64, uint64, error)

ImportDocuments is used to write input data to the database. It returns the number of documents successfully imported to the appropriate namespace, the number of failures, and any error encountered in doing this

type Options 

type Options struct {
	*options.ToolOptions
	*InputOptions
	*IngestOptions
	ParsedArgs []string
}

Options contains all the possible options that can be used to configure mongoimport.

func ParseOptions 

func ParseOptions(rawArgs []string, versionStr, gitCommit string) (Options, error)

ParseOptions reads command line arguments and converts them into options used to configure mongoimport.

type ParseGrace 

type ParseGrace int

func ParsePG 

func ParsePG(pg string) (res ParseGrace)

ParsePG interprets the user-provided parseGrace, assuming it is valid.

func ValidatePG 

func ValidatePG(pg string) (ParseGrace, error)

ValidatePG ensures the user-provided parseGrace is one of the allowed values.

type TSVConverter 

type TSVConverter struct {
	// contains filtered or unexported fields
}

TSVConverter implements the Converter interface for TSV input.

func (TSVConverter) Convert 

func (c TSVConverter) Convert() (b bson.D, err error)

Convert implements the Converter interface for TSV input. It converts a TSVConverter struct to a BSON document.

func (TSVConverter) Print 

func (c TSVConverter) Print()

type TSVInputReader 

type TSVInputReader struct {
	// contains filtered or unexported fields
}

TSVInputReader is a struct that implements the InputReader interface for a TSV input source.

func NewTSVInputReader 

func NewTSVInputReader(colSpecs []ColumnSpec, in io.Reader, rejects io.Writer, numDecoders int, ignoreBlanks bool, useArrayIndexFields bool) *TSVInputReader

NewTSVInputReader returns a TSVInputReader configured to read input from the given io.Reader, extracting the specified columns only.

func (*TSVInputReader) ReadAndValidateHeader 

func (r *TSVInputReader) ReadAndValidateHeader() (err error)

ReadAndValidateHeader reads the header from the underlying reader and validates the header fields. It sets err if the read/validation fails.

func (*TSVInputReader) ReadAndValidateTypedHeader 

func (r *TSVInputReader) ReadAndValidateTypedHeader(parseGrace ParseGrace) (err error)

ReadAndValidateTypedHeader reads the header from the underlying reader and validates the header fields. It sets err if the read/validation fails.

func (*TSVInputReader) StreamDocument 

func (r *TSVInputReader) StreamDocument(ordered bool, readDocs chan bson.D) (retErr error)

StreamDocument takes a boolean indicating if the documents should be streamed in read order and a channel on which to stream the documents processed from the underlying reader. Returns a non-nil error if streaming fails.

Source Files

View all Source files

Directories

Path Synopsis
Package csv reads and writes comma-separated values (CSV) files.
 Package csv reads and writes comma-separated values (CSV) files.
Main package for the mongoimport tool.
 Main package for the mongoimport tool.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.