queued

Documentation

Overview

Package filesystem provides a client with the ability to import data into Kusto via a variety of fileystems such as local storage or blobstore.

Index

• Constants
• func CompleteFormatFromFileName(props *properties.All, from string) error
• func CompressionDiscovery(fName string) properties.CompressionType
• func IsLocalPath(s string) (bool, error)
• type Ingestion
  • func New(db, table string, mgr *resources.Manager, options ...Option) (*Ingestion, error)
  • func (i *Ingestion) Blob(ctx context.Context, from string, fileSize int64, props properties.All) error
  • func (i *Ingestion) Close() error
  • func (i *Ingestion) Local(ctx context.Context, from string, props properties.All) error
  • func (i *Ingestion) Reader(ctx context.Context, reader io.Reader, props properties.All) (string, error)
• type Option
  • func WithStaticBuffer(bufferSize int, maxBuffers int) Option
• type Queued

Constants

const (
	BlockSize   = 8 * _1MiB
	Concurrency = 50
)

Functions

func CompleteFormatFromFileName

func CompleteFormatFromFileName(props *properties.All, from string) error

func CompressionDiscovery

func CompressionDiscovery(fName string) properties.CompressionType

CompressionDiscovery looks at the file extension. If it is one we support, we return that CompressionType that represents that value. Otherwise we return CTNone to indicate that the file should not be compressed.

func IsLocalPath

func IsLocalPath(s string) (bool, error)

IsLocalPath detects whether a path points to a file system accessiable file If this file requires another protocol http protocol it will return false If the file requires another protocol(ftp, https, etc) it will return an error

Types

type Ingestion

type Ingestion struct {
	// contains filtered or unexported fields
}

Ingestion provides methods for taking data from a filesystem of some type and ingesting it into Kusto. This object is scoped for a single database and table.

func New

func New(db, table string, mgr *resources.Manager, options ...Option) (*Ingestion, error)

New is the constructor for Ingestion.

func (*Ingestion) Blob

func (i *Ingestion) Blob(ctx context.Context, from string, fileSize int64, props properties.All) error

Blob ingests a file from Azure Blob Storage into Kusto.

func (*Ingestion) Close

func (i *Ingestion) Close() error

func (*Ingestion) Local

func (i *Ingestion) Local(ctx context.Context, from string, props properties.All) error

Local ingests a local file into Kusto.

func (*Ingestion) Reader

func (i *Ingestion) Reader(ctx context.Context, reader io.Reader, props properties.All) (string, error)

Reader uploads a file via an io.Reader. If the function succeeds, it returns the path of the created blob.

type Option

type Option func(s *Ingestion)

Option is an optional argument to New().

func WithStaticBuffer

func WithStaticBuffer(bufferSize int, maxBuffers int) Option

WithStaticBuffer sets a static buffer with a buffer size and max amount of buffers for uploading blobs to kusto.

type Queued

type Queued interface {
	io.Closer
	Local(ctx context.Context, from string, props properties.All) error
	Reader(ctx context.Context, reader io.Reader, props properties.All) (string, error)
	Blob(ctx context.Context, from string, fileSize int64, props properties.All) error
}

Queued provides methods for taking data from various sources and ingesting it into Kusto using queued ingestion.