queued

Documentation

Overview

Package filesystem provides a client with the ability to import data into Kusto via a variety of fileystems such as local storage or blobstore.

Index

• Constants
• func CompleteFormatFromFileName(props *properties.All, from string) error
• func GenBlobName(databaseName string, tableName string, time time.Time, guid string, ...) string
• func IsLocalPath(s string) (bool, error)
• func ShouldCompress(props *properties.All, compressionFileExtension ingestoptions.CompressionType) bool
• type Ingestion
  • func New(db, table string, mgr *resources.Manager, http *http.Client, ...) (*Ingestion, error)
  • func (i *Ingestion) Blob(ctx context.Context, from string, fileSize int64, props properties.All) error
  • func (i *Ingestion) Close() error
  • func (i *Ingestion) Local(ctx context.Context, from string, props properties.All) error
  • func (i *Ingestion) Reader(ctx context.Context, reader io.Reader, props properties.All) (string, error)
• type Option
  • func WithStaticBuffer(bufferSize int, maxBuffers int) Option
• type Queued

Constants

const (
	BlockSize             = 8 * _1MiB
	Concurrency           = 50
	StorageMaxRetryPolicy = 3
)

Functions

func CompleteFormatFromFileName

func CompleteFormatFromFileName(props *properties.All, from string) error

func GenBlobName

func GenBlobName(databaseName string, tableName string, time time.Time, guid string, fileName string, compressionFileExtension ingestoptions.CompressionType, shouldCompress bool, dataFormat string) string

func IsLocalPath

func IsLocalPath(s string) (bool, error)

IsLocalPath detects whether a path points to a file system accessiable file If this file requires another protocol http protocol it will return false If the file requires another protocol(ftp, https, etc) it will return an error

func ShouldCompress

func ShouldCompress(props *properties.All, compressionFileExtension ingestoptions.CompressionType) bool

Do not compress if user specified in DontCompress or CompressionType, if the file extension shows compression, or if the format is binary.

Types

type Ingestion

type Ingestion struct {
	// contains filtered or unexported fields
}

Ingestion provides methods for taking data from a filesystem of some type and ingesting it into Kusto. This object is scoped for a single database and table.

func New

func New(db, table string, mgr *resources.Manager, http *http.Client, applicationForTracing string, clientVersionForTracing string, options ...Option) (*Ingestion, error)

New is the constructor for Ingestion.

func (*Ingestion) Blob

func (i *Ingestion) Blob(ctx context.Context, from string, fileSize int64, props properties.All) error

Blob ingests a file from Azure Blob Storage into Kusto.

func (*Ingestion) Close

func (i *Ingestion) Close() error

func (*Ingestion) Local

func (i *Ingestion) Local(ctx context.Context, from string, props properties.All) error

Local ingests a local file into Kusto.

func (*Ingestion) Reader

func (i *Ingestion) Reader(ctx context.Context, reader io.Reader, props properties.All) (string, error)

Reader uploads a file via an io.Reader. If the function succeeds, it returns the path of the created blob.

type Option

type Option func(s *Ingestion)

Option is an optional argument to New().

func WithStaticBuffer

func WithStaticBuffer(bufferSize int, maxBuffers int) Option

WithStaticBuffer sets a static buffer with a buffer size and max amount of buffers for uploading blobs to kusto.

type Queued

type Queued interface {
	io.Closer
	Local(ctx context.Context, from string, props properties.All) error
	Reader(ctx context.Context, reader io.Reader, props properties.All) (string, error)
	Blob(ctx context.Context, from string, fileSize int64, props properties.All) error
}

Queued provides methods for taking data from various sources and ingesting it into Kusto using queued ingestion.