bigquery_schema

Documentation

Index

• Variables
• func InferBQSchema[T any](it iterator.RecordIterator[T]) (it2 iterator.RecordIterator[T], getSchema func() TableFieldSchema, err error)
• type FieldType
• type TableFieldSchema

Variables

var ErrInvalidRootType = errors.New("invalid root type, must be map or struct")

Functions

func InferBQSchema

func InferBQSchema[T any](it iterator.RecordIterator[T]) (
	it2 iterator.RecordIterator[T],
	getSchema func() TableFieldSchema,
	err error,
)

InferBQSchema infers a bigquery schema from records yielded by the iterator. records are assumed to originate from JSON parsed data and as such strings can contain dates, timestamps, integers, floats will be converted to correct type (e.g. "2020-01-01" will be inferred to be a date). this tries to mimic how BigQuery infers schema from JSON data. if two records have conflicting types for the same field, the field will be inferred to be of the type which can capture both values; most often this is "STRING" or "JSON" but an float can capture integer values (ish) and as such a field containing both integers and floats will be inferred to be a float. Conflicting types such as bool and float will be inferred as "JSON".

Types

type FieldType

type FieldType string

FieldType is the type of field.

const (
	// StringFieldType is a string field type.
	StringFieldType FieldType = "STRING"
	// BytesFieldType is a bytes field type.
	BytesFieldType FieldType = "BYTES"
	// IntegerFieldType is a integer field type.
	IntegerFieldType FieldType = "INTEGER"
	// FloatFieldType is a float field type.
	FloatFieldType FieldType = "FLOAT"
	// BooleanFieldType is a boolean field type.
	BooleanFieldType FieldType = "BOOLEAN"
	// TimestampFieldType is a timestamp field type.
	TimestampFieldType FieldType = "TIMESTAMP"
	// RecordFieldType is a record field type. It is typically used to create columns with repeated or nested data.
	RecordFieldType FieldType = "RECORD"
	// DateFieldType is a date field type.
	DateFieldType FieldType = "DATE"
	// TimeFieldType is a time field type.
	TimeFieldType FieldType = "TIME"
	// DateTimeFieldType is a datetime field type.
	DateTimeFieldType FieldType = "DATETIME"
	// NumericFieldType is a numeric field type. Numeric types include integer types, floating point types and the
	// NUMERIC data type.
	NumericFieldType FieldType = "NUMERIC"
	// GeographyFieldType is a string field type.  Geography types represent a set of points
	// on the Earth's surface, represented in Well Known Text (WKT) format.
	GeographyFieldType FieldType = "GEOGRAPHY"
	// BigNumericFieldType is a numeric field type that supports values of larger precision
	// and scale than the NumericFieldType.
	BigNumericFieldType FieldType = "BIGNUMERIC"
	// IntervalFieldType is a representation of a duration or an amount of time.
	IntervalFieldType FieldType = "INTERVAL"
	// JSONFieldType is a representation of a json object.
	JSONFieldType FieldType = "JSON"

	// Default case
	NOFieldType FieldType = ""
)

type TableFieldSchema

type TableFieldSchema struct {

	// Description: [Optional] The field description. The maximum length is
	// 1,024 characters.
	Description string `json:"description,omitempty"`

	// Fields: [Optional] Describes the nested schema fields if the type
	// property is set to RECORD.
	Fields []*TableFieldSchema `json:"fields,omitempty"`

	// Mode: [Optional] The field mode. Possible values include NULLABLE,
	// REQUIRED and REPEATED. The default value is NULLABLE.
	Mode string `json:"mode,omitempty"`

	// Name: [Required] The field name. The name must contain only letters
	// (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a
	// letter or underscore. The maximum length is 300 characters.
	Name string `json:"name,omitempty"`

	// Precision: [Optional] Precision (maximum number of total digits in
	// base 10) and scale (maximum number of digits in the fractional part
	// in base 10) constraints for values of this field for NUMERIC or
	// BIGNUMERIC. It is invalid to set precision or scale if type ≠
	// "NUMERIC" and ≠ "BIGNUMERIC". If precision and scale are not
	// specified, no value range constraint is imposed on this field insofar
	// as values are permitted by the type. Values of this NUMERIC or
	// BIGNUMERIC field must be in this range when: - Precision (P) and
	// scale (S) are specified: [-10P-S + 10-S, 10P-S - 10-S] - Precision
	// (P) is specified but not scale (and thus scale is interpreted to be
	// equal to zero): [-10P + 1, 10P - 1]. Acceptable values for precision
	// and scale if both are specified: - If type = "NUMERIC": 1 ≤
	// precision - scale ≤ 29 and 0 ≤ scale ≤ 9. - If type =
	// "BIGNUMERIC": 1 ≤ precision - scale ≤ 38 and 0 ≤ scale ≤ 38.
	// Acceptable values for precision if only precision is specified but
	// not scale (and thus scale is interpreted to be equal to zero): - If
	// type = "NUMERIC": 1 ≤ precision ≤ 29. - If type = "BIGNUMERIC": 1
	// ≤ precision ≤ 38. If scale is specified but not precision, then
	// it is invalid.
	Precision int64 `json:"precision,omitempty,string"`

	// Scale: [Optional] See documentation for precision.
	Scale int64 `json:"scale,omitempty,string"`

	// Type: [Required] The field data type. Possible values include STRING,
	// BYTES, INTEGER, INT64 (same as INTEGER), FLOAT, FLOAT64 (same as
	// FLOAT), NUMERIC, BIGNUMERIC, BOOLEAN, BOOL (same as BOOLEAN),
	// TIMESTAMP, DATE, TIME, DATETIME, INTERVAL, RECORD (where RECORD
	// indicates that the field contains a nested schema) or STRUCT (same as
	// RECORD).
	Type string `json:"type,omitempty"`
	// contains filtered or unexported fields
}