avro_test_data

package
v1.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 30, 2023 License: MIT Imports: 12 Imported by: 0

Documentation

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func ArraysBulkAvroWriter 

func ArraysBulkAvroWriter(writer io.Writer, writeTime time.Time, request <-chan *Arrays) <-chan error

ArraysBulkAvroWriter will begin a go routine writing an Avro Container File to the writer and add each item from the request channel. If an error is encountered it will be sent on the returned error channel. The given writeTime will be used for all data items written by this function. When the returned request channel is closed this function will finalize the Container File and exit. The returned error channel will be closed just before the go routine exits. Note: That though a nil item will be written as delete it will also be written without an ID or other identifying field and so this is of limited value. In general deletes should be done using WriteAvroDeletedCF.

Example 
input := []*Arrays{{}, {}, {}}
inputChan := make(chan *Arrays)

devnull, _ := os.Open("/dev/null")
defer devnull.Close()

errChan := ArraysBulkAvroWriter(devnull, time.Now(), inputChan)

for _, item := range input {
	select {
	case err := <-errChan:
		fmt.Print(err)
		return
	case inputChan <- item:
	}
}

// Check for any final errors, the errorChan should be closed when the BulkWriter is finished processing
for err := range errChan {
	if err != nil {
		fmt.Print(err)
		return
	}
}

Output:

func ComplexBulkAvroWriter 

func ComplexBulkAvroWriter(writer io.Writer, writeTime time.Time, request <-chan *Complex) <-chan error

ComplexBulkAvroWriter will begin a go routine writing an Avro Container File to the writer and add each item from the request channel. If an error is encountered it will be sent on the returned error channel. The given writeTime will be used for all data items written by this function. When the returned request channel is closed this function will finalize the Container File and exit. The returned error channel will be closed just before the go routine exits. Note: That though a nil item will be written as delete it will also be written without an ID or other identifying field and so this is of limited value. In general deletes should be done using WriteAvroDeletedCF.

Example 
input := []*Complex{{}, {}, {}}
inputChan := make(chan *Complex)

devnull, _ := os.Open("/dev/null")
defer devnull.Close()

errChan := ComplexBulkAvroWriter(devnull, time.Now(), inputChan)

for _, item := range input {
	select {
	case err := <-errChan:
		fmt.Print(err)
		return
	case inputChan <- item:
	}
}

// Check for any final errors, the errorChan should be closed when the BulkWriter is finished processing
for err := range errChan {
	if err != nil {
		fmt.Print(err)
		return
	}
}

Output:

func NestedBulkAvroWriter 

func NestedBulkAvroWriter(writer io.Writer, writeTime time.Time, request <-chan *Nested) <-chan error

NestedBulkAvroWriter will begin a go routine writing an Avro Container File to the writer and add each item from the request channel. If an error is encountered it will be sent on the returned error channel. The given writeTime will be used for all data items written by this function. When the returned request channel is closed this function will finalize the Container File and exit. The returned error channel will be closed just before the go routine exits. Note: That though a nil item will be written as delete it will also be written without an ID or other identifying field and so this is of limited value. In general deletes should be done using WriteAvroDeletedCF.

Example 
input := []*Nested{{}, {}, {}}
inputChan := make(chan *Nested)

devnull, _ := os.Open("/dev/null")
defer devnull.Close()

errChan := NestedBulkAvroWriter(devnull, time.Now(), inputChan)

for _, item := range input {
	select {
	case err := <-errChan:
		fmt.Print(err)
		return
	case inputChan <- item:
	}
}

// Check for any final errors, the errorChan should be closed when the BulkWriter is finished processing
for err := range errChan {
	if err != nil {
		fmt.Print(err)
		return
	}
}

Output:

func RepeatsBulkAvroWriter 

func RepeatsBulkAvroWriter(writer io.Writer, writeTime time.Time, request <-chan *Repeats) <-chan error

RepeatsBulkAvroWriter will begin a go routine writing an Avro Container File to the writer and add each item from the request channel. If an error is encountered it will be sent on the returned error channel. The given writeTime will be used for all data items written by this function. When the returned request channel is closed this function will finalize the Container File and exit. The returned error channel will be closed just before the go routine exits. Note: That though a nil item will be written as delete it will also be written without an ID or other identifying field and so this is of limited value. In general deletes should be done using WriteAvroDeletedCF.

Example 
input := []*Repeats{{}, {}, {}}
inputChan := make(chan *Repeats)

devnull, _ := os.Open("/dev/null")
defer devnull.Close()

errChan := RepeatsBulkAvroWriter(devnull, time.Now(), inputChan)

for _, item := range input {
	select {
	case err := <-errChan:
		fmt.Print(err)
		return
	case inputChan <- item:
	}
}

// Check for any final errors, the errorChan should be closed when the BulkWriter is finished processing
for err := range errChan {
	if err != nil {
		fmt.Print(err)
		return
	}
}

Output:

func SimpleBulkAvroWriter 

func SimpleBulkAvroWriter(writer io.Writer, writeTime time.Time, request <-chan *Simple) <-chan error

SimpleBulkAvroWriter will begin a go routine writing an Avro Container File to the writer and add each item from the request channel. If an error is encountered it will be sent on the returned error channel. The given writeTime will be used for all data items written by this function. When the returned request channel is closed this function will finalize the Container File and exit. The returned error channel will be closed just before the go routine exits. Note: That though a nil item will be written as delete it will also be written without an ID or other identifying field and so this is of limited value. In general deletes should be done using WriteAvroDeletedCF.

Example 
input := []*Simple{{}, {}, {}}
inputChan := make(chan *Simple)

devnull, _ := os.Open("/dev/null")
defer devnull.Close()

errChan := SimpleBulkAvroWriter(devnull, time.Now(), inputChan)

for _, item := range input {
	select {
	case err := <-errChan:
		fmt.Print(err)
		return
	case inputChan <- item:
	}
}

// Check for any final errors, the errorChan should be closed when the BulkWriter is finished processing
for err := range errChan {
	if err != nil {
		fmt.Print(err)
		return
	}
}

Output:

func TimesBulkAvroWriter 

func TimesBulkAvroWriter(writer io.Writer, writeTime time.Time, request <-chan *Times) <-chan error

TimesBulkAvroWriter will begin a go routine writing an Avro Container File to the writer and add each item from the request channel. If an error is encountered it will be sent on the returned error channel. The given writeTime will be used for all data items written by this function. When the returned request channel is closed this function will finalize the Container File and exit. The returned error channel will be closed just before the go routine exits. Note: That though a nil item will be written as delete it will also be written without an ID or other identifying field and so this is of limited value. In general deletes should be done using WriteAvroDeletedCF.

Example 
input := []*Times{{}, {}, {}}
inputChan := make(chan *Times)

devnull, _ := os.Open("/dev/null")
defer devnull.Close()

errChan := TimesBulkAvroWriter(devnull, time.Now(), inputChan)

for _, item := range input {
	select {
	case err := <-errChan:
		fmt.Print(err)
		return
	case inputChan <- item:
	}
}

// Check for any final errors, the errorChan should be closed when the BulkWriter is finished processing
for err := range errChan {
	if err != nil {
		fmt.Print(err)
		return
	}
}

Output:

func TimesPointersBulkAvroWriter added in v1.0.0 

func TimesPointersBulkAvroWriter(writer io.Writer, writeTime time.Time, request <-chan *TimesPointers) <-chan error

TimesPointersBulkAvroWriter will begin a go routine writing an Avro Container File to the writer and add each item from the request channel. If an error is encountered it will be sent on the returned error channel. The given writeTime will be used for all data items written by this function. When the returned request channel is closed this function will finalize the Container File and exit. The returned error channel will be closed just before the go routine exits. Note: That though a nil item will be written as delete it will also be written without an ID or other identifying field and so this is of limited value. In general deletes should be done using WriteAvroDeletedCF.

Example 
input := []*TimesPointers{{}, {}, {}}
inputChan := make(chan *TimesPointers)

devnull, _ := os.Open("/dev/null")
defer devnull.Close()

errChan := TimesPointersBulkAvroWriter(devnull, time.Now(), inputChan)

for _, item := range input {
	select {
	case err := <-errChan:
		fmt.Print(err)
		return
	case inputChan <- item:
	}
}

// Check for any final errors, the errorChan should be closed when the BulkWriter is finished processing
for err := range errChan {
	if err != nil {
		fmt.Print(err)
		return
	}
}

Output:

Types

type Arrays 

type Arrays struct {
	Heights []int64 `json:"heights,omitempty"`
	Parents []struct {
		Count    int64     `json:"count"`
		Children []string  `json:"children"`
		Date     time.Time `json:"date"`
		Info     struct {
			Name string `json:"name"`
			Age  int64  `json:"age"`
		} `json:"info"`
	} `json:"parents"`
}

func (*Arrays) WriteAvroCF 

func (z *Arrays) WriteAvroCF(writer io.Writer, writeTime time.Time) error

WriteAvroCF writes an Avro Containter File to the given io.Writer using snappy compression for the data. The time is used as the AvroWriteTime, if the time is the Zero value then the current time is used. NOTE: If the type has a field in an embedded struct with the same name as a field not in the embedded struct the value will be pulled from the field not in the embedded struct.

func (*Arrays) WriteAvroDeletedCF 

func (z *Arrays) WriteAvroDeletedCF(writer io.Writer, writeTime time.Time) error

WriteAvroDeletedCF works nearly identically to WriteAvroCF but sets the AvroDeleted metadata field to true.

type Complex 

type Complex struct {
	Simple

	Caption string `json:"caption"`
	Credit  string `json:"credit"`
	// The available cropped images
	Crops []struct {
		Height float64 `json:"height"`
		Name   string  `json:"name"`
		// full path to the cropped image file
		Path string `json:"path"`
		// a long
		// multi-line description
		RelativePath string  `json:"relativePath"`
		Width        float64 `json:"width"`
	} `json:"crops"`
	Cutline        string    `json:"cutline,omitempty"`
	DatePhotoTaken time.Time `json:"datePhotoTaken"`
	Orientation    string    `json:"orientation"`
	OriginalSize   struct {
		Height float64 `json:"height"`
		Width  float64 `json:"width"`
	} `json:"originalSize"`
	// a type
	Type string `json:"type"`
	// Universal Resource Locator
	URL struct {
		// The full Canonical URL
		Absolute string `json:"absolute"`
		Meta     struct {
			Description string `json:"description"`
			SiteName    string `json:"siteName"`
		} `json:"meta,omitempty"`
		Publish string `json:"publish"`
	} `json:"URL"`
}

func (*Complex) WriteAvroCF 

func (z *Complex) WriteAvroCF(writer io.Writer, writeTime time.Time) error

WriteAvroCF writes an Avro Containter File to the given io.Writer using snappy compression for the data. The time is used as the AvroWriteTime, if the time is the Zero value then the current time is used. NOTE: If the type has a field in an embedded struct with the same name as a field not in the embedded struct the value will be pulled from the field not in the embedded struct.

func (*Complex) WriteAvroDeletedCF 

func (z *Complex) WriteAvroDeletedCF(writer io.Writer, writeTime time.Time) error

WriteAvroDeletedCF works nearly identically to WriteAvroCF but sets the AvroDeleted metadata field to true.

type Nested 

type Nested struct {
	// Information related to the International Fact-Checking Network (IFCN) program
	FactCheckClaims []struct {
		// The factCheck appearanceURLs
		AppearanceURLs []struct {
			// The original flag
			Original bool `json:"original,omitempty"`
			// The appearance url
			Url string `json:"url"`
		} `json:"appearanceURLs,omitempty"`
		// The factCheck author
		Author string `json:"author,omitempty"`
		// The factCheck claim
		Claim string `json:"claim,omitempty"`
		// The date of the factCheck
		Date string `json:"date,omitempty"`
		// The factCheck rating
		Rating string `json:"rating,omitempty"`
	} `json:"factCheckClaims"`
}

func (*Nested) WriteAvroCF 

func (z *Nested) WriteAvroCF(writer io.Writer, writeTime time.Time) error

WriteAvroCF writes an Avro Containter File to the given io.Writer using snappy compression for the data. The time is used as the AvroWriteTime, if the time is the Zero value then the current time is used. NOTE: If the type has a field in an embedded struct with the same name as a field not in the embedded struct the value will be pulled from the field not in the embedded struct.

func (*Nested) WriteAvroDeletedCF 

func (z *Nested) WriteAvroDeletedCF(writer io.Writer, writeTime time.Time) error

WriteAvroDeletedCF works nearly identically to WriteAvroCF but sets the AvroDeleted metadata field to true.

type Repeats 

type Repeats struct {
	Height      int64 `json:"height,omitempty"`
	SomeDateObj struct {
		Type    string `json:"type"`
		Visible bool   `json:"visible,omitempty"`
	} `json:"someDateObj,omitempty"`
	Type    string  `json:"type"`
	Visible bool    `json:"visible,omitempty"`
	Width   float64 `json:"width,omitempty"`
}

func (*Repeats) WriteAvroCF 

func (z *Repeats) WriteAvroCF(writer io.Writer, writeTime time.Time) error

WriteAvroCF writes an Avro Containter File to the given io.Writer using snappy compression for the data. The time is used as the AvroWriteTime, if the time is the Zero value then the current time is used. NOTE: If the type has a field in an embedded struct with the same name as a field not in the embedded struct the value will be pulled from the field not in the embedded struct.

func (*Repeats) WriteAvroDeletedCF 

func (z *Repeats) WriteAvroDeletedCF(writer io.Writer, writeTime time.Time) error

WriteAvroDeletedCF works nearly identically to WriteAvroCF but sets the AvroDeleted metadata field to true.

type Simple 

type Simple struct {
	Contributors []struct {
		ContributorId string `json:"contributorId,omitempty"`
		Id            string `json:"id"`
		Name          string `json:"name"`
	} `json:"contributors,omitempty"`
	Height      int64 `json:"height,omitempty"`
	SomeDateObj struct {
		Dates []time.Time `json:"dates,omitempty"`
	} `json:"someDateObj,omitempty"`
	Type    string  `json:"type"`
	Visible bool    `json:"visible,omitempty"`
	Width   float64 `json:"width,omitempty"`
}

func (*Simple) WriteAvroCF 

func (z *Simple) WriteAvroCF(writer io.Writer, writeTime time.Time) error

WriteAvroCF writes an Avro Containter File to the given io.Writer using snappy compression for the data. The time is used as the AvroWriteTime, if the time is the Zero value then the current time is used. NOTE: If the type has a field in an embedded struct with the same name as a field not in the embedded struct the value will be pulled from the field not in the embedded struct.

func (*Simple) WriteAvroDeletedCF 

func (z *Simple) WriteAvroDeletedCF(writer io.Writer, writeTime time.Time) error

WriteAvroDeletedCF works nearly identically to WriteAvroCF but sets the AvroDeleted metadata field to true.

type Times 

type Times struct {
	NonRequiredDate time.Time `json:"nonRequiredDate,omitempty"`
	RequiredDate    time.Time `json:"requiredDate"`
}

func (*Times) WriteAvroCF 

func (z *Times) WriteAvroCF(writer io.Writer, writeTime time.Time) error

WriteAvroCF writes an Avro Containter File to the given io.Writer using snappy compression for the data. The time is used as the AvroWriteTime, if the time is the Zero value then the current time is used. NOTE: If the type has a field in an embedded struct with the same name as a field not in the embedded struct the value will be pulled from the field not in the embedded struct.

func (*Times) WriteAvroDeletedCF 

func (z *Times) WriteAvroDeletedCF(writer io.Writer, writeTime time.Time) error

WriteAvroDeletedCF works nearly identically to WriteAvroCF but sets the AvroDeleted metadata field to true.

type TimesPointers added in v1.0.0 

type TimesPointers struct {
	// Information related to the International Fact-Checking Network (IFCN) program
	FactCheckClaims []*TimesPointersFactCheckClaims `json:"factCheckClaims,omitempty"`
	NonRequiredDate *time.Time                      `json:"nonRequiredDate,omitempty"`
	RequiredDate    time.Time                       `json:"requiredDate"`
}

func (*TimesPointers) WriteAvroCF added in v1.0.0 

func (z *TimesPointers) WriteAvroCF(writer io.Writer, writeTime time.Time) error

WriteAvroCF writes an Avro Containter File to the given io.Writer using snappy compression for the data. The time is used as the AvroWriteTime, if the time is the Zero value then the current time is used. NOTE: If the type has a field in an embedded struct with the same name as a field not in the embedded struct the value will be pulled from the field not in the embedded struct.

func (*TimesPointers) WriteAvroDeletedCF added in v1.0.0 

func (z *TimesPointers) WriteAvroDeletedCF(writer io.Writer, writeTime time.Time) error

WriteAvroDeletedCF works nearly identically to WriteAvroCF but sets the AvroDeleted metadata field to true.

type TimesPointersFactCheckClaims added in v1.0.0 

type TimesPointersFactCheckClaims struct {
	// The factCheck appearanceURLs
	AppearanceURLs []*TimesPointersFactCheckClaimsAppearanceURLs `json:"appearanceURLs,omitempty"`
	// The factCheck author
	Author string `json:"author,omitempty"`
	// The factCheck claim
	Claim string `json:"claim,omitempty"`
	// The date of the factCheck
	Date string `json:"date,omitempty"`
	// The factCheck rating
	Rating string `json:"rating,omitempty"`
}

type TimesPointersFactCheckClaimsAppearanceURLs added in v1.0.0 

type TimesPointersFactCheckClaimsAppearanceURLs struct {
	// The original flag
	Original bool `json:"original,omitempty"`
	// The appearance url
	Url string `json:"url"`
}

Source Files

View all Source files

Directories

Path Synopsis
avro
arrays
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
complex
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
nested
Code generated by github.com/actgardner/gogen-avro/v7.
 Code generated by github.com/actgardner/gogen-avro/v7.
repeats
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
simple
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
times
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
timespointers
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
avro/complex
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
avro/nested
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
avro/simple
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
avro/complex
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
avro/nested
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
avro/simple
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.
avro/times
Code generated by github.com/actgardner/gogen-avro.
 Code generated by github.com/actgardner/gogen-avro.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.