Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( DefaultWrtp = parquet.NewWriterProperties( parquet.WithDictionaryDefault(true), parquet.WithVersion(parquet.V2_LATEST), parquet.WithCompression(compress.Codecs.Zstd), parquet.WithStats(true), parquet.WithRootName("bodkin"), ) )
Functions ¶
This section is empty.
Types ¶
type ParquetWriter ¶
type ParquetWriter struct {
// contains filtered or unexported fields
}
func NewParquetWriter ¶
func NewParquetWriter(sc *arrow.Schema, wrtp *parquet.WriterProperties, path string) (*ParquetWriter, *schema.Schema, error)
NewParquetWriter creates a new ParquetWriter.
sc is the Arrow schema to use for writing records. wrtp are the Parquet writer properties to use.
Returns a ParquetWriter and an error. The error will be non-nil if: - Failed to get the Parquet schema from the Arrow schema. - Failed to create the destination file. - Failed to create the Parquet file writer.
Example: ```go pw, err := NewParquetWriter(schema, parquet.NewWriterProperties(parquet.WithCompression(parquet.CompressionCodec_SNAPPY)))
if err != nil {
log.Fatal(err)
}
```
func (*ParquetWriter) Close ¶
func (pw *ParquetWriter) Close() error
Close closes the Parquet writer.
Returns an error if failed to close the Parquet file writer.
func (*ParquetWriter) RecordCount ¶
func (pw *ParquetWriter) RecordCount() int
RecordCount returns the total number of records written.
func (*ParquetWriter) Write ¶
func (pw *ParquetWriter) Write(jsonData []byte) error
Write writes a single record to the Parquet file.
jsonData is the JSON encoded record data.
Returns an error if: - Failed to unmarshal the JSON data. - Failed to write the record to Parquet.
Increments the record count and creates a new row group if the current row group exceeds the default row group byte limit.
Example: ```go err := pw.Write([]byte(`{"id":1,"name":"foo"}`))
if err != nil {
log.Fatal(err)
}
```
func (*ParquetWriter) WriteRecord ¶
func (pw *ParquetWriter) WriteRecord(rec arrow.Record) error
Source Files