athenaconv

README

Provides conversion from athena outputs to strongly defined data models.

Getting started

Given the following data struct you define:

type MyModel struct {
    ID                        int       `athenaconv:"id"`
    Name                      string    `athenaconv:"name"`
    SourceComputersCount      int64     `athenaconv:"source_computers_count"`
    SourceComputerExternalIDs []string  `athenaconv:"source_computer_ids"`
    SourceComputerNames       []string  `athenaconv:"source_computer_names"`
    TestTimestamp             time.Time `athenaconv:"test_timestamp"`
    TestDate                  time.Time `athenaconv:"test_date"`
    TestBool                  bool      `athenaconv:"test_bool"`
}

And the following sql:

select
    id,
    name,
    count(source_id) as source_computers_count,
    array_agg(source_id) as source_computer_ids,
    array_agg(source_name) as source_computer_names,
    timestamp '2012-10-31 08:11:22' as test_timestamp,
    date '2021-12-31' as test_date,
    true as test_bool
from my_glue_catalog_table
group by id, name

You can convert your athena.GetQueryResultOutput object to strongly-typed struct MyModel by doing this:

mapper, err := athenaconv.NewMapperFor(reflect.TypeOf(MyModel{}))
if err != nil {
    handleError(err)
}

var mapped []interface{}
mapped, err = mapper.FromAthenaResultSetV2(ctx, queryResultOutput.ResultSet)
if err != nil {
    handleError(err)
}
for _, mappedItem := range mapped {
    mappedItemModel := mappedItem.(*MyModel)
    fmt.Printf("%+v\n", *mappedItemModel)
}

Supported data types

See conversion.go in this repo and supported data types in athena for more details.

Athena data type	Go data type	Comments
varchar	string
boolean	bool
integer	int/int32
bigint	int64
timestamp	time.Time
date	time.Time
array	[]string	Individual items within array should not contain comma
other data types	string	Other data types currently unsupported, default to string (no conversion)

Supported AWS SDK version

• github.com/aws/aws-sdk-go-v2/service/athena/types

Roadmap / items to review

• Add more data type support in conversion.go
• Review usage of logging (best practice for logging in golang packages)

Documentation

Index

• type DataMapper
  • func NewMapperFor(modelType reflect.Type) (DataMapper, error)

Types

type DataMapper

type DataMapper interface {
	FromAthenaResultSetV2(ctx context.Context, input *types.ResultSet) ([]interface{}, error)
}

DataMapper provides abstraction to convert athena ResultSet object to arbitrary user-defined struct

func NewMapperFor

func NewMapperFor(modelType reflect.Type) (DataMapper, error)

NewMapperFor creates new DataMapper for given reflect.Type reflect.Type should be of struct value type, not pointer to struct.

Example:

mapper, err := athenaconv.NewMapperFor(reflect.TypeOf(MyStruct{}))