athena

README

This is forked from speee/go-athena.

The following are the main changes that have been made:

Changes

• Use Amazon Athena SDK functions for prepared statements instead of using SQL queries.

---

This is forked from segmentio/go-athena and we described what changes we added here.

go-athena

go-athena is a simple Golang database/sql driver for Amazon Athena.

import (
    "database/sql"
    _ "github.com/basemachina/go-athena"
)

func main() {
  db, _ := sql.Open("athena", "db=default&output_location=s3://results")
  rows, _ := db.Query("SELECT url, code from cloudfront")

  for rows.Next() {
    var url string
    var code int
    rows.Scan(&url, &code)
  }
}

It provides a higher-level, idiomatic wrapper over the AWS Go SDK, comparable to the Athena JDBC driver AWS provides for Java users.

For example,

• Instead of manually parsing types from strings, you can use database/sql.Rows.Scan()
• Instead of reaching for semaphores, you can use database/sql.DB.SetMaxOpenConns
• And, so on...

Caveats

database/sql exposes lots of methods that aren't supported in Athena. For example, Athena doesn't support transactions so Begin() is irrelevant. If a method must be supplied to satisfy a standard library interface but is unsupported, the driver will panic indicating so. If there are new offerings in Athena and/or helpful additions, feel free to PR.

Result Mode

go-athena has the following modes to get the result of the query.

• API (default)
• DL
• GZIP DL

Note

• DL and GZIP DL Mode are used only in the Select statement.
  • Other statements automatically use API mode under DL or GZIP DL Mode.
• Detailed explanation is described here.
• Usages of Result Mode.

Performance

Response time for fetching 1,000,000 records

package	response time
segmentio/go-athena	2m 33.4132205s
speee/go-athena API mode	2m 26.475804292s
speee/go-athena DL mode	20.719727417s
speee/go-athena GZIP mode	17.661648209s

Detailed explanation is described here.

Prepared Statements

You can use Athena Prepared Statements. Click here for details on how to use.

Testing

Athena doesn't have a local version and revolves around S3 so our tests are integration tests against AWS itself. Thus, our tests require AWS credentials. The simplest way to provide them is via AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, but you can use anything supported by the Default Credential Provider Chain.

The tests support a few environment variables:

• ATHENA_DATABASE can be used to override the default database "go_athena_tests"
• S3_BUCKET can be used to override the default S3 bucket of "go-athena-tests"
• ATHENA_REGION or AWS_DEFAULT_REGION can be used to override the default region of "us-east-1"
• ATHENA_WORK_GROUP can be used to override the default workgroup of "primary"

Documentation

Index

• Constants
• Variables
• func Open(cfg Config) (*sql.DB, error)
• func SetAPIMode(ctx context.Context) context.Context
• func SetCatalog(ctx context.Context, catalog string) context.Context
• func SetDLMode(ctx context.Context) context.Context
• func SetGzipDLMode(ctx context.Context) context.Context
• func SetResultMode(ctx context.Context, mode ResultMode) context.Context
• func SetTimeout(ctx context.Context, timeout uint) context.Context
• type Config
• type Driver
  • func NewDriver(cfg *Config) *Driver
  • func (d *Driver) Open(connStr string) (driver.Conn, error)
• type ResultMode

Constants

const (
	// TimestampLayout is the Go time layout string for an Athena `timestamp`.
	TimestampLayout             = "2006-01-02 15:04:05.999"
	TimestampWithTimeZoneLayout = "2006-01-02 15:04:05.999 MST"
	DateLayout                  = "2006-01-02"
)

const (
	CATALOG_AWS_DATA_CATALOG string = "AwsDataCatalog"
)

Variables

var (
	ErrConfigRequired    = errors.New("config is required")
	ErrDatabaseRequired  = errors.New("database is required")
	ErrSessionRequired   = errors.New("session is required")
	ErrInvalidResultMode = errors.New("invalid result mode")
)

Error types

var CatalogContextKey = string(catalogKey)

CatalogContextKey is deprecated, use catalogKey instead

var ResultModeContextKey = string(resultModeKey)

ResultModeContextKey is deprecated, use resultModeKey instead

var TimeoutContextKey = string(timeoutKey)

TimeoutContextKey is deprecated, use timeoutKey instead

Functions

func Open

func Open(cfg Config) (*sql.DB, error)

Open is a more robust version of `db.Open`, as it accepts a raw aws.Config. This is useful if you have a complex AWS session since the driver doesn't currently attempt to serialize all options into a string.

func SetAPIMode

func SetAPIMode(ctx context.Context) context.Context

SetAPIMode sets APIMode to ResultMode in context

func SetCatalog

func SetCatalog(ctx context.Context, catalog string) context.Context

SetCatalog sets catalog in context

func SetDLMode

func SetDLMode(ctx context.Context) context.Context

SetDLMode sets DownloadMode to ResultMode in context

func SetGzipDLMode

func SetGzipDLMode(ctx context.Context) context.Context

SetGzipDLMode sets GzipDLMode to ResultMode in context

func SetResultMode

func SetResultMode(ctx context.Context, mode ResultMode) context.Context

SetResultMode sets the ResultMode in context

func SetTimeout

func SetTimeout(ctx context.Context, timeout uint) context.Context

SetTimeout sets timeout in context

Types

type Config

type Config struct {
	Config         aws.Config
	Database       string
	OutputLocation string
	WorkGroup      string

	PollFrequency time.Duration

	ResultMode ResultMode
	Timeout    uint
	Catalog    string
}

Config is the input to Open().

type Driver

type Driver struct {
	// contains filtered or unexported fields
}

Driver is a sql.Driver. It's intended for db/sql.Open().

func NewDriver

func NewDriver(cfg *Config) *Driver

NewDriver allows you to register your own driver with `sql.Register`. It's useful for more complex use cases. Read more in PR #3. https://github.com/segmentio/go-athena/pull/3

Generally, sql.Open() or athena.Open() should suffice.

func (*Driver) Open

func (d *Driver) Open(connStr string) (driver.Conn, error)

Open should be used via `db/sql.Open("athena", "<params>")`. The following parameters are supported in URI query format (k=v&k2=v2&...)

- `db` (required) This is the Athena database name. In the UI, this defaults to "default", but the driver requires it regardless.

- `output_location` (required) This is the S3 location Athena will dump query results in the format "s3://bucket/and/so/forth". In the AWS UI, this defaults to "s3://aws-athena-query-results-<ACCOUNTID>-<REGION>", but the driver requires it.

- `poll_frequency` (optional) Athena's API requires polling to retrieve query results. This is the frequency at which the driver will poll for results. It should be a time/Duration.String(). A completely arbitrary default of "5s" was chosen.

- `region` (optional) Override AWS region. Useful if it is not set with environment variable.

- `workgroup` (optional) Athena's workgroup. This defaults to "primary".

Credentials must be accessible via the SDK's Default Credential Provider Chain. For more advanced AWS credentials/session/config management, please supply a custom AWS session directly via `athena.Open()`.

type ResultMode

type ResultMode int

ResultMode Results mode

const (
	// ResultModeAPI api access Mode
	ResultModeAPI ResultMode = 0

	// ResultModeDL download results Mode
	ResultModeDL ResultMode = 1

	// ResultModeGzipDL ctas query and download gzip file Mode
	ResultModeGzipDL ResultMode = 2
)