s3setup

Documentation

Index

• func AccessBucket(sess *session.Session) (*s3.S3, *s3manager.Downloader)
• func CredCheck(sess *session.Session)
• func DownloadFromBucket(config configuration.Configuration, svc *s3.S3, dwn *s3manager.Downloader, ...) error
• func ProcessDate(fromDate time.Time) []string
• func SetupSession(config configuration.Configuration) *session.Session

Functions

func AccessBucket

func AccessBucket(sess *session.Session) (*s3.S3, *s3manager.Downloader)

This function initiates the service for downloading files in s3

func CredCheck

func CredCheck(sess *session.Session)

CredCheck checks the credentials for a given AWS session. The function takes a pointer to an AWS session as its only parameter. It uses the Get method of the session's Credentials field to retrieve credentials and check if they are valid. If the Get method returns an error, the function logs a fatal error message. If the credentials are read successfully, the function logs an info message indicating that the credentials were read successfully. This function is useful for ensuring that the credentials used to authenticate the AWS session are valid before performing any operations that require authentication.

func DownloadFromBucket

func DownloadFromBucket(config configuration.Configuration, svc *s3.S3, dwn *s3manager.Downloader, dates []string, bprefix string) error

This function downloads files from AWS S3 bucket and extracts the contents of any Parquet files it downloads. It takes in several arguments: - config: a configuration object that contains information such as the S3 bucket name and a local download directory - svc: an S3 client object that allows the function to make API requests to the S3 bucket - dwn: an S3 downloader object that is used to download the files from the bucket - dates: a slice of strings representing dates, which are used to filter the files that are downloaded - bprefix: a string representing a prefix to be appended to the bucket's file prefix The function first initializes a continuationToken variable to nil. It then enters a loop that makes API requests to the S3 bucket to list the objects within it. For each object in the bucket, the function checks if it contains any of the dates in the dates slice, and if it does, it downloads and extracts the file. If the API response is truncated (i.e., there are more objects in the bucket than could be returned in a single response), the function updates the continuationToken variable and continues the loop. If the response is not truncated, the loop breaks and the function returns nil.

func ProcessDate

func ProcessDate(fromDate time.Time) []string

Builds dates from given date to current date, incrementing one day at a time.

func SetupSession

func SetupSession(config configuration.Configuration) *session.Session

SetupSession creates an AWS session using the provided configuration. If there is an error creating the session, it logs a fatal error and returns nil. Otherwise, it returns the newly created session.