core

Documentation

Overview

Copyright © 2019 AltoStack <info@altostack.io>

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2019 AltoStack <info@altostack.io>

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2019 AltoStack <info@altostack.io>

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

• func MarshalDynamoAttributeMap(attrs map[string]*dynamodb.AttributeValue) ([]byte, error)
• type AwsHelper
  • func NewAwsHelper(region string) *AwsHelper
  • func (h *AwsHelper) ChannelToS3(bucketName, s3Folder string, s3BufferSize int, destination *AwsHelper)
  • func (h *AwsHelper) ChannelToTable(tableName string, batchSize int64, waitPeriod time.Duration)
  • func (h *AwsHelper) CheckTableEmpty(tbl string) (int64, error)
  • func (h *AwsHelper) DumpBuffer(bucketName, s3Folder string, buff *bytes.Buffer)
  • func (h *AwsHelper) ExistsInS3(bucketName, s3Path string) (bool, error)
  • func (h *AwsHelper) GetFromS3(bucketName, s3Path string) (*io.ReadCloser, error)
  • func (h *AwsHelper) LoadManifestFromS3(bucketName, manifestPath string) error
  • func (h *AwsHelper) ReaderToChannel(dataReader *io.ReadCloser) error
  • func (h *AwsHelper) S3ToDynamo(tableName string, batchSize int64, waitPeriod time.Duration) error
  • func (h *AwsHelper) TableToChannel(tableName string, batchSize int64, waitPeriod time.Duration) error
  • func (h *AwsHelper) UploadToS3(bucketName, s3Key string, data []byte)
• type CustomAttributeValue
  • func (attr *CustomAttributeValue) Marshal(in *dynamodb.AttributeValue)
  • func (attr *CustomAttributeValue) Unmarshal(out *dynamodb.AttributeValue)
• type S3Manifest
• type S3ManifestEntry

Functions

func MarshalDynamoAttributeMap

func MarshalDynamoAttributeMap(attrs map[string]*dynamodb.AttributeValue) ([]byte, error)

MarshalDynamoAttributeMap only purpose is to create a json-format representation of the AttributeValue map without the empty fields. We wouldn't need that if the fields of the struct was set to `json:",omitempty"` and we can't use the dynamodbattribute package because you can't really do a field for field copy with it.

Types

type AwsHelper

type AwsHelper struct {
	AwsSession client.ConfigProvider
	DynamoSvc  dynamodbiface.DynamoDBAPI
	Wg         sync.WaitGroup
	DataPipe   chan map[string]*dynamodb.AttributeValue
	ManifestS3 S3Manifest
}

AwsHelper supports a set of helpers around DynamoDB and s3

func NewAwsHelper

func NewAwsHelper(region string) *AwsHelper

NewAwsHelper creates a new AwsHelper, initializing an AWS session and a few objects like a channel or a DynamoDB client

func (*AwsHelper) ChannelToS3

func (h *AwsHelper) ChannelToS3(bucketName, s3Folder string, s3BufferSize int, destination *AwsHelper)

ChannelToS3 reads from the given channel and sends the data the given bucket in files of s3BufferSize max size

func (*AwsHelper) ChannelToTable

func (h *AwsHelper) ChannelToTable(tableName string, batchSize int64, waitPeriod time.Duration)

ChannelToTable puts the data from the channel into the given Dynamo table

func (*AwsHelper) CheckTableEmpty

func (h *AwsHelper) CheckTableEmpty(tbl string) (int64, error)

CheckTableEmpty checks if the table exists and is empty. Returns -1 if does not exist, returns the number of items if exists and -10 in case of an error. If the table status is not "ACTIVE", the returned number will be -2 for "CREATING", -4 for "UPDATING", -8 for "DELETING"

func (*AwsHelper) DumpBuffer

func (h *AwsHelper) DumpBuffer(bucketName, s3Folder string, buff *bytes.Buffer)

DumpBuffer dumps the content of the given buffer to a new randomly generated file name in the given s3 path in the given bucket and resets the said buffer

func (*AwsHelper) ExistsInS3

func (h *AwsHelper) ExistsInS3(bucketName, s3Path string) (bool, error)

ExistsInS3 checks that a given path in s3 exists as a file

func (*AwsHelper) GetFromS3

func (h *AwsHelper) GetFromS3(bucketName, s3Path string) (*io.ReadCloser, error)

GetFromS3 download a file from s3 to memory (as the files are small by default - just a few Mb).

func (*AwsHelper) LoadManifestFromS3

func (h *AwsHelper) LoadManifestFromS3(bucketName, manifestPath string) error

LoadManifestFromS3 downloads the given manifest file and load it in the ManifestS3 attribute of the struct

func (*AwsHelper) ReaderToChannel

func (h *AwsHelper) ReaderToChannel(dataReader *io.ReadCloser) error

ReaderToChannel reads the data from a actions line by line, serializes it and sends it to the struct's channel

func (*AwsHelper) S3ToDynamo

func (h *AwsHelper) S3ToDynamo(tableName string, batchSize int64, waitPeriod time.Duration) error

S3ToDynamo pulls the s3 files from AwsHelper.ManifestS3 and import them inside the given table using the given batch size (and wait period between each batch)

func (*AwsHelper) TableToChannel

func (h *AwsHelper) TableToChannel(tableName string, batchSize int64, waitPeriod time.Duration) error

TableToChannel scans an entire DynamoDB table, putting all the output records to a given channel and increment a given wait group

func (*AwsHelper) UploadToS3

func (h *AwsHelper) UploadToS3(bucketName, s3Key string, data []byte)

UploadToS3 writes the content of a bytes array to the given s3 path

type CustomAttributeValue

type CustomAttributeValue struct {
	B    []byte                           `json:"b,omitempty"`
	BOOL *bool                            `json:"bool,omitempty"`
	BS   [][]byte                         `json:"bs,omitempty"`
	N    *string                          `json:"n,omitempty"`
	NS   []*string                        `json:"ns,omitempty"`
	NULL *bool                            `json:"null,omitempty"`
	S    *string                          `json:"s,omitempty"`
	SS   []*string                        `json:"ss,omitempty"`
	L    []*CustomAttributeValue          `json:"l,omitempty"`
	M    map[string]*CustomAttributeValue `json:"m,omitempty"`
}

CustomAttributeValue is here to get rid of the empty fields during the json Marshaling as the dynamodb.AttributeValue don't have any json tag set

func (*CustomAttributeValue) Marshal

func (attr *CustomAttributeValue) Marshal(in *dynamodb.AttributeValue)

Marshal translates a *dynamodb.AttributeValue into a *CustomAttributeValue field for field

func (*CustomAttributeValue) Unmarshal

func (attr *CustomAttributeValue) Unmarshal(out *dynamodb.AttributeValue)

Unmarshal translates the current CustomAttributeValue to a *dynamodb.AttributeValue

type S3Manifest

type S3Manifest struct {
	Name    string            `json:"name"`
	Version int               `json:"version"`
	Entries []S3ManifestEntry `json:"entries"`
}

S3Manifest represents the actions manifest stored in the s3 folder of the actions

type S3ManifestEntry

type S3ManifestEntry struct {
	URL       string `json:"url"`
	Mandatory bool   `json:"mandatory"`
}

S3ManifestEntry represents an entry in the actions manifest stored in the s3 folder of the actions