awsdiskusagehandler

README

Mailio DiskUsage Handler

GO DiskUsage handler for Mailio.

This module implements an interface for retrieving disk usage information per user ID collected by the AWS Inventory System. It is designed to be integrated into a larger project. This module focuses on collecting disk usage data and storing it in memory.

Install

go get github.com/mailio/go-mailio-diskusage-handler

Features

• periodically retrieves the information from AWS Inventory System about disk usage
• prepares the data user by user basis
• stores all accumulated file information in the memory

Usage

init

apiKey := os.Getenv("api_key")
secretKey := os.Getenv("secret")
region := os.Getenv("region")
handler := NewAwsDiskUsageHandler(apiKey, secretKey, region, "bucket/folder/inventory",24*60*60)
defer handler.Stop()

NewAwsDiskUsageHandler input parameters:

• apiKey (AWS API KEY)
• secretKey (AWS SECRET)
• region (AWS REGION) of stored inventory
• bucket/folder/inventory (root folder of your inventory)
• 246060 cron job to be repeated in seconds

query

du, err := handler.GetDiskUsage(randomKey)

returns object of type: mailiotypes "github.com/mailio/go-mailio-server/diskusage/types"

type DiskUsage struct {
	SizeBytes   int64  `json:"sizeBytes" validate:"required"`
	Address     string `json:"address" validate:"required"`
	NumberFiles int64  `json:"numberFiles,omitempty"`
}

Development

Create .env file to store AWS keys and secrets:

api_key=AKI...
secret=aki...
region=us-east-1

Checkout the handler_test.go and handler.go

Read AWS Intenvory docs

Documentation

Index

• Variables
• type AwsDiskUsageHandler
  • func NewAwsDiskUsageHandler(apiKey, secret, region, inventoryPath string, refreshPeriodSeconds int64) *AwsDiskUsageHandler
  • func (h *AwsDiskUsageHandler) GetDiskUsage(userAddress string) (*mailiotypes.DiskUsage, error)
  • func (du *AwsDiskUsageHandler) Stop()
• type File
• type Inventory

Variables

var ErrNotFound = errors.New("item not found")

custom errors

Types

type AwsDiskUsageHandler

type AwsDiskUsageHandler struct {
	// contains filtered or unexported fields
}

func NewAwsDiskUsageHandler

func NewAwsDiskUsageHandler(apiKey, secret, region, inventoryPath string, refreshPeriodSeconds int64) *AwsDiskUsageHandler

inventoryPath from docs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-location.html Parameters: - apiKey: Aws api key - secret: Aws secret - region: Aws region - inventoryPath: bucket/path/inventory

func (*AwsDiskUsageHandler) GetDiskUsage

func (h *AwsDiskUsageHandler) GetDiskUsage(userAddress string) (*mailiotypes.DiskUsage, error)

GetDiskUsage returns the disk usage for the given user address. If the disk usage is not found, the function returns an NotFoundError.

func (*AwsDiskUsageHandler) Stop

func (du *AwsDiskUsageHandler) Stop()

stop cron job must be called from the external module

type File

type File struct {
	Key         string `json:"key"`
	Size        int    `json:"size"`
	MD5Checksum string `json:"MD5checksum"`
}

file structure specifically for Parquest format Example:

{
    "sourceBucket": "example-source-bucket",
    "destinationBucket": "arn:aws:s3:::example-destination-bucket",
    "version": "2016-11-30",
    "creationTimestamp" : "1514944800000",
    "fileFormat": "Parquet",
    "fileSchema": "message s3.inventory { required binary bucket (UTF8); required binary key (UTF8); optional binary version_id (UTF8); optional boolean is_latest; optional boolean is_delete_marker; optional int64 size; optional int64 last_modified_date (TIMESTAMP_MILLIS); optional binary e_tag (UTF8); optional binary storage_class (UTF8); optional boolean is_multipart_uploaded; optional binary replication_status (UTF8); optional binary encryption_status (UTF8); optional int64 object_lock_retain_until_date (TIMESTAMP_MILLIS); optional binary object_lock_mode (UTF8); optional binary object_lock_legal_hold_status (UTF8); optional binary intelligent_tiering_access_tier (UTF8); optional binary bucket_key_status (UTF8); optional binary checksum_algorithm (UTF8); optional binary object_access_control_list (UTF8); optional binary object_owner (UTF8);}",
    "files": [
        {
           "key": "inventory/example-source-bucket/data/d754c470-85bb-4255-9218-47023c8b4910.parquet",
            "size": 56291,
            "MD5checksum": "5825f2e18e1695c2d030b9f6eexample"
        }
    ]
}

docs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-location.html

type Inventory

type Inventory struct {
	SourceBucket      string `json:"sourceBucket"`
	DestinationBucket string `json:"destinationBucket"`
	Version           string `json:"version"`
	CreationTimestamp string `json:"creationTimestamp"`
	FileFormat        string `json:"fileFormat"`
	FileSchema        string `json:"fileSchema"`
	Files             []File `json:"files"`
}

Inventory represents the entire JSON structure.