instance_mapper

README

Cloud Instance Mapper

This tool creates a score-based mapping of different IaaS VM instance types from EC2 and Azure Compute to Google Cloud Platform instance types. You can also map AWS RDS instances to Google Cloud SQL database types.

When set up correctly with authentication, it will retrieve the latest instance type details from each of the cloud providers's APIs.

Building

• Requires Golang 1.16+

You can build it by running:

go install github.com/GoogleCloudPlatform/professional-services/tools/instance_mapper/cmd/instance_mapper

Configuring

AWS IAM role

Attach the following policies to the desired role: AmazonEC2ReadOnlyAccess and AmazonRDSReadOnlyAccess. Assign a policy for the user or group that allows sts:AssumeRole to the role you created.

Example of configuring the AWS IAM role using aws cli tool:

# aws iam create-user --user-name InstanceMapper
{
    "User": {
        "Path": "/",
        "UserName": "InstanceMapper",
        "UserId": "A123451234512345Y",
        "Arn": "arn:aws:iam::1234567890:user/InstanceMapper"
    }
}
# cat <<EOF >assume-role.json 
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Principal": { "AWS": "arn:aws:iam::1234567890:user/InstanceMapper" },
          "Action": "sts:AssumeRole"
      }
  ]
}
EOF
# aws iam create-role --role-name Instance-Mapper-Readonly --assume-role-policy-document file://assume-role.json
{
    "Role": {
        "Path": "/",
        "RoleName": "Instance-Mapper-Readonly",
        "RoleId": "A1234512345612345X",
        "Arn": "arn:aws:iam::1234567890:role/Instance-Mapper-Readonly",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "arn:aws:iam::1234567890:user/InstanceMapper"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}
# aws iam attach-role-policy --role-name Instance-Mapper-Readonly --policy-arn "arn:aws:iam::aws:policy/AmazonRDSReadOnlyAccess"
# aws iam attach-role-policy --role-name Instance-Mapper-Readonly --policy-arn "arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess"
# aws iam create-access-key --user-name InstanceMapper
{
    "AccessKey": {
        "UserName": "InstanceMapper",
        "AccessKeyId": "A123456123456789U",
        "Status": "Active",
        "SecretAccessKey": "This-Is-Your-Secret-Access-Key"
    }
}
# export AWS_ACCESS_KEY_ID="A123456123456789U"
# export AWS_SECRET_ACCESS_KEY="This-Is-Your-Secret-Access-Key"

Running

To get the available command line flags, run:

instance_mapper -help

Example:

instance_mapper 
  -azure-vm \
  -azure-subscription-id 5ed066d4-803b-4d57-bf9e-5a64e5fe3f05 \
  -aws-ec2 \
  -aws-role arn:aws:iam::1234567890:role/Instance-Mapper-Readonly \
  -gcp-project my-project-id > instances.csv

The tool outputs a CSV formatted file on standard out, that looks something like this:

Instance type	Memory	vCPUs	GPUs	GPU type	Total GPU memory	Instance type	Memory	vCPUs	GPUs	GPU type	Total GPU memory	Instance type	Memory	vCPUs	GPUs	GPU type	Total GPU memory	Instance type	Memory	vCPUs	GPUs	GPU type	Total GPU memory
c5d.12xlarge	96.00 GB	48	0		0	n2d-highcpu-96	96.00 GB	96	0		0	n2d-standard-48	192.00 GB	48	0		0	n2d-highmem-48	384.00 GB	48	0		0
c4.large	3.75 GB	2	0		0	e2-medium	4.00 GB	2	0		0	e2-standard-2	8.00 GB	2	0		0	e2-small	2.00 GB	2	0		0
r4.xlarge	30.50 GB	4	0		0	e2-highmem-4	32.00 GB	4	0		0	n2d-highmem-4	32.00 GB	4	0		0	n2-highmem-4	32.00 GB	4	0		0
c6gd.2xlarge	16.00 GB	8	0		0	e2-standard-4	16.00 GB	4	0		0	e2-standard-8	32.00 GB	8	0		0	e2-highcpu-16	16.00 GB	16	0		0
Standard_D1	3.50 GB	1	0		0	n1-standard-1	3.75 GB	1	0		0	t2d-standard-1	4.00 GB	1	0		0	n1-highcpu-4	3.60 GB	4	0		0
Standard_E80is_v4	504.00 GB	80	0		0	n2d-highmem-64	512.00 GB	64	0		0	n2-highmem-64	512.00 GB	64	0		0	n2d-highmem-80	640.00 GB	80	0		0
Standard_M64s_v2	1024.00 GB	64	0		0	m1-ultramem-40	961.00 GB	40	0		0	n2d-highcpu-64	64.00 GB	64	0		0	n2d-standard-64	256.00 GB	64	0		0
Standard_ND40s_v3	672.00 GB	40	8		0	n2d-highmem-80	640.00 GB	80	0		0
Standard_D16s_v3	64.00 GB	16	0		0	e2-standard-16	64.00 GB	16	0		0	n2d-standard-16	64.00 GB	16	0		0	n2-standard-16	64.00 GB	16	0		0

Mapping files

There are two mapping files: gpu-mapping.yaml (for mapping GPU types between cloud providers) and sql-mapping.yaml (for mapping SQL engine types and versions).

Adding pricing information

You can use the public pricelist.json to augment the output with price estimates for the GCP instance types. Download the file and pass the following flags to instance mapper: -gcp-price-list pricelist.json -gcp-price-list-region europe-west4

Customizing matcher

To show the default CEL matchers, run:

instance_mapper -show-cel

You can copy the matchers as basis for your custom files to two files (one for compute instances, one for SQL instances) and specify them using -custom-matcher and -custom-matcher-sql flags.

Limitations

• Azure API does not provide: a GPU type description, shared tenancy support, bare metal, CPU clockspeed

Documentation

Index

• func GetEnv() (*cel.Env, error)
• func SetUserAgent(inner http.RoundTripper, userAgent string, debug bool) http.RoundTripper
• type AddUserAgent
  • func (ug *AddUserAgent) RoundTrip(r *http.Request) (*http.Response, error)
• type DatabaseVersion
• type GPUMap
• type GcpPriceList
• type InstanceData
• type InstanceMapper
  • func (im *InstanceMapper) LoadAWSInstanceTypes(ctx context.Context, role string, debug bool) error
  • func (im *InstanceMapper) LoadAWSRDSInstanceTypes(ctx context.Context, role string, rdsEngines []string, debug bool) error
  • func (im *InstanceMapper) LoadAzureInstanceTypes(ctx context.Context, subscriptionId string, debug bool) error
  • func (im *InstanceMapper) LoadGCPCloudSQLInstanceTypes(ctx context.Context, projectId string, sqlMap *SQLMap, debug bool) error
  • func (im *InstanceMapper) LoadGCPInstanceTypes(ctx context.Context, projectId string, priceList *GcpPriceList, debug bool) error
  • func (im *InstanceMapper) LoadGCPPriceList(fileName string) (*GcpPriceList, error)
  • func (im *InstanceMapper) MapInstances(csvWriter *csv.Writer, source *map[string]map[string]InstanceType, ...) error
• type InstanceType
  • func (it InstanceType) CSVHeaders(showPrices bool) []string
  • func (it InstanceType) GetHourlyPrice(priceList *GcpPriceList) (float64, error)
  • func (it InstanceType) String() string
  • func (it InstanceType) ToCSV(showPrices bool, region string) []string
  • func (it InstanceType) ToMap() *map[string]interface{}
• type MapError
  • func (e *MapError) Error() string
• type MatcherLib
  • func (MatcherLib) CompileOptions() []cel.EnvOption
  • func (MatcherLib) ProgramOptions() []cel.ProgramOption
• type SQLMap
• type SQLMapItem

Functions

func GetEnv

func GetEnv() (*cel.Env, error)

func SetUserAgent

func SetUserAgent(inner http.RoundTripper, userAgent string, debug bool) http.RoundTripper

Types

type AddUserAgent

type AddUserAgent struct {
	Agent string
	Debug bool
	// contains filtered or unexported fields
}

func (*AddUserAgent) RoundTrip

func (ug *AddUserAgent) RoundTrip(r *http.Request) (*http.Response, error)

type DatabaseVersion

type DatabaseVersion struct {
	DatabaseType         string `yaml:"db_type,omitempty"`
	DatabaseMajorVersion string `yaml:"db_version_major,omitempty"`
	DatabaseMinorVersion string `yaml:"db_version_minor,omitempty"`
	DatabaseOriginal     string `yaml:"db_type,omit"`
}

type GPUMap

type GPUMap struct {
	Mapping map[string]string `yaml:"gpu_mapping"`
}

type GcpPriceList

type GcpPriceList struct {
	Comment   string                 `json:"comment"`
	Version   string                 `json:"version"`
	Updated   string                 `json:"updated"`
	PriceList map[string]interface{} `json:"gcp_price_list"`
}

type InstanceData

type InstanceData struct {
	AwsInstances    map[string]map[string]InstanceType `yaml:"aws"`
	GcpInstances    map[string]map[string]InstanceType `yaml:"gcp"`
	AwsRdsInstances map[string]map[string]InstanceType `yaml:"aws_rds"`
	GcpSqlInstances map[string]map[string]InstanceType `yaml:"gcp_sql"`
	AzureInstances  map[string]map[string]InstanceType `yaml:"azure"`
}

type InstanceMapper

type InstanceMapper struct {
	InstanceData InstanceData
}

func (*InstanceMapper) LoadAWSInstanceTypes

func (im *InstanceMapper) LoadAWSInstanceTypes(ctx context.Context, role string, debug bool) error

func (*InstanceMapper) LoadAWSRDSInstanceTypes

func (im *InstanceMapper) LoadAWSRDSInstanceTypes(ctx context.Context, role string, rdsEngines []string, debug bool) error

func (*InstanceMapper) LoadAzureInstanceTypes

func (im *InstanceMapper) LoadAzureInstanceTypes(ctx context.Context, subscriptionId string, debug bool) error

func (*InstanceMapper) LoadGCPCloudSQLInstanceTypes

func (im *InstanceMapper) LoadGCPCloudSQLInstanceTypes(ctx context.Context, projectId string, sqlMap *SQLMap, debug bool) error

func (*InstanceMapper) LoadGCPInstanceTypes

func (im *InstanceMapper) LoadGCPInstanceTypes(ctx context.Context, projectId string, priceList *GcpPriceList, debug bool) error

func (*InstanceMapper) LoadGCPPriceList

func (im *InstanceMapper) LoadGCPPriceList(fileName string) (*GcpPriceList, error)

func (*InstanceMapper) MapInstances

func (im *InstanceMapper) MapInstances(csvWriter *csv.Writer, source *map[string]map[string]InstanceType, target *map[string]map[string]InstanceType, matcher cel.Program, gpuMap GPUMap, priceListRegion string, numberOfResults int) error

type InstanceType

type InstanceType struct {
	InstanceTypeId       string              `yaml:"id"`
	InstanceFamily       string              `yaml:"family"`
	Region               string              `yaml:"region"`
	Description          string              `yaml:"description,omitempty"`
	BareMetal            bool                `yaml:"bare_metal,omitempty"`
	SharedTenancy        bool                `yaml:"shared_tenancy,omitempty"`
	GPUs                 int                 `yaml:"total_gpus,omitempty"`
	GPUType              string              `yaml:"gpu_type,omitempty"`
	GPUMemory            int                 `yaml:"total_gpu_memory,omitempty"`
	Memory               int                 `yaml:"total_memory"`
	VCPUs                int                 `yaml:"total_vcpus"`
	GHz                  float64             `yaml:"cpu_clockspeed"`
	Bandwidth            float64             `yaml:"network_bandwidth,omitempty"`
	DatabaseType         string              `yaml:"db_type,omitempty"`
	DatabaseMajorVersion string              `yaml:"db_version_major,omitempty"`
	DatabaseMinorVersion string              `yaml:"db_version_minor,omitempty"`
	DatabaseOriginal     string              `yaml:"db_original_type,omitempty"`
	PricingPerRegion     *map[string]float64 `yaml:"prices,omitempty"`
}

func (InstanceType) CSVHeaders

func (it InstanceType) CSVHeaders(showPrices bool) []string

func (InstanceType) GetHourlyPrice

func (it InstanceType) GetHourlyPrice(priceList *GcpPriceList) (float64, error)

func (InstanceType) String

func (it InstanceType) String() string

func (InstanceType) ToCSV

func (it InstanceType) ToCSV(showPrices bool, region string) []string

func (InstanceType) ToMap

func (it InstanceType) ToMap() *map[string]interface{}

type MapError

type MapError struct {
	Msg string
	Err error
}

func (*MapError) Error

func (e *MapError) Error() string

type MatcherLib

type MatcherLib struct{}

func (MatcherLib) CompileOptions

func (MatcherLib) CompileOptions() []cel.EnvOption

func (MatcherLib) ProgramOptions

func (MatcherLib) ProgramOptions() []cel.ProgramOption

type SQLMap

type SQLMap struct {
	Mapping map[string]SQLMapItem `yaml:"sql_mapping"`
}

type SQLMapItem

type SQLMapItem struct {
	Engine string `yaml:"engine"`
	Major  string `yaml:"major"`
	Minor  string `yaml:"minor"`
}