ec2

package
v0.4.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 3, 2024 License: Apache-2.0 Imports: 21 Imported by: 0

README

ec2 cost module

This module is responsible for collecting pricing information for EC2 instances. See metrics for more information on the metrics that are collected.

Overview

EC2 instances are a foundational component in the AWS ecosystem. They can be used as bare bone virtual machines, or used as the underlying infrastructure for services like

  1. EC2 instances
  2. ECS Clusters that use ec2 instances*
  3. EKS Clusters

This module aims to emit metrics generically for ec2 instances that can be used for the services above. A conscious decision was made the keep ec2 + eks implementations coupled. See #215 for more details on why, as this decision can be reversed in the future.

*Fargate is a serverless product which builds upon ec2 instances, but with a specific caveat: Pricing is based upon the requests by workloads

![WARNING] Even though Fargate uses ec2 instances under the hood, it would require a separate module since the pricing comes from a different end point

Pricing Map

The pricing map is generated based on the machine type and the region where the instance is running.

Here's how the data structure looks like:

--> root
    --> region
        --> machine type
           --> reservation type(on-demand, spot)
              --> price

Regions are populated by making a describe region api call to find the regions enabled for the account. The price keeps track of the hourly cost per:

  1. price per cpu
  2. price per GiB of ram
  3. total price

The pricing information for the compute instances is collected from the AWS Pricing API. Detailed documentation around the pricing API can be found here. One of the main challenges with EKS compute instance pricing is that the pricing is for the full instance and not broken down by resource. This means that the pricing information is not available for the CPU and memory separately. cloudcost-exporter makes the assumption that the ratio of costs is relatively similar to that of GKE instances. When fetching the list prices, cloudcost-exporter will use the ratio from GCP to break down the cost of the instance into CPU and memory.

Collecting Machines

The following attributes must be available from the ec2 instance to make the lookup:

  • region
  • machine type
  • reservation type

Every time the collector is scraped, a list of machines is collected by region in a seperate goroutine. This allows the collector to scrape each region in parallel, making the largest region be the bottleneck. For simplicity, there is no cache, though this is a nice feature to add in the future.

Cost Calculations

Here's some example PromQL queries that can be used to calculate the costs of ec2 instances:

// Calculate the total hourly cost of all ec2 instances
sum(cloudcost_aws_ec2_instance_total_usd_per_houry)
// Calculate the total hourly cost by region
sum by (region) (cloudcost_aws_ec2_instance_total_usd_per_houry)
// Calculate the total hourly cost by machine type
sum by (machine_type) (cloudcost_aws_ec2_instance_total_usd_per_houry)
// Calculate the total hourly cost by reservation type
sum by (reservation) (cloudcost_aws_ec2_instance_total_usd_per_houry)

You can do more interesting queries if you run yet-another-cloudwatch-exporter and export the following metrics:

  • aws_ec2_info

All of these examples assume that you have created the tag name referenced in the examples.

// Calculate the total hourly cost by team
// Assumes a tag called `Team` has been created on the ec2 instances
sum by (team) (
    cloudcost_aws_ec2_instance_total_usd_per_houry
    * on (instance_id) group_right()
    label_join(aws_ec2_info, "team", "tag_Team")
)

// Calculate the total hourly cost by team and environment
// Assumes a tag called `Team` has been created on the ec2 instances
// Assumes a tag called `Environment` has been created on the ec2 instances
sum by (team, environment) (
    cloudcost_aws_ec2_instance_total_usd_per_houry
    * on (instance_id) group_right()
    label_join(
        label_join(aws_ec2_info, "environment", "tag_Environment")
    "team", "tag_Team")
)

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrClientNotFound = errors.New("no client found")

	ErrGeneratePricingMap = errors.New("error generating pricing map")
)
View Source 
var (
	InstanceCPUHourlyCostDesc = prometheus.NewDesc(
		prometheus.BuildFQName(cloudcostexporter.MetricPrefix, subsystem, "instance_cpu_usd_per_core_hour"),
		"The cpu cost a ec2 instance in USD/(core*h)",
		[]string{"instance", "instance_id", "region", "family", "machine_type", "cluster_name", "price_tier"},
		nil,
	)
	InstanceMemoryHourlyCostDesc = prometheus.NewDesc(
		prometheus.BuildFQName(cloudcostexporter.MetricPrefix, subsystem, "instance_memory_usd_per_gib_hour"),
		"The memory cost of a ec2 instance in USD/(GiB*h)",
		[]string{"instance", "instance_id", "region", "family", "machine_type", "cluster_name", "price_tier"},
		nil,
	)
	InstanceTotalHourlyCostDesc = prometheus.NewDesc(
		prometheus.BuildFQName(cloudcostexporter.MetricPrefix, subsystem, "instance_total_usd_per_hour"),
		"The total cost of the ec2 instance in USD/h",
		[]string{"instance", "instance_id", "region", "family", "machine_type", "cluster_name", "price_tier"},
		nil,
	)
	PersistentVolumeHourlyCostDesc = prometheus.NewDesc(
		prometheus.BuildFQName(cloudcostexporter.MetricPrefix, subsystem, "persistent_volume_usd_per_hour"),
		"The cost of an AWS EBS Volume in USD/h.",
		[]string{"persistentvolume", "region", "availability_zone", "disk", "type", "size_gib", "state"},
		nil,
	)
)
View Source 
var (
	ErrInstanceTypeAlreadyExists = errors.New("instance type already exists in the map")
	ErrParseAttributes           = errors.New("error parsing attribute")
	ErrRegionNotFound            = errors.New("no region found")
	ErrInstanceTypeNotFound      = errors.New("no instance type found")
	ErrVolumeTypeNotFound        = errors.New("volume type not found")
	ErrListSpotPrices            = errors.New("error listing spot prices")
	ErrListOnDemandPrices        = errors.New("error listing ondemand prices")
	ErrListStoragePrices         = errors.New("error listing storage prices")
)

Functions

func ClusterNameFromInstance 

func ClusterNameFromInstance(instance types.Instance) string

func ListComputeInstances 

func ListComputeInstances(ctx context.Context, client ec2.EC2) ([]types.Reservation, error)

func ListEBSVolumes added in v0.4.0 

func ListEBSVolumes(ctx context.Context, client ec2.EC2) ([]types.Volume, error)

func ListOnDemandPrices 

func ListOnDemandPrices(ctx context.Context, region string, client pricingClient.Pricing) ([]string, error)

func ListSpotPrices 

func ListSpotPrices(ctx context.Context, client ec2client.EC2) ([]ec2Types.SpotPrice, error)

func ListStoragePrices added in v0.4.0 

func ListStoragePrices(ctx context.Context, region string, client pricingClient.Pricing) ([]string, error)

func NameFromVolume added in v0.4.0 

func NameFromVolume(volume types.Volume) string

Types

type Collector 

type Collector struct {
	Regions        []ec2Types.Region
	ScrapeInterval time.Duration

	NextComputeScrape time.Time
	NextStorageScrape time.Time
	// contains filtered or unexported fields
}

Collector is a prometheus collector that collects metrics from AWS EKS clusters.

func New 

func New(config *Config, ps pricingClient.Pricing) *Collector

New creates an ec2 collector

func (*Collector) CheckReadiness added in v0.3.0 

func (c *Collector) CheckReadiness() bool

func (*Collector) Collect 

func (c *Collector) Collect(ch chan<- prometheus.Metric) error

Collect satisfies the provider.Collector interface.

func (*Collector) CollectMetrics 

func (c *Collector) CollectMetrics(_ chan<- prometheus.Metric) float64

CollectMetrics is a no-op function that satisfies the provider.Collector interface. Deprecated: CollectMetrics is deprecated and will be removed in a future release.

func (*Collector) Describe 

func (c *Collector) Describe(ch chan<- *prometheus.Desc) error

func (*Collector) Name 

func (c *Collector) Name() string

func (*Collector) Register 

func (c *Collector) Register(_ provider.Registry) error

type ComputePricingMap added in v0.4.0 

type ComputePricingMap struct {
	// Regions is a map of region code to FamilyPricing
	// key is the region
	// value is a map of instance type to PriceTiers
	Regions         map[string]*FamilyPricing
	InstanceDetails map[string]InstanceAttributes
	// contains filtered or unexported fields
}

ComputePricingMap collects a map of FamilyPricing structs where the key is the region

func NewComputePricingMap added in v0.4.0 

func NewComputePricingMap(l *slog.Logger) *ComputePricingMap

func (*ComputePricingMap) AddInstanceDetails added in v0.4.0 

func (cpm *ComputePricingMap) AddInstanceDetails(attributes InstanceAttributes)

func (*ComputePricingMap) AddToComputePricingMap added in v0.4.0 

func (cpm *ComputePricingMap) AddToComputePricingMap(price float64, attribute InstanceAttributes) error

AddToComputePricingMap adds a price to the compute pricing map. The price is weighted based upon the instance type's CPU and RAM.

func (*ComputePricingMap) CheckReadiness added in v0.4.0 

func (cpm *ComputePricingMap) CheckReadiness() bool

func (*ComputePricingMap) GenerateComputePricingMap added in v0.4.0 

func (cpm *ComputePricingMap) GenerateComputePricingMap(ondemandPrices []string, spotPrices []ec2Types.SpotPrice) error

GenerateComputePricingMap accepts a list of ondemand prices and a list of spot prices. The method needs to 1. Parse out the ondemand prices and generate a productTerm map for each instance type 2. Parse out spot prices and use the productTerm map to generate a spot price map

func (*ComputePricingMap) GetPriceForInstanceType added in v0.4.0 

func (cpm *ComputePricingMap) GetPriceForInstanceType(region string, instanceType string) (*Prices, error)

type Config 

type Config struct {
	ScrapeInterval time.Duration
	Regions        []ec2Types.Region
	RegionClients  map[string]ec2client.EC2
	Logger         *slog.Logger
}

type FamilyPricing 

type FamilyPricing struct {
	Family map[string]*Prices // Each Family can have many PriceTiers
}

FamilyPricing is a map of instance type to a list of PriceTiers where the key is the ec2 compute instance type

type InstanceAttributes added in v0.4.0 

type InstanceAttributes struct {
	Region            string `json:"regionCode"`
	InstanceType      string `json:"instanceType"`
	VCPU              string `json:"vcpu"`
	Memory            string `json:"memory"`
	InstanceFamily    string `json:"instanceFamily"`
	PhysicalProcessor string `json:"physicalProcessor"`
	Tenancy           string `json:"tenancy"`
	MarketOption      string `json:"marketOption"`
	OperatingSystem   string `json:"operatingSystem"`
	ClockSpeed        string `json:"clockSpeed"`
	UsageType         string `json:"usageType"`
}

InstanceAttributes represents ec2 instance attributes that are pulled from AWS api's describing instances. It's specifically pulled out of productTerm to enable usage during tests.

type Prices 

type Prices struct {
	Cpu   float64
	Ram   float64
	Total float64
}

ComputePrices holds the price of a ec2 instances CPU and RAM. The price is in USD

type StoragePricing added in v0.4.0 

type StoragePricing struct {
	Storage map[string]float64
}

StoragePricing is a map where the key is the storage type and the value is the price

type StoragePricingMap added in v0.4.0 

type StoragePricingMap struct {
	// Regions is a map of region code to StoragePricing
	// key is the region
	// value is a map of storage classes to prices
	Regions map[string]*StoragePricing
	// contains filtered or unexported fields
}

StoragePricingMap collects a map of StoragePricing structs where the key is the region

func NewStoragePricingMap added in v0.4.0 

func NewStoragePricingMap(l *slog.Logger) *StoragePricingMap

func (*StoragePricingMap) CheckReadiness added in v0.4.0 

func (spm *StoragePricingMap) CheckReadiness() bool

func (*StoragePricingMap) GenerateStoragePricingMap added in v0.4.0 

func (spm *StoragePricingMap) GenerateStoragePricingMap(storagePrices []string) error

GenerateStoragePricingMap receives a json with all the prices of the available storage options It iterates over the storage classes and parses the price for each one.

func (*StoragePricingMap) GetPriceForVolumeType added in v0.4.0 

func (spm *StoragePricingMap) GetPriceForVolumeType(region string, volumeType string, size int32) (float64, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.