README
¶
ec2 cost module
This module is responsible for collecting pricing information for EC2 instances. See metrics for more information on the metrics that are collected.
Overview
EC2 instances are a foundational component in the AWS ecosystem. They can be used as bare bone virtual machines, or used as the underlying infrastructure for services like
- EC2 instances
- ECS Clusters that use ec2 instances*
- EKS Clusters
This module aims to emit metrics generically for ec2 instances that can be used for the services above. A conscious decision was made the keep ec2 + eks implementations coupled. See #215 for more details on why, as this decision can be reversed in the future.
*Fargate is a serverless product which builds upon ec2 instances, but with a specific caveat: Pricing is based upon the requests by workloads
![WARNING] Even though Fargate uses ec2 instances under the hood, it would require a separate module since the pricing comes from a different end point
Pricing Map
The pricing map is generated based on the machine type and the region where the instance is running.
Here's how the data structure looks like:
--> root
--> region
--> machine type
--> reservation type(on-demand, spot)
--> price
Regions are populated by making a describe region api call to find the regions enabled for the account. The price keeps track of the hourly cost per:
- price per cpu
- price per GiB of ram
- total price
The pricing information for the compute instances is collected from the AWS Pricing API.
Detailed documentation around the pricing API can be found here.
One of the main challenges with EKS compute instance pricing is that the pricing is for the full instance and not broken down by resource.
This means that the pricing information is not available for the CPU and memory separately.
cloudcost-exporter
makes the assumption that the ratio of costs is relatively similar to that of GKE instances.
When fetching the list prices, cloudcost-exporter
will use the ratio from GCP to break down the cost of the instance into CPU and memory.
Collecting Machines
The following attributes must be available from the ec2 instance to make the lookup:
- region
- machine type
- reservation type
Every time the collector is scraped, a list of machines is collected by region in a seperate goroutine. This allows the collector to scrape each region in parallel, making the largest region be the bottleneck. For simplicity, there is no cache, though this is a nice feature to add in the future.
Cost Calculations
Here's some example PromQL queries that can be used to calculate the costs of ec2 instances:
// Calculate the total hourly cost of all ec2 instances
sum(cloudcost_aws_ec2_instance_total_usd_per_houry)
// Calculate the total hourly cost by region
sum by (region) (cloudcost_aws_ec2_instance_total_usd_per_houry)
// Calculate the total hourly cost by machine type
sum by (machine_type) (cloudcost_aws_ec2_instance_total_usd_per_houry)
// Calculate the total hourly cost by reservation type
sum by (reservation) (cloudcost_aws_ec2_instance_total_usd_per_houry)
You can do more interesting queries if you run yet-another-cloudwatch-exporter and export the following metrics:
aws_ec2_info
All of these examples assume that you have created the tag name referenced in the examples.
// Calculate the total hourly cost by team
// Assumes a tag called `Team` has been created on the ec2 instances
sum by (team) (
cloudcost_aws_ec2_instance_total_usd_per_houry
* on (instance_id) group_right()
label_join(aws_ec2_info, "team", "tag_Team")
)
// Calculate the total hourly cost by team and environment
// Assumes a tag called `Team` has been created on the ec2 instances
// Assumes a tag called `Environment` has been created on the ec2 instances
sum by (team, environment) (
cloudcost_aws_ec2_instance_total_usd_per_houry
* on (instance_id) group_right()
label_join(
label_join(aws_ec2_info, "environment", "tag_Environment")
"team", "tag_Team")
)
Documentation
¶
Index ¶
- Variables
- func ClusterNameFromInstance(instance types.Instance) string
- func ListComputeInstances(ctx context.Context, client ec2.EC2) ([]types.Reservation, error)
- func ListEBSVolumes(ctx context.Context, client ec2.EC2) ([]types.Volume, error)
- func ListOnDemandPrices(ctx context.Context, region string, client pricingClient.Pricing) ([]string, error)
- func ListSpotPrices(ctx context.Context, client ec2client.EC2) ([]ec2Types.SpotPrice, error)
- func ListStoragePrices(ctx context.Context, region string, client pricingClient.Pricing) ([]string, error)
- func NameFromVolume(volume types.Volume) string
- type Collector
- type ComputePricingMap
- func (cpm *ComputePricingMap) AddInstanceDetails(attributes InstanceAttributes)
- func (cpm *ComputePricingMap) AddToComputePricingMap(price float64, attribute InstanceAttributes) error
- func (cpm *ComputePricingMap) GenerateComputePricingMap(ondemandPrices []string, spotPrices []ec2Types.SpotPrice) error
- func (cpm *ComputePricingMap) GetPriceForInstanceType(region string, instanceType string) (*Prices, error)
- type Config
- type FamilyPricing
- type InstanceAttributes
- type Prices
- type StoragePricing
- type StoragePricingMap
Constants ¶
This section is empty.
Variables ¶
var ( ErrClientNotFound = errors.New("no client found") ErrGeneratePricingMap = errors.New("error generating pricing map") )
var ( InstanceCPUHourlyCostDesc = utils.GenerateDesc( cloudcostexporter.MetricPrefix, subsystem, utils.InstanceCPUCostSuffix, "The cpu cost a ec2 instance in USD/(core*h)", []string{"instance", "instance_id", "region", "family", "machine_type", "cluster_name", "price_tier", "architecture"}, ) InstanceMemoryHourlyCostDesc = utils.GenerateDesc( cloudcostexporter.MetricPrefix, subsystem, utils.InstanceMemoryCostSuffix, "The memory cost of a ec2 instance in USD/(GiB*h)", []string{"instance", "instance_id", "region", "family", "machine_type", "cluster_name", "price_tier", "architecture"}, ) InstanceTotalHourlyCostDesc = utils.GenerateDesc( cloudcostexporter.MetricPrefix, subsystem, utils.InstanceTotalCostSuffix, "The total cost of the ec2 instance in USD/h", []string{"instance", "instance_id", "region", "family", "machine_type", "cluster_name", "price_tier", "architecture"}, ) PersistentVolumeHourlyCostDesc = utils.GenerateDesc( cloudcostexporter.MetricPrefix, subsystem, utils.PersistentVolumeCostSuffix, "The cost of an AWS EBS Volume in USD/h.", []string{"persistentvolume", "region", "availability_zone", "disk", "type", "size_gib", "state"}, ) )
var ( ErrInstanceTypeAlreadyExists = errors.New("instance type already exists in the map") ErrParseAttributes = errors.New("error parsing attribute") ErrRegionNotFound = errors.New("no region found") ErrInstanceTypeNotFound = errors.New("no instance type found") ErrVolumeTypeNotFound = errors.New("volume type not found") ErrListSpotPrices = errors.New("error listing spot prices") ErrListOnDemandPrices = errors.New("error listing ondemand prices") ErrListStoragePrices = errors.New("error listing storage prices") )
Functions ¶
func ClusterNameFromInstance ¶
func ListComputeInstances ¶
func ListEBSVolumes ¶ added in v0.4.0
func ListOnDemandPrices ¶
func ListSpotPrices ¶
func ListStoragePrices ¶ added in v0.4.0
func NameFromVolume ¶ added in v0.4.0
Types ¶
type Collector ¶
type Collector struct { Regions []ec2Types.Region ScrapeInterval time.Duration NextComputeScrape time.Time NextStorageScrape time.Time // contains filtered or unexported fields }
Collector is a prometheus collector that collects metrics from AWS EKS clusters.
func New ¶
func New(config *Config, ps pricingClient.Pricing) *Collector
New creates an ec2 collector
func (*Collector) Collect ¶
func (c *Collector) Collect(ch chan<- prometheus.Metric) error
Collect satisfies the provider.Collector interface.
func (*Collector) CollectMetrics ¶
func (c *Collector) CollectMetrics(_ chan<- prometheus.Metric) float64
CollectMetrics is a no-op function that satisfies the provider.Collector interface. Deprecated: CollectMetrics is deprecated and will be removed in a future release.
type ComputePricingMap ¶ added in v0.4.0
type ComputePricingMap struct { // Regions is a map of region code to FamilyPricing // key is the region // value is a map of instance type to PriceTiers Regions map[string]*FamilyPricing InstanceDetails map[string]InstanceAttributes // contains filtered or unexported fields }
ComputePricingMap collects a map of FamilyPricing structs where the key is the region
func NewComputePricingMap ¶ added in v0.4.0
func NewComputePricingMap(l *slog.Logger) *ComputePricingMap
func (*ComputePricingMap) AddInstanceDetails ¶ added in v0.4.0
func (cpm *ComputePricingMap) AddInstanceDetails(attributes InstanceAttributes)
func (*ComputePricingMap) AddToComputePricingMap ¶ added in v0.4.0
func (cpm *ComputePricingMap) AddToComputePricingMap(price float64, attribute InstanceAttributes) error
AddToComputePricingMap adds a price to the compute pricing map. The price is weighted based upon the instance type's CPU and RAM.
func (*ComputePricingMap) GenerateComputePricingMap ¶ added in v0.4.0
func (cpm *ComputePricingMap) GenerateComputePricingMap(ondemandPrices []string, spotPrices []ec2Types.SpotPrice) error
GenerateComputePricingMap accepts a list of ondemand prices and a list of spot prices. The method needs to 1. Parse out the ondemand prices and generate a productTerm map for each instance type 2. Parse out spot prices and use the productTerm map to generate a spot price map
func (*ComputePricingMap) GetPriceForInstanceType ¶ added in v0.4.0
func (cpm *ComputePricingMap) GetPriceForInstanceType(region string, instanceType string) (*Prices, error)
type FamilyPricing ¶
FamilyPricing is a map of instance type to a list of PriceTiers where the key is the ec2 compute instance type
type InstanceAttributes ¶ added in v0.4.0
type InstanceAttributes struct { Region string `json:"regionCode"` InstanceType string `json:"instanceType"` VCPU string `json:"vcpu"` Memory string `json:"memory"` InstanceFamily string `json:"instanceFamily"` PhysicalProcessor string `json:"physicalProcessor"` Tenancy string `json:"tenancy"` MarketOption string `json:"marketOption"` OperatingSystem string `json:"operatingSystem"` ClockSpeed string `json:"clockSpeed"` UsageType string `json:"usageType"` }
InstanceAttributes represents ec2 instance attributes that are pulled from AWS api's describing instances. It's specifically pulled out of productTerm to enable usage during tests.
type StoragePricing ¶ added in v0.4.0
StoragePricing is a map where the key is the storage type and the value is the price
type StoragePricingMap ¶ added in v0.4.0
type StoragePricingMap struct { // Regions is a map of region code to StoragePricing // key is the region // value is a map of storage classes to prices Regions map[string]*StoragePricing // contains filtered or unexported fields }
StoragePricingMap collects a map of StoragePricing structs where the key is the region
func NewStoragePricingMap ¶ added in v0.4.0
func NewStoragePricingMap(l *slog.Logger) *StoragePricingMap
func (*StoragePricingMap) GenerateStoragePricingMap ¶ added in v0.4.0
func (spm *StoragePricingMap) GenerateStoragePricingMap(storagePrices []string) error
GenerateStoragePricingMap receives a json with all the prices of the available storage options It iterates over the storage classes and parses the price for each one.