gke

README

GKE Module

Collects and exports costs associated with GKE instances. It's built on top of main of the same primitives as the GCP module. Specifically we share

• PricingMap
• MachineSpec
• ListInstances

What differs between the two is that the module will filter out instances that are not GKE instances. This is done by checking the labels field of the instance and looking for the cluster name. If no cluster name is found, the instance is not considered a GKE instance and is filtered out.

The primary motivation for this module was to ensure we could support the following cases with ease:

1. Collecting costs for GKE instances
2. Collecting costs for Compute instances that may not be a GKE instance
3. Collecting costs for Persistent Volumes that may be attached to a GKE instance

See the Design Doc for the rationale for a separate module. TL;DR; We do not want to emit metrics with a exporter_cluster label that is empty or make the setup process more complex needed.

Disk Pricing

Running the gke module also collects costs associated with Persistent Volumes. Persistent Volumes are attached to GKE instances and are billed as disks. The price is based off of a combination of the following attributes:

• region
• disk type
• disk size

For simplicity, cloudcost-exporter has implemented the following disk types:

• Standard(hard disk drives)
• SSD(solid state drives)
• Local SSD

According to the documentation, pricing for storage is for JEDEC Binary GB or IEC gibibytes(GiB). One of the more confusing bits is that the documentation for disk implies that the size is in GB, but doesn't specify if it's a decimal GB or Binary GB. cloudcost-exporter is assuming that the size is in binary GB which aligns with the pricing documentation.

Documentation

Index

• Constants
• Variables
• func ListDisks(project string, zone string, service *compute.Service) ([]*compute.Disk, error)
• type Collector
  • func New(config *Config, computeService *compute.Service, ...) (*Collector, error)
  • func (c *Collector) Collect(ch chan<- prometheus.Metric) error
  • func (c *Collector) CollectMetrics(ch chan<- prometheus.Metric) float64
  • func (c *Collector) Describe(ch chan<- *prometheus.Desc) error
  • func (c *Collector) Name() string
  • func (c *Collector) Register(_ provider.Registry) error
• type Config
• type Disk
  • func NewDisk(disk *compute.Disk, project string) *Disk
  • func (d Disk) DiskType() string
  • func (d Disk) Name() string
  • func (d Disk) Namespace() string
  • func (d Disk) Region() string
  • func (d Disk) StorageClass() string
  • func (d Disk) UseStatus() string
• type FamilyPricing
  • func NewMachineTypePricing() *FamilyPricing
• type MachineSpec
  • func ListInstancesInZone(projectID, zone string, c *compute.Service) ([]*MachineSpec, error)
  • func NewMachineSpec(instance *compute.Instance) *MachineSpec
  • func (m *MachineSpec) GetClusterName() string
• type ParsedSkuData
  • func NewParsedSkuData(region string, priceTier PriceTier, price int32, description string, ...) *ParsedSkuData
• type PriceTier
• type PriceTiers
  • func NewPriceTiers() *PriceTiers
• type Prices
• type PricingMap
  • func GeneratePricingMap(skus []*billingpb.Sku) (*PricingMap, error)
  • func NewComputePricingMap() *PricingMap
  • func NewPricingMap(ctx context.Context, billingService *billingv1.CloudCatalogClient) (*PricingMap, error)
  • func (m PricingMap) GetCostOfInstance(instance *MachineSpec) (float64, float64, error)
  • func (m PricingMap) GetCostOfStorage(region, storageClass string) (float64, error)
  • func (pm *PricingMap) Populate(ctx context.Context, billingService *billingv1.CloudCatalogClient) error
• type Resource
• type StoragePricing
  • func NewStoragePricing() *StoragePricing

Constants

const (
	BootDiskLabel = "goog-gke-node"
)

const PriceRefreshInterval = 24 * time.Hour

Variables

var (
	GkeClusterLabel = "goog-k8s-cluster-name"
	GkeRegionLabel  = "goog-k8s-cluster-location"
)

var (
	ErrSkuNotFound            = errors.New("no sku was interested in us")
	ErrSkuIsNil               = errors.New("sku is nil")
	ErrSkuNotParsable         = errors.New("can't parse sku")
	ErrSkuNotRelevant         = errors.New("sku isn't relevant for the current use cases")
	ErrPricingDataIsOff       = errors.New("pricing data in sku isn't parsable")
	ErrRegionNotFound         = errors.New("region wasn't found in pricing map")
	ErrFamilyTypeNotFound     = errors.New("family wasn't found in pricing map for this region")
	ErrInitializingPricingMap = errors.New("failed to populate pricing map")
)

var (
	ErrListInstances = errors.New("no list price was found for the sku")
)

Functions

func ListDisks

func ListDisks(project string, zone string, service *compute.Service) ([]*compute.Disk, error)

ListDisks will list all disks in a given zone and return a slice of compute.Disk

Types

type Collector

type Collector struct {
	Projects   []string
	PricingMap *PricingMap
	NextScrape time.Time
	// contains filtered or unexported fields
}

func New

func New(config *Config, computeService *compute.Service, billingService *billingv1.CloudCatalogClient) (*Collector, error)

func (*Collector) Collect

func (c *Collector) Collect(ch chan<- prometheus.Metric) error

func (*Collector) CollectMetrics

func (c *Collector) CollectMetrics(ch chan<- prometheus.Metric) float64

func (*Collector) Describe

func (c *Collector) Describe(ch chan<- *prometheus.Desc) error

func (*Collector) Name

func (c *Collector) Name() string

func (*Collector) Register

func (c *Collector) Register(_ provider.Registry) error

type Config

type Config struct {
	Projects       string
	ScrapeInterval time.Duration
	Logger         *slog.Logger
}

type Disk

type Disk struct {
	Cluster string

	Project string

	Size int64
	// contains filtered or unexported fields
}

func NewDisk

func NewDisk(disk *compute.Disk, project string) *Disk

func (Disk) DiskType

func (d Disk) DiskType() string

DiskType will search through the labels to determine the type of disk. If the disk has a label "goog-gke-node" it will return "boot_disk" Otherwise it returns persistent_volume

func (Disk) Name

func (d Disk) Name() string

Name will return the name of the disk. If the disk has a label "kubernetes.io/created-for/pv/name" it will return the value stored in that key. otherwise it will return the disk name that is directly associated with the disk.

func (Disk) Namespace

func (d Disk) Namespace() string

Namespace will search through the description fields for the namespace of the disk. If the namespace can't be determined An empty string is return.

func (Disk) Region

func (d Disk) Region() string

Region will return the region of the disk by search through the zone field and returning the region. If the region can't be determined It will return an empty string

func (Disk) StorageClass

func (d Disk) StorageClass() string

StorageClass will return the storage class of the disk by looking at the type. Type in GCP is represented as a URL and as such we're looking for the last part of the URL to determine the storage class

func (Disk) UseStatus

func (d Disk) UseStatus() string

UseStatus will return two constant strings to tell apart disks that are sitting idle from those that are mounted to a pod It's named UseStatus and not just Status because the GCP API already has a field Status that holds a different concept that we don't want to overwrite. From their docs: Status: [Output Only] The status of disk creation. - CREATING: Disk is provisioning. - RESTORING: Source data is being copied into the disk. - FAILED: Disk creation failed. - READY: Disk is ready for use. - DELETING: Disk is deleting. UNAVAILABLE - Disk is currently unavailable and cannot be accessed,

type FamilyPricing

type FamilyPricing struct {
	Family map[string]*PriceTiers
}

FamilyPricing is a map where the key is the family and the value is the price tiers

func NewMachineTypePricing

func NewMachineTypePricing() *FamilyPricing

type MachineSpec

type MachineSpec struct {
	Instance     string
	Zone         string
	Region       string
	Family       string
	MachineType  string
	SpotInstance bool
	Labels       map[string]string
	PriceTier    string
}

MachineSpec is a slimmed down representation of a google compute.Instance struct

func ListInstancesInZone

func ListInstancesInZone(projectID, zone string, c *compute.Service) ([]*MachineSpec, error)

ListInstancesInZone will list all instances in a given zone and return a slice of MachineSpecs

func NewMachineSpec

func NewMachineSpec(instance *compute.Instance) *MachineSpec

NewMachineSpec will create a new MachineSpec from compute.Instance objects. It's responsible for determining the machine family and region that it operates in

func (*MachineSpec) GetClusterName

func (m *MachineSpec) GetClusterName() string

type ParsedSkuData

type ParsedSkuData struct {
	Region          string
	PriceTier       PriceTier
	Price           int32
	Description     string
	ComputeResource Resource
}

func NewParsedSkuData

func NewParsedSkuData(region string, priceTier PriceTier, price int32, description string, computeResource Resource) *ParsedSkuData

type PriceTier

type PriceTier int64

const (
	OnDemand PriceTier = iota
	Spot
)

type PriceTiers

type PriceTiers struct {
	OnDemand Prices
	Spot     Prices
}

func NewPriceTiers

func NewPriceTiers() *PriceTiers

type Prices

type Prices struct {
	Cpu float64
	Ram float64
}

type PricingMap

type PricingMap struct {
	Compute map[string]*FamilyPricing
	Storage map[string]*StoragePricing
}

PricingMap is a map of regions to a map of family to price tiers

func GeneratePricingMap

func GeneratePricingMap(skus []*billingpb.Sku) (*PricingMap, error)

Paula: deprecate this function in favour of func (pm *PricingMap) Populate(skus []*billingpb.Sku) (*PricingMap, error)

func NewComputePricingMap

func NewComputePricingMap() *PricingMap

NewComputePricingMap returns a new PricingMap in a way that can be used afterwards.

func NewPricingMap

func NewPricingMap(ctx context.Context, billingService *billingv1.CloudCatalogClient) (*PricingMap, error)

NewPricingMap returns a new PricingMap in a way that can be used afterwards.

func (PricingMap) GetCostOfInstance

func (m PricingMap) GetCostOfInstance(instance *MachineSpec) (float64, float64, error)

func (PricingMap) GetCostOfStorage

func (m PricingMap) GetCostOfStorage(region, storageClass string) (float64, error)

func (*PricingMap) Populate

func (pm *PricingMap) Populate(ctx context.Context, billingService *billingv1.CloudCatalogClient) error

type Resource

type Resource int64

const (
	Cpu Resource = iota
	Ram
	Storage
)

type StoragePricing

type StoragePricing struct {
	Storage map[string]float64
}

StoragePricing is a map where the key is the storage type and the value is the price

func NewStoragePricing

func NewStoragePricing() *StoragePricing