gsh

package module
v0.0.0-...-bc06d7b Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 15, 2024 License: Apache-2.0 Imports: 16 Imported by: 0

README

gardener-scaling-history

[!NOTE] This is prototype for Proof of Concept only.

Repository providers 2 apps:

  • recorder: monitors and records the scaling data for one or more gardner clusters: machine deployments, machine classes, priority classes, autoscaling config, scheduled and unscheduled pods, nodes, etc. Recorded data is dumped into a SQLite DB per cluster.
  • replayer: The replayer can replay scenarios against either the virtual CA or the scaling recommender
    • replay-CA: replays the recorded scaling data of a gardener cluster against a virtual k8s cluster.
      • effectively deploys scheduled and un-scheduled pods in a configurable batch interval and waits till nodes belonging to CA node groups are scaled.
      • This is primarily meant for simulating autoscaling and hence ideally should be run against a virtual cluster like one setup by https://github.com/unmarshall/kvcl/
    • replay-SR: replays the scenario json (produced by replay-CA) against the new scaling-recommender.

Launch the Recorder

Local Launch
  1. Prepare clusters.csv for recordering. This is a CSV file with 3 columns: landscape name, project name, shoot name An example clusters.csv file for recording data for the g2 cluster of project i034796 in the dev landscape is shown below. 
    dev,i034796,g2
  2. Store clusters.csv in a directory configDir.
  3. Kindly set environment variables
    1. export MODE=local
    2. export CONFIG_DIR=<configDir> # Dir of clusters.csv
    3. export DB_DIR=/tmp # Directory where recorder stores SQLite <clusterName>.db files
  4. Execute: go run cmd/recorder/main.go
Remote Launch
  1. Login into utility-int cluster gardenctl target --garden sap-landscape-live --project garden-ops --shoot utility-int
  2. Export your docker hub username: export DOCKERHUB_USER=<dockerHubUser>
  3. Login into Docker Hub: docker login -u $DOCKERHUB_USER -p <dockerHubPass>
  4. Run ./hack/build-recorder.sh
  5. Run ./hack/build-recorder.sh

Download Recorder Databases

Download Automatically into gen dir
  1. Ensure you have garden live landscape access.
  2. Kindly run the script ./hack/download-db.sh
Download Manually
  1. Port forward the recorder Pod's 8080 port locally
    1. kubectl port-forward -n robot pod/scaling-history-recorder 8080:8080
  2. Use curl to list recorder SQLite DBs:
    1. curl localhost:8080/api/db
    2. This will list the .db files
    live_hc-ap11_prod-haas.db
live_hc-ap11_prod-hdl.db
live_hc-ap11_prod-orc.db
live_hc-canary_prod-haas.db
live_hc-canary_prod-hna0.db
live_hc-eu20_prod-az-haas.db
live_hc-eu20_prod-az-orc.db
  3. Use curl to download a specific DB
    1. cd /tmp; curl -kLO localhost:8080/api/db/live_hc-ap11_prod-hdl.db
  4. End the port-forwarding.
  5. Use any DB Browser of your choice to open downloaded DB

Launch the Replayer

The replayer operates replays scenarios and generates a report. It needs a INPUT_DATA_PATH which is a path to the recorded DB Path which can also be a generated scenario json.

In either case it generates another scenario report.

Replay Virtual CA
Launch Replayer Locally with virtual CA to generate report
  1. Launch the virtual cluster: KVCL.
  2. Export INPUT_DATA_PATH. Ex: export INPUT_DATA_PATH=dev_i034796_g2.db
  3. (optional) You may also optionally change default of /tmp for the scenario REPORT_DIR. Ex: REPORT_DIR=/tmp
  4. (optional) You may also optionally change default of /tmp/kvcl.yaml for the virtual KUBECONFIG. Ex: KUBECONFIG=cfg/virtual-kubeconfig.yaml
  5. Execute replayer: go run cmd/replayer/main.go
Launch Replayer Remotely with virtual CA to generate report.
  1. Login into utility-int cluster gardenctl target --garden sap-landscape-live --project garden-ops --shoot utility-int
  2. Export your docker hub username: export DOCKERHUB_USER=<dockerHubUser>
  3. Login into Docker Hub: docker login -u $DOCKERHUB_USER -p <dockerHubPass>
  4. Ensure you are at the base dir of the gardener-scaling-history
  5. Run ./hack/build-replayer.sh remote
  6. Run ./hack/replay-ca.sh
    1. This will list all db's available
    2. Choose the db you want to run the replayer with.
    3. It will then deploy a scaling-history-replayer-xxxxx Pod that will run the replayer and generate the report into the configured REPORT_DIR ( /reports mapped to a PV by default)
    4. Replay reports look like <landscape>_<project>_<shoot>_ca-replay-<number>.json where number corresponds to the scaling event index. 2. Ex: live_hc-eu10_prod-haas_ca-replay-0.json
  7. You can download the reports as described in the subsequent section.
Replay Scaling Recommender

[!NOTE] This Replay mode operates on the scenario report json generated by Replay Virtual CA.

Launch replay scaling recommender locally
  1. Run ./hack/build-replay.sh local
  2. Run export INPUT_DATA_PATH=<path to new scenario json produced by replay CA>
    1. (eg: export INPUT_DATA_PATH=$GOPATH/src/github.tools.sap/I034796/gardener-scaling-reports/independent-scenarios/live_hc-eu30_prod-gc-dmi_ca-replay-5.json)
  3. Run go run cmd/replayer/main.go
Launch replay scaling recommender remotely
  1. Login into utility-int cluster gardenctl target --garden sap-landscape-live --project garden-ops --shoot utility-int
  2. Export your docker hub username: export DOCKERHUB_USER=<dockerHubUser>
  3. Login into Docker Hub: docker login -u $DOCKERHUB_USER -p <dockerHubPass>
  4. Ensure you are at the base dir of the gardener-scaling-history
  5. Run ./hack/build-replayer.sh remote
  6. Run ./hack/replay-sr.sh
    1. This will list all replay-CA reports available
    2. Choose the report you want to run the replayer with.
    3. It will then deploy a scaling-history-replayer-xxxxx Pod that will run the replayer, generate the report and then upload the generated report into the /data/reports directory of the PV scaling-history-data
    4. Replay reports look like <landscape>_<project>_<shoot>_sr-replay-<number>.json where number corresponds to the scaling event index.
      1. Ex: live_hc-eu10_prod-haas_sr-replay-0.json
  7. You can download the reports as described in the subsequent section.

Download Reports

Download reports into /tmp dir
  1. Ensure you have garden live landscape access.
  2. Kindly run the script ./hack/download-reports.sh
  3. This will download replay reports into your local /tmp directory.
Report Download API
  1. You can get a listing of reports by executing curl http://10.47.254.238/api/reports
  2. You can download a specific report into your current directory by using: curl -LO <reportUrl>
    1. Ex: curl -LO http://10.47.254.238/api/reports/live_hc-canary_prod-hna1_ca-replay-1.json

Download Logs

Download Logs API
  1. You can get a listing of all logs files by executing ``curl http://10.47.254.238/api/logs`
  2. You can get a listing of logs files for a cluster by executing curl http://10.47.254.238/api/logs/<clusterName>

Documentation

Index

Constants

View Source 
const PoolLabel = "worker.gardener.cloud/pool"
View Source 
const PoolLabelAlt = "worker_gardener_cloud_pool"
View Source 
const PoolLabelAlt1 = "worker.gardener.cloud_pool"

Variables

View Source 
var PoolLabels = []string{
	PoolLabel, PoolLabelAlt, PoolLabelAlt1,
}

Functions

func GardenerTimestampFromUnstructured 

func GardenerTimestampFromUnstructured(worker *unstructured.Unstructured) (gardenerTimeStamp time.Time, err error)

func IsEqualQuantity 

func IsEqualQuantity(a, b resource.Quantity) bool

func MachineDeploymentInfoFromUnstructured 

func MachineDeploymentInfoFromUnstructured(mcd *unstructured.Unstructured, snapshotTime time.Time) (mcdInfo gsc.MachineDeploymentInfo, err error)

func NodeInfoFromNode 

func NodeInfoFromNode(n *corev1.Node, allocatableVolumes int) gsc.NodeInfo

func ResourceListFromMap 

func ResourceListFromMap(input map[string]any) (corev1.ResourceList, error)

func WorkerPoolInfosFromUnstructured 

func WorkerPoolInfosFromUnstructured(worker *unstructured.Unstructured) (map[string]gsc.WorkerPoolInfo, error)

Types

type ExecutionMode 

type ExecutionMode string
var (
	InUtilityClusterMode ExecutionMode = "in-utility-cluster"
	LocalMode            ExecutionMode = "local"
)

type FileInfo 

type FileInfo struct {
	Name         string
	URL          string
	LastModified time.Time
	Size         uint64
	ReadableSize string
}

type FileInfos 

type FileInfos struct {
	Items []FileInfo
}

type MachineClassInfo 

type MachineClassInfo struct {
	gsc.SnapshotMeta

	// Instance type of the node belonging to nodeGroup
	InstanceType string

	// PoolName is the name of the gardener shoot worker pool that this machine class belongs to
	PoolName string

	// Region of the node belonging to nodeGroup
	Region string

	// Zone of the node that will be associated with this machine class
	Zone string

	// Labels is the machine class provider spec labels.
	Labels map[string]string

	// Capacity contains subfields to track all node resources required to scale nodegroup from zero
	Capacity corev1.ResourceList

	DeletionTimestamp time.Time
	Hash              string
}

func MachineClassInfoFromUnstructured 

func MachineClassInfoFromUnstructured(mcc *unstructured.Unstructured, snapshotTime time.Time) (mccInfo MachineClassInfo, err error)

func (MachineClassInfo) GetHash 

func (m MachineClassInfo) GetHash() string

func (MachineClassInfo) String 

func (m MachineClassInfo) String() string

type PodInfoKey 

type PodInfoKey struct {
	UID  string
	Name string
	Hash string
}

type PoolZone 

type PoolZone struct {
	PoolName string
	Zone     string
}

func (PoolZone) String 

func (pz PoolZone) String() string

type Recorder 

type Recorder interface {
	io.Closer
	Start() error
	IsStarted() bool
}

Recorder monitors the cluster denoted by given kubeconfig and records events and cluster data into cluster database

type RecorderParams 

type RecorderParams struct {
	Mode                ExecutionMode
	Landscape           string
	ProjectName         string
	ShootName           string
	SeedName            string
	ShootNameSpace      string
	ShootKubeConfigPath string
	SeedKubeConfigPath  string
	DBDir               string
	// contains filtered or unexported fields
}

func (RecorderParams) ShootLabel 

func (rp RecorderParams) ShootLabel() string

func (RecorderParams) String 

func (rp RecorderParams) String() string

type ReplayReport 

type ReplayReport struct {
	StartTime time.Time
	Scenarios []Scenario
}

type Replayer 

type Replayer interface {
	io.Closer
	Start() error
	GetRecordedClusterSnapshot(runPrevMarkTime, runMarkTime time.Time) (gsc.ClusterSnapshot, error)
	GetParams() ReplayerParams
}

type ReplayerParams 

type ReplayerParams struct {
	Mode                         ExecutionMode
	InputDataPath                string
	ReportDir                    string
	VirtualAutoScalerConfigPath  string
	VirtualClusterKubeConfigPath string
	DeployParallel               int
	ReplayInterval               time.Duration
	AutoLaunchDependencies       bool
}

type ResourceStats 

type ResourceStats struct {
	TotalUtilCPU  resource.Quantity
	TotalUtilMem  resource.Quantity
	AvailAllocCPU resource.Quantity
	AvailAllocMem resource.Quantity
}

type ScalingResult 

type ScalingResult struct {
	ScaledUpNodeGroups     map[string]int
	ScaledUpNodes          []gsc.NodeInfo
	PendingUnscheduledPods []gsc.PodInfo
	NodesUtilization       map[string]corev1.ResourceList
	EmptyNodeNames         []string
	ScheduledPods          []gsc.PodInfo
}

type Scenario 

type Scenario struct {
	BeginTime       time.Time
	ClusterSnapshot gsc.ClusterSnapshot
	ScalingResult   ScalingResult
}

func (Scenario) GetResourceStat 

func (report Scenario) GetResourceStat() (stats ResourceStats, err error)

Source Files

View all Source files

Directories

Path Synopsis
cmd
app
comparer
dataverify
desktopapp
emptynodes
recorder
replayer

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.