e2e

package
v1.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 17, 2016 License: Apache-2.0 Imports: 78 Imported by: 0

Documentation

Overview

This is a utility for prometheus pushing functionality.

Index

Constants

View Source 
const (
	ADD    = "ADD"
	DEL    = "DEL"
	UPDATE = "UPDATE"
)
View Source 
const NodeStartupThreshold = 4 * time.Second

NodeStartupThreshold is a rough estimate of the time allocated for a pod to start on a node.

Variables

View Source 
var InterestingApiServerMetrics = []string{
	"apiserver_request_count",
	"apiserver_request_latencies_bucket",
	"etcd_helper_cache_entry_count",
	"etcd_helper_cache_hit_count",
	"etcd_helper_cache_miss_count",
	"etcd_request_cache_add_latencies_summary",
	"etcd_request_cache_get_latencies_summary",
	"etcd_request_latencies_summary",
	"go_gc_duration_seconds",
	"go_goroutines",
	"process_cpu_seconds_total",
	"process_open_fds",
	"process_resident_memory_bytes",
	"process_start_time_seconds",
	"process_virtual_memory_bytes",
}
View Source 
var InterestingKubeletMetrics = []string{
	"go_gc_duration_seconds",
	"go_goroutines",
	"kubelet_container_manager_latency_microseconds",
	"kubelet_docker_errors",
	"kubelet_docker_operations_latency_microseconds",
	"kubelet_generate_pod_status_latency_microseconds",
	"kubelet_pod_start_latency_microseconds",
	"kubelet_pod_worker_latency_microseconds",
	"kubelet_pod_worker_start_latency_microseconds",
	"kubelet_sync_pods_latency_microseconds",
	"process_cpu_seconds_total",
	"process_open_fds",
	"process_resident_memory_bytes",
	"process_start_time_seconds",
	"process_virtual_memory_bytes",
}
View Source 
var MaxContainerFailures = 0

Maximum container failures this test tolerates before failing.

View Source 
var ServiceNodePortRange = utilnet.PortRange{Base: 30000, Size: 2768}

This should match whatever the default/configured range is

Functions

func BadEvents 

func BadEvents(events []*api.Event) int

Prints the histogram of the events and returns the number of bad events.

func CheckCadvisorHealthOnAllNodes 

func CheckCadvisorHealthOnAllNodes(c *client.Client, timeout time.Duration)

func ClusterLevelLoggingWithElasticsearch 

func ClusterLevelLoggingWithElasticsearch(f *Framework)

ClusterLevelLoggingWithElasticsearch is an end to end test for cluster level logging.

func ClusterLevelLoggingWithKibana 

func ClusterLevelLoggingWithKibana(f *Framework)

ClusterLevelLoggingWithKibana is an end to end test that checks to see if Kibana is alive.

func CoreDump 

func CoreDump(dir string)

func DeleteRC 

func DeleteRC(c *client.Client, ns, name string) error

Delete a Replication Controller and all pods it spawned

func FailedContainers 

func FailedContainers(pod *api.Pod) map[string]ContainerFailures

FailedContainers inspects all containers in a pod and returns failure information for containers that have failed or been restarted. A map is returned where the key is the containerID and the value is a struct containing the restart and failure information

func Failf 

func Failf(format string, a ...interface{})

func GetKubeletPods 

func GetKubeletPods(c *client.Client, node string) (*api.PodList, error)

GetKubeletPods retrieves the list of running pods on the kubelet. The pods includes necessary information (e.g., UID, name, namespace for pods/containers), but do not contain the full spec.

func HighLatencyRequests 

func HighLatencyRequests(c *client.Client) (int, error)

Prints top five summary metrics for request types with latency and returns number of such request types above threshold.

func LaunchHostExecPod added in v1.1.2 

func LaunchHostExecPod(client *client.Client, ns, name string) *api.Pod

LaunchHostExecPod launches a hostexec pod in the given namespace and waits until it's Running

func LaunchNetTestPodPerNode 

func LaunchNetTestPodPerNode(f *Framework, nodes *api.NodeList, name, version string) []string

func ListSchedulableNodesOrDie added in v1.1.3 

func ListSchedulableNodesOrDie(c *client.Client) *api.NodeList

Convenient wrapper around listing nodes supporting retries.

func LogSSHResult added in v1.1.2 

func LogSSHResult(result SSHResult)

func Logf 

func Logf(format string, a ...interface{})

func NewHostExecPodSpec added in v1.1.2 

func NewHostExecPodSpec(ns, name string) *api.Pod

NewHostExecPodSpec returns the pod spec of hostexec pod

func NewRestartConfig added in v1.0.7 

func NewRestartConfig(nodeName, daemonName string, healthzPort int, pollInterval, pollTimeout time.Duration) *restartDaemonConfig

NewRestartConfig creates a restartDaemonConfig for the given node and daemon.

func NodeAddresses added in v1.1.2 

func NodeAddresses(nodelist *api.NodeList, addrType api.NodeAddressType) []string

NodeAddresses returns the first address of the given type of each node.

func NodeSSHHosts 

func NodeSSHHosts(c *client.Client) ([]string, error)

NodeSSHHosts returns SSH-able host names for all nodes. It returns an error if it can't find an external IP for every node, though it still returns all hosts that it found in that case.

func OpenWebSocketForURL added in v1.0.7 

func OpenWebSocketForURL(url *url.URL, config *client.Config, protocols []string) (*websocket.Conn, error)

OpenWebSocketForURL constructs a websocket connection to the provided URL, using the client config, with the specified protocols.

func ReserveCpu 

func ReserveCpu(f *Framework, id string, millicores int)

func ReserveMemory 

func ReserveMemory(f *Framework, id string, megabytes int)

func RunDeployment added in v1.1.2 

func RunDeployment(config DeploymentConfig) error

RunDeployment Launches (and verifies correctness) of a Deployment and will wait for all pods it spawns to become "Running". It's the caller's responsibility to clean up externally (i.e. use the namespace lifecycle for handling cleanup).

func RunHostCmd added in v1.1.2 

func RunHostCmd(ns, name, cmd string) (string, error)

RunHostCmd runs the given cmd in the context of the given pod using `kubectl exec` inside of a shell.

func RunHostCmdOrDie added in v1.1.2 

func RunHostCmdOrDie(ns, name, cmd string) string

RunHostCmdOrDie calls RunHostCmd and dies on error.

func RunRC 

func RunRC(config RCConfig) error

RunRC Launches (and verifies correctness) of a Replication Controller and will wait for all pods it spawns to become "Running". It's the caller's responsibility to clean up externally (i.e. use the namespace lifecycle for handling cleanup).

func ScaleRC 

func ScaleRC(c *client.Client, ns, name string, size uint, wait bool) error

func ServeImageOrFail 

func ServeImageOrFail(f *Framework, test string, image string)

A basic test to check the deployment of an image using a replication controller. The image serves its hostname which is checked for each replica.

func SetTestContext 

func SetTestContext(t TestContextType)

func SkipIfProviderIs 

func SkipIfProviderIs(unsupportedProviders ...string)

func SkipUnlessNodeCountIsAtLeast 

func SkipUnlessNodeCountIsAtLeast(minNodeCount int)

func SkipUnlessProviderIs 

func SkipUnlessProviderIs(supportedProviders ...string)

func Skipf 

func Skipf(format string, args ...interface{})

func VerifyPodStartupLatency added in v1.0.7 

func VerifyPodStartupLatency(latency PodStartupLatency) error

Verifies whether 50, 90 and 99th percentiles of PodStartupLatency are within the threshold.

func VerifySchedulerLatency added in v1.1.2 

func VerifySchedulerLatency(c *client.Client) error

Verifies (currently just by logging them) the scheduling latencies.

Types

type APICall added in v1.0.7 

type APICall struct {
	Resource string        `json:"resource"`
	Verb     string        `json:"verb"`
	Latency  LatencyMetric `json:"latency"`
}

type APIResponsiveness added in v1.0.7 

type APIResponsiveness struct {
	APICalls []APICall `json:"apicalls"`
}

func (APIResponsiveness) Len added in v1.0.7 

func (a APIResponsiveness) Len() int

func (APIResponsiveness) Less added in v1.0.7 

func (a APIResponsiveness) Less(i, j int) bool

func (APIResponsiveness) Swap added in v1.0.7 

func (a APIResponsiveness) Swap(i, j int)

type CloudConfig 

type CloudConfig struct {
	ProjectID         string
	Zone              string
	Cluster           string
	MasterName        string
	NodeInstanceGroup string
	NumNodes          int
	ClusterTag        string
	ServiceAccount    string

	Provider cloudprovider.Interface
}

type ContainerFailures 

type ContainerFailures struct {
	// contains filtered or unexported fields
}

type DeploymentConfig added in v1.1.2 

type DeploymentConfig struct {
	RCConfig
}

type Framework 

type Framework struct {
	BaseName string

	Namespace                *api.Namespace
	Client                   *client.Client
	NamespaceDeletionTimeout time.Duration
	// contains filtered or unexported fields
}

Framework supports common operations used by e2e tests; it will keep a client & a namespace for you. Eventual goal is to merge this with integration test framework.

func InitializeFramework 

func InitializeFramework(baseName string, nsCreateFunc func(string, *client.Client, map[string]string) (*api.Namespace, error)) *Framework

InitializeFramework initialize the framework by allowing to pass a custom namespace creation function.

func NewFramework 

func NewFramework(baseName string) *Framework

NewFramework makes a new framework and sets up a BeforeEach/AfterEach for you (you can write additional before/after each functions).

func (*Framework) ReadFileViaContainer 

func (f *Framework) ReadFileViaContainer(podName, containerName string, path string) (string, error)

Read a file using kubectl exec cat <path>

func (*Framework) TestContainerOutput 

func (f *Framework) TestContainerOutput(scenarioName string, pod *api.Pod, containerIndex int, expectedOutput []string)

Runs the given pod and verifies that the output of exact container matches the desired output.

func (*Framework) TestContainerOutputRegexp added in v1.0.7 

func (f *Framework) TestContainerOutputRegexp(scenarioName string, pod *api.Pod, containerIndex int, expectedOutput []string)

Runs the given pod and verifies that the output of exact container matches the desired regexps.

func (*Framework) WaitForAnEndpoint 

func (f *Framework) WaitForAnEndpoint(serviceName string) error

WaitForAnEndpoint waits for at least one endpoint to become available in the service's corresponding endpoints object.

func (*Framework) WaitForPodRunning 

func (f *Framework) WaitForPodRunning(podName string) error

WaitForPodRunning waits for the pod to run in the namespace.

func (*Framework) WaitForPodRunningSlow added in v1.1.2 

func (f *Framework) WaitForPodRunningSlow(podName string) error

WaitForPodRunningSlow waits for the pod to run in the namespace. It has a longer timeout then WaitForPodRunning (util.slowPodStartTimeout).

func (*Framework) WaitForPodTerminated added in v1.1.3 

func (f *Framework) WaitForPodTerminated(podName, reason string) error

WaitForPodTerminated waits for the pod to be terminated with the given reason.

func (*Framework) WriteFileViaContainer 

func (f *Framework) WriteFileViaContainer(podName, containerName string, path string, contents string) error

Write a file using kubectl exec echo <contents> > <path> via specified container Because of the primitive technique we're using here, we only allow ASCII alphanumeric characters

type HPAScaleTest added in v1.1.2 

type HPAScaleTest struct {
	// contains filtered or unexported fields
}

HPAScaleTest struct is used by the scale(...) function.

type IngressController added in v1.1.2 

type IngressController struct {
	UID     string
	Project string
	// contains filtered or unexported fields
}

func (*IngressController) Cleanup added in v1.1.2 

func (cont *IngressController) Cleanup(del bool) error

type KubeProxyTestConfig added in v1.0.7 

type KubeProxyTestConfig struct {
	// contains filtered or unexported fields
}

type KubeletManagedHostConfig added in v1.0.8 

type KubeletManagedHostConfig struct {
	// contains filtered or unexported fields
}

type KubeletMetric 

type KubeletMetric struct {
	// eg: list, info, create
	Operation string
	// eg: sync_pods, pod_worker
	Method string
	// 0 <= quantile <=1, e.g. 0.95 is 95%tile, 0.5 is median.
	Quantile float64
	Latency  time.Duration
}

KubeletMetric stores metrics scraped from the kubelet server's /metric endpoint. TODO: Get some more structure around the metrics and this type

func HighLatencyKubeletOperations 

func HighLatencyKubeletOperations(c *client.Client, threshold time.Duration, nodeName string) ([]KubeletMetric, error)

HighLatencyKubeletOperations logs and counts the high latency metrics exported by the kubelet server via /metrics.

func ParseKubeletMetrics 

func ParseKubeletMetrics(metricsBlob string) ([]KubeletMetric, error)

ParseKubeletMetrics reads metrics from the kubelet server running on the given node

type KubeletMetricByLatency 

type KubeletMetricByLatency []KubeletMetric

KubeletMetricByLatency implements sort.Interface for []KubeletMetric based on the latency field.

func (KubeletMetricByLatency) Len 

func (a KubeletMetricByLatency) Len() int

func (KubeletMetricByLatency) Less 

func (a KubeletMetricByLatency) Less(i, j int) bool

func (KubeletMetricByLatency) Swap 

func (a KubeletMetricByLatency) Swap(i, j int)

type LBCTester added in v1.0.7 

type LBCTester interface {
	// contains filtered or unexported methods
}

LBCTester is an interface used to test loadbalancer controllers.

type LatencyMetric 

type LatencyMetric struct {
	Perc50 time.Duration `json:"Perc50"`
	Perc90 time.Duration `json:"Perc90"`
	Perc99 time.Duration `json:"Perc99"`
}

Dashboard metrics

type LogSizeDataTimeseries added in v1.1.3 

type LogSizeDataTimeseries map[string]map[string][]TimestampedSize

type LogSizeGatherer added in v1.1.2 

type LogSizeGatherer struct {
	// contains filtered or unexported fields
}

LogSizeGatherer is a worker which grabs a WorkItem from the channel and does assigned work.

func (*LogSizeGatherer) Run added in v1.1.2 

func (g *LogSizeGatherer) Run()

func (*LogSizeGatherer) Work added in v1.1.2 

func (g *LogSizeGatherer) Work() bool

Work does a single unit of work: tries to take out a WorkItem from the queue, ssh-es into a given machine, gathers data, writes it to the shared <data> map, and creates a gorouting which reinserts work item into the queue with a <pollingPeriod> delay. Returns false if worker should exit.

type LogsSizeData added in v1.1.2 

type LogsSizeData struct {
	// contains filtered or unexported fields
}

func (*LogsSizeData) AddNewData added in v1.1.2 

func (d *LogsSizeData) AddNewData(ip, path string, timestamp time.Time, size int)

type LogsSizeDataSummary added in v1.1.3 

type LogsSizeDataSummary map[string]map[string]SingleLogSummary

node -> file -> data

func (*LogsSizeDataSummary) PrintHumanReadable added in v1.1.3 

func (s *LogsSizeDataSummary) PrintHumanReadable() string

TODO: make sure that we don't need locking here

func (*LogsSizeDataSummary) PrintJSON added in v1.1.3 

func (s *LogsSizeDataSummary) PrintJSON() string

type LogsSizeVerifier added in v1.1.2 

type LogsSizeVerifier struct {
	// contains filtered or unexported fields
}

LogsSizeVerifier gathers data about log files sizes from master and node machines. It oversees a <workersNo> workers which do the gathering.

func NewLogsVerifier added in v1.1.2 

func NewLogsVerifier(c *client.Client, stopChannel chan bool) *LogsSizeVerifier

NewLogsVerifier creates a new LogsSizeVerifier which will stop when stopChannel is closed

func (*LogsSizeVerifier) GetSummary added in v1.1.3 

func (s *LogsSizeVerifier) GetSummary() *LogsSizeDataSummary

GetSummary returns a summary (average generation rate and number of probes) of the data gathered by LogSizeVerifier

func (*LogsSizeVerifier) Run added in v1.1.2 

func (v *LogsSizeVerifier) Run()

Run starts log size gathering. It starts a gorouting for every worker and then blocks until stopChannel is closed

type MetricsForE2E added in v1.1.3 

type MetricsForE2E metrics.MetricsCollection

func (*MetricsForE2E) PrintHumanReadable added in v1.1.3 

func (m *MetricsForE2E) PrintHumanReadable() string

func (*MetricsForE2E) PrintJSON added in v1.1.3 

func (m *MetricsForE2E) PrintJSON() string

type PodDiff 

type PodDiff map[string]*podInfo

PodDiff is a map of pod name to podInfos

func Diff 

func Diff(oldPods []*api.Pod, curPods []*api.Pod) PodDiff

Diff computes a PodDiff given 2 lists of pods.

func (PodDiff) Print 

func (p PodDiff) Print(ignorePhases sets.String)

Print formats and prints the give PodDiff.

type PodStartupLatency added in v1.0.7 

type PodStartupLatency struct {
	Latency LatencyMetric `json:"latency"`
}

type PortsByPodName 

type PortsByPodName map[string][]int

type PortsByPodUID 

type PortsByPodUID map[types.UID][]int

type PrivilegedPodTestConfig added in v1.0.7 

type PrivilegedPodTestConfig struct {
	// contains filtered or unexported fields
}

type RCConfig 

type RCConfig struct {
	Client        *client.Client
	Image         string
	Command       []string
	Name          string
	Namespace     string
	PollInterval  time.Duration
	Timeout       time.Duration
	PodStatusFile *os.File
	Replicas      int
	CpuRequest    int64 // millicores
	CpuLimit      int64 // millicores
	MemRequest    int64 // bytes
	MemLimit      int64 // bytes

	// Env vars, set the same for every pod.
	Env map[string]string

	// Extra labels added to every pod.
	Labels map[string]string

	// Ports to declare in the container (map of name to containerPort).
	Ports map[string]int

	// Pointer to a list of pods; if non-nil, will be set to a list of pods
	// created by this RC by RunRC.
	CreatedPods *[]*api.Pod

	// Maximum allowable container failures. If exceeded, RunRC returns an error.
	// Defaults to replicas*0.1 if unspecified.
	MaxContainerFailures *int
}

type ResourceConsumer added in v1.0.7 

type ResourceConsumer struct {
	// contains filtered or unexported fields
}

ResourceConsumer is a tool for testing. It helps create specified usage of CPU or memory (Warning: memory not supported) typical use case: rc.ConsumeCPU(600) // ... check your assumption here rc.ConsumeCPU(300) // ... check your assumption here

func NewDynamicResourceConsumer added in v1.0.7 

func NewDynamicResourceConsumer(name, kind string, replicas, initCPUTotal, initMemoryTotal int, cpuLimit, memLimit int64, framework *Framework) *ResourceConsumer

func NewStaticResourceConsumer added in v1.0.7 

func NewStaticResourceConsumer(name string, replicas, initCPUTotal, initMemoryTotal int, cpuLimit, memLimit int64, framework *Framework) *ResourceConsumer

TODO this still defaults to replication controller

func (*ResourceConsumer) CleanUp added in v1.0.7 

func (rc *ResourceConsumer) CleanUp()

func (*ResourceConsumer) ConsumeCPU added in v1.0.7 

func (rc *ResourceConsumer) ConsumeCPU(millicores int)

ConsumeCPU consumes given number of CPU

func (*ResourceConsumer) ConsumeMem added in v1.0.7 

func (rc *ResourceConsumer) ConsumeMem(megabytes int)

ConsumeMem consumes given number of Mem

func (*ResourceConsumer) EnsureDesiredReplicas added in v1.1.2 

func (rc *ResourceConsumer) EnsureDesiredReplicas(desiredReplicas int, timeout time.Duration)

func (*ResourceConsumer) GetReplicas added in v1.0.7 

func (rc *ResourceConsumer) GetReplicas() int

func (*ResourceConsumer) WaitForReplicas added in v1.0.7 

func (rc *ResourceConsumer) WaitForReplicas(desiredReplicas int)

type ResourceUsageSummary added in v1.1.3 

type ResourceUsageSummary map[string][]SingleContainerSummary

we can't have int here, as JSON does not accept integer keys.

func (*ResourceUsageSummary) PrintHumanReadable added in v1.1.3 

func (s *ResourceUsageSummary) PrintHumanReadable() string

func (*ResourceUsageSummary) PrintJSON added in v1.1.3 

func (s *ResourceUsageSummary) PrintJSON() string

type SSHResult added in v1.1.2 

type SSHResult struct {
	User   string
	Host   string
	Cmd    string
	Stdout string
	Stderr string
	Code   int
}

func SSH 

func SSH(cmd, host, provider string) (SSHResult, error)

SSH synchronously SSHs to a node running on provider and runs cmd. If there is no error performing the SSH, the stdout, stderr, and exit code are returned.

type SchedulingLatency added in v1.1.2 

type SchedulingLatency struct {
	Scheduling LatencyMetric `json:"scheduling:`
	Binding    LatencyMetric `json:"binding"`
	Total      LatencyMetric `json:"total"`
}

type ServerTest added in v1.1.3 

type ServerTest struct {
	ServiceName string
	Namespace   string
	Client      *client.Client

	TestId string
	Labels map[string]string
	// contains filtered or unexported fields
}

Simple helper class to avoid too much boilerplate in tests

func NewNetcatTest added in v1.1.3 

func NewNetcatTest(client *client.Client, namespace string, serviceName string) *ServerTest

func NewServerTest added in v1.1.3 

func NewServerTest(client *client.Client, namespace string, serviceName string) *ServerTest

func (*ServerTest) BuildServiceSpec added in v1.1.3 

func (t *ServerTest) BuildServiceSpec() *api.Service

Build default config for a service (which can then be changed)

func (*ServerTest) Cleanup added in v1.1.3 

func (t *ServerTest) Cleanup() []error

func (*ServerTest) CreateNetcatRC added in v1.1.3 

func (t *ServerTest) CreateNetcatRC(replicas int) *api.ReplicationController

CreateNetcatRC creates rc-backed pods with a netcat listener configuration and records it for cleanup.

func (*ServerTest) CreateService added in v1.1.3 

func (t *ServerTest) CreateService(service *api.Service) (*api.Service, error)

Create a service, and record it for cleanup

func (*ServerTest) CreateWebserverRC added in v1.1.3 

func (t *ServerTest) CreateWebserverRC(replicas int) *api.ReplicationController

CreateWebserverRC creates rc-backed pods with the well-known webserver configuration and records it for cleanup.

func (*ServerTest) DeleteService added in v1.1.3 

func (t *ServerTest) DeleteService(serviceName string) error

Delete a service, and remove it from the cleanup list

type SingleContainerSummary added in v1.1.3 

type SingleContainerSummary struct {
	Name string
	Cpu  float64
	Mem  int64
}

type SingleLogSummary added in v1.1.3 

type SingleLogSummary struct {
	AverageGenerationRate int
	NumberOfProbes        int
}

type State 

type State struct {
	Received map[string]int
}

partially cloned from webserver.go

type TestContextType 

type TestContextType struct {
	KubeConfig    string
	KubeContext   string
	KubeVolumeDir string
	CertDir       string
	Host          string
	RepoRoot      string
	Provider      string
	CloudConfig   CloudConfig
	KubectlPath   string
	OutputDir     string
	ReportDir     string

	MinStartupPods        int
	UpgradeTarget         string
	PrometheusPushGateway string
	VerifyServiceAccount  bool
	DeleteNamespace       bool
	CleanStart            bool
	// If set to true framework will start a goroutine monitoring resource usage of system add-ons.
	// It will read the data every 30 seconds from all Nodes and print summary during afterEach.
	GatherKubeSystemResourceUsageData bool
	GatherLogsSizes                   bool
	GatherMetricsAfterTest            bool
	// Currently supported values are 'hr' for human-readable and 'json'. It's a comma separated list.
	OutputPrintType string
	// contains filtered or unexported fields
}

type TestDataSummary added in v1.1.3 

type TestDataSummary interface {
	PrintHumanReadable() string
	PrintJSON() string
}

type TimestampedSize added in v1.1.2 

type TimestampedSize struct {
	// contains filtered or unexported fields
}

TimestampedSize contains a size together with a time of measurement.

type VolumeTestConfig 

type VolumeTestConfig struct {
	// contains filtered or unexported fields
}

Configuration of one tests. The test consist of: - server pod - runs serverImage, exports ports[] - client pod - does not need any special configuration

type WorkItem added in v1.1.2 

type WorkItem struct {
	// contains filtered or unexported fields
}

WorkItem is a command for a worker that contains an IP of machine from which we want to gather data and paths to all files we're interested in.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.