flinkk8soperator

README

Flinkk8soperator

FlinkK8sOperator is a Kubernetes operator that manages Flink applications on Kubernetes. The operator acts as control plane to manage the complete deployment lifecycle of the application.

Project Status

Alpha

The FlinkK8sOperator is still under active development and has not been extensively tested in production environment. Backward compatibility of the APIs is not guaranteed for alpha releases.

Prerequisites

• Version >= 1.9 of Kubernetes.
• Version >= 1.7 of Apache Flink.

Overview

The goal of running Flink on Kubernetes is to enable more flexible, lighter-weight deployment of streaming applications, without needing to manage infrastructure. The Flink operator aims to abstract out the complexity of hosting, configuring, managing and operating Flink clusters from application developers. It achieves this by extending any kubernetes cluster using a custom resources.

The Operator creates flink clusters dynamically using the specified custom resource. Flink clusters in kubernetes consist of the following:

• JobManager Deployment
• TaskManager Deployment
• JobManager Service
• JobManager Ingress for the UI

Deploying and managing Flink applications in Kubernetes involves two steps:

• Building Flink application packaged as a docker image: A docker image is built containing the application source code with the necessary dependencies built in. This is required to bootstrap the Jobmanager and Taskmanager pods. At Lyft we use Source-To-Image S2I as the image build tool that provides a common builder image with Apache Flink pre-installed. The docker image could be built using any pre-existing workflows at an organization.

• Creating the Flink application custom resource: The custom resource for Flink application provides the spec for configuring and managing flink clusters in Kubernetes. The FlinkK8sOperator, deployed on Kubernetes, continuously monitors the resource and the corresponding flink cluster, and performs action based on the diff.

Documentation

• Quick start guide
• User guide
• Flink application custom resource
• Operator state machine