kube2ram
English | 简体中文
Context
In current Alibaba ACK cluster runtime container, if you want to communication with other Alibaba Cloud resource as oss, slb..., you should call ECS metadata API to fetch the interim security credentials for RAM authorization.
But in a multi-tenanted containers based world, different users will deloy multiple containers in the same worker node, since only one worker RAM role could be fetch through ECS instance profile, the different containers could only sharing the security credentials mapping to the same RAM role, which is not acceptable from the security isolation perspective.
kube2ram is just the solution based on this issue, it could deploy on each worker node as a daemonst, and redirect the tracffic that is going to the ECS metadata server from bussiness container to kube2ram instance, make a call to the ECS API to fetch interim credentials and return these to the caller. Other calls will be still proxied to the ECS metadata server.
Configuration
RAM roles
Call the metadata server http://100.100.100.200/latest/meta-data/ram/security-credentials/ on worker node to get the default RAM role name (prefix with KubernetesWorkerRole
), then open RAM console-> RAM roles page, search and find the role with the given name, then go to the RAM policies page and append the policy below in Policy Document tab.(In this case you grant the target KubernetesWorkerRole to assume any other RAM role, and you can also set one or more RAM role in the Resource
vaule for fine-grained control):
{
"Version": "1",
"Statement": [
.../*omit the granted policies here*/
{
"Action": [
"sts:AssumeRole"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
also you need to find all of the target assumeroles, and make sure the account uid exists in trust Principal
as below:
{
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"RAM": [
"acs:ram::xxxxxxx:root"
]
}
}
],
"Version": "1"
}
- xxxxxxx need replace by the Alibaba Cloud account uid who own this worker node.
kube2ram daemonset
Deploy the kube2ram daemonset with the template below, please notice that the kube2ram daemon and iptables rule need to run before all other pods that would require access to Alibaba Cloud resources.
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app.kubernetes.io/name: kube2ram
name: kube2ram
namespace: kube-system
spec:
selector:
matchLabels:
app.kubernetes.io/name: kube2ram
template:
metadata:
labels:
app.kubernetes.io/name: kube2ram
spec:
containers:
- name: kube2ram
image: registry.cn-hangzhou.aliyuncs.com/acs/kube2ram:1.0.0
imagePullPolicy: Always
args:
- "--app-port=8181"
- "--iptables=true"
- "--host-ip=$(HOST_IP)"
- "--host-interface=cni0"
- "--verbose"
- "--auto-discover-default-role"
env:
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
ports:
- containerPort: 8181
protocol: TCP
securityContext:
privileged: true
hostNetwork: true
serviceAccountName: kube2ram
iptables
To prevent containers from directly accessing the ECS metadata API and gaining unwanted access to Alibaba Cloud resources, the traffic to 100.100.100.200
must be proxied for all user containers.
iptables \
--append PREROUTING \
--protocol tcp \
--destination 100.100.100.200 \
--dport 80 \
--in-interface docker0 \
--jump DNAT \
--table nat \
--to-destination `http://100.100.100.200/latest/meta-data/private-ipv4`:8181
This rule can be added automatically by setting --iptables=true
, setting the HOST_IP
environment variable, and running the container in a privileged security context.
Warning: It is possible that other pods are started on an instance before kube2ram has started. Using --iptables=true
(instead of applying the rule before starting the kubelet) could give those pods the opportunity to access the real ECS metadata API, assume the role of the ECS instance and thereby have all permissions the instance role has (including assuming potential other roles). Use with care if you don't trust the users of your kubernetes cluster or if you are running pods (that could be exploited) that have permissions to create other pods (e.g. controllers / operators).
Note that the interface --in-interface above or using the --host-interface cli flag may be different than docker0 depending on which virtual network you use e.g.
- flannel use
cni0
- Terway or Calico,use
cali+
(如:cali1234567890)
- kops (on kubenet), use
cbr0
- CNI, use
cni0
- weave use
weave
- kube-router use
kube-bridge
- OpenShift use
tun0
- Cilium use
lxc+
kubernetes annotation
Add an ram.aliyuncs.com/role
annotation to your pods with the role that you want to assume in this pod.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
annotations:
ram.aliyuncs.com/role: kube2ram-arn
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.9.1
ports:
- containerPort: 80
You can use --default-role
to set a fallback role to use when annotation is not set.
annotation in other kubernetes deploy model,e.g.CronJob
:
apiVersion: batch/v1
kind: CronJob
metadata:
name: my-cronjob
spec:
schedule: "00 11 * * 2"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 3600
jobTemplate:
spec:
template:
metadata:
annotations:
ram.aliyuncs.com/role: kube2ram-arn
spec:
restartPolicy: OnFailure
containers:
- name: job
image: my-image
Namespace Restrictions
By using the flag --namespace-restrictions you can enable a mode in which the roles that pods can assume is restricted by an annotation on the pod's namespace. This annotation should be in the form of a json array.
To allow the ACK pod specified above to run in the default namespace your namespace would look like the following.
apiVersion: v1
kind: Namespace
metadata:
annotations:
ram.aliyuncs.com/allowed-roles: |
["role-arn"]
name: default
Notice: to allow all roles prefixed with my-custom-path/ to be assumed by pods in the default namespace, the default namespace would be annotated as follows:
apiVersion: v1
kind: Namespace
metadata:
annotations:
ram.aliyuncs.com/allowed-roles: |
["my-custom-path/*"]
name: default
If you prefer regexp to glob-based matching you can specify --namespace-restriction-format=regexp, then you can use a regexp in your annotation:
apiVersion: v1
kind: Namespace
metadata:
annotations:
ram.aliyuncs.com/allowed-roles: |
["my-custom-path/.*"]
name: default
RBAC
This is the basic RBAC setup to get kube2ram working correctly when your cluster is using rbac. Below is the bare minimum to get kube2ram working.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube2ram
namespace: kube-system
---
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kube2ram
rules:
- apiGroups: [""]
resources: ["namespaces","pods"]
verbs: ["get","watch","list"]
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kube2ram
subjects:
- kind: ServiceAccount
name: kube2ram
namespace: kube-system
roleRef:
kind: ClusterRole
name: kube2ram
apiGroup: rbac.authorization.k8s.io
kind: List
Debug
By using the --debug flag you can enable some extra features making debugging easier:
/debug/store
endpoint enabled to dump knowledge of namespaces and role association.
Base ARN auto discovery
By using flag --auto-discover-base-arn
,kube2ram will auto discovery the base ARN via ECS metadata server.
Using KubernetesWoker instance role as default role
By using flag --auto-discover-default-role
,kube2ram will auto discovery the ARN of the RAM role attached to ECS and use it as the fallback role to use when annotation is not set.
Options
$ kube2ram --help
Usage of kube2ram:
--api-server string Endpoint for the api server
--api-token string Token to authenticate with the api server
--app-port string Kube2iam server http port (default "8181")
--auto-discover-base-arn Queries ECS Metadata to determine the base ARN
--auto-discover-default-role Queries ECS Metadata to determine the default RAM Role and base ARN, cannot be used with --default-role, overwrites any previous setting for --base-role-arn
--backoff-max-elapsed-time duration Max elapsed time for backoff when querying for role. (default 2s)
--backoff-max-interval duration Max interval for backoff when querying for role. (default 1s)
--base-role-arn string Base role ARN
--ram-role-session-ttl Length of session when assuming the roles (default 15m)
--debug Enable debug features
--default-role string Fallback role to use when annotation is not set
--host-interface string Host interface for proxying ECS metadata (default "docker0")
--host-ip string IP address of host
--ram-role-key string Pod annotation key used to retrieve the RAM role (default "ram.aliyuncs.com/role")
--iptables Add iptables rule (also requires --host-ip)
--log-format string Log format (text/json) (default "text")
--log-level string Log level (default "info")
--metadata-addr string Address for the ECS metadata (default "100.100.100.200")
--metrics-port string Metrics server http port (default: same as kube2ram server port) (default "8181")
--namespace-key string Namespace annotation key used to retrieve the RAM roles allowed (value in annotation should be json array) (default "ram.aliyuncs.com/allowed-roles")
--namespace-restriction-format string Namespace Restriction Format (glob/regexp) (default "glob")
--namespace-restrictions Enable namespace restrictions
--node string Name of the node where kube2ram is running
--verbose Verbose
--version Print the version and exits