Kubernetes Privilege Escalation (CVE-2018-1002105)

Introduction

Kubernetes is an open source production grade container orchestration system for deploying and managing docker/container applications. There are managed kubernetes orchestration service providers like Amazon Elastic Container Service for Kubernetes (EKS), Azure Kubernetes Service (AKS) etc.

kubectl

Kubernetes cluster users can perform management tasks using kubectl binary which talks to API Server. Example kubectl commands

# display pod resource
kubectl get pods -n my_namespace

# Execute a command in a container
kubectl -n my_namespace exec -it pods_name -- sh


# Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in the pod
kubectl -n my_namespace port-forward pod/mypod 5000 6000

# Get output from ruby-container from pod my-pod-pd

kubectl attach my-pod-pd -c ruby-container

kubectl execution flow (source: 1ambda.github.io)

kubelet

kubelet, kube-proxy run's on each compute node (VM, Worker, EC2 Instance etc), kubelet listens on TCP port 10250 and 10255 (with no authentication/authorization). API Server acts as Reverse Proxy to kubelet and API Aggregation. API Server connects to the kubelet to fulfill commands like exec, port=forward and opens a websocket connection which connects stdin, stdout, or stderr to user’s original call [01].

API Aggregation

Installing or writing additional API's into Kubernetes API Server i.e. extending core API Server

Vulnerability

Vulnerability is in Kubernetes API Server, crafted request can execute arbitrary commands on the backend servers (pods) through the same channel client established to backend through API Server [02]

Check nodes Kubernetes version

$ kubectl get nodes -o wide
NAME        STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
pd-worker-01 Ready node 13d v1.12.3 10.250.0.6 Container Linux by CoreOS 1745.7.0 (Rhyolite) 4.14.48-coreos-r2 docker://18.3.1
pd-worker-02 Ready node 13d v1.12.3 10.250.0.5 Container Linux by CoreOS 1745.7.0 (Rhyolite) 4.14.48-coreos-r2 docker://18.3.1
pd-worker-03 Ready node 13d v1.12.3 10.250.0.4 Container Linux by CoreOS 1745.7.0 (Rhyolite) 4.14.48-coreos-r2 docker://18.3.1

Vulnerable API Servers

If API server response looks as bellow and using vulnerable API versions of Kubernetes the you are vulnerable using anonymous-user escalation, patch Kubernetes immediately.
HTTP response error code 403 indicates Forbidden i.e. related to Authorization implies we successfully passed through Authentication phase.
{ "kind": "Status", "apiVersion": "v1", "metadata": { }, "status": "Failure", "message": "forbidden: User \"system:anonymous\" cannot get path \"/api/v1/\"", "reason": "Forbidden", "details": { }, "code": 403 }

anonymous user

By default, requests to the kubelet’s HTTPS endpoint that are not rejected by other configured authentication methods are treated as anonymous requests, and given a username of system:anonymous and a group of system:unauthenticated.

Mitigations

There are three levels of escalation mitigations

1. anonymous user -> aggregated API server

API Server admission-controller parameter anonymous-auth is set to fault

$ kubectl get po kube-apiserver-01 -n prod -o yaml | grep -i "anonymous-auth" - --anonymous-auth=false 
$ kubectl get po kube-apiserver-01 -n stage -o yaml | grep -i "anonymous-auth" - --anonymous-auth=false

2. authenticated user -> aggregated API server

Suspend aggregated API servers usage

3. authorized pod exec/attach/portforward -> kubelet API

Remove pod exec/attach/portforward permissions for users

References

[01]. https://docs.openshift.com/container-platform/3.11/architecture/networking/remote_commands.html
[02]. https://docs.openshift.com/container-platform/3.11/architecture/networking/remote_commands.html
[03]. https://elastisys.com/2018/12/04/kubernetes-critical-security-flaw-cve-2018-1002105/
[04]. https://github.com/kubernetes/kubernetes/issues/71411