Crypto mining attacks in containerised environments are not new. Researchers have found different kinds of crypto mining activities running inside misconfigured containers. Microsoft, in April, had disclosed large-scale crypto-mining attacks against Kubernetes clusters which were discovered by Azure Security Center, thus helping users protect Kubernetes clusters from security threats.

We have seen how Microsoft is very proactive in monitoring security threats. For cloud security, Azure Security Center (ASC) monitors and protects thousands of Kubernetes clusters running on top of Azure Kubernetes Service (AKS). Azure Security Center routinely searches for and does research on new attack vectors against Kubernetes workloads.

Recently, a new campaign was seen lately by ASC, which targeted Kubeflow, a machine learning toolkit for Kubernetes. Kubeflow is an open-source project, which began as a project for running TensorFlow workloads on Kubernetes.

ASC witnessed the deployment of a malicious image from a public repository on several different clusters. By examining the image’s layers, it found that the image was running an XMRIG miner. 

Nodes which are utilised for ML tasks are usually relatively powerful, and in some cases include Graphical Processing Units (GPUs). This makes Kubernetes clusters which are used for ML tasks an ideal target for crypto mining campaigns, which was the target of the recent attack.

Using the processing power of the clusters, the hackers wanted to mine cryptocurrency for themselves. The cryptocurrency, in this case, was Monero (XMR), which is popular among hackers because it’s incredibly hard to track by law enforcement authorities. 

Why Kubeflow Machine Learning Framework Was Used

Kubeflow has expanded and become a well-known framework for performing machine learning tasks in Kubernetes. Kubeflow is a containerised service: the many tasks run as containers in the cluster. So, if attackers anyhow gain access to Kubeflow, they have many ways to run the malicious image in clusters.

Kubeflow framework consists of several different services. Some of those services cover frameworks for training models, Katib and Jupyter notebook server, and others. The framework is divided into various namespaces, which are a collection of Kubeflow services, and resources created within a namespace are isolated to that namespace.

Kubeflow lets users create a Jupyter notebook server and enable them to choose the image for the notebook server, including a choice to use a custom image. This image does not have to be a legitimate notebook image, and so attackers can run their own image using this feature.

Attackers can use this to deploy a malicious container from a real Jupyter notebook and use a new or existing notebook to deploy their code. The code, in this case, ran from the notebook server, which is a container with a mounted service account. 

Exposed Dashboard To Internet Led To The Attack

According to Microsoft, Kubeflow presents its UI functionality through a dashboard, which is exposed by Istio ingress gateway and is default-accessible just internally, and users should utilise port-forward to access the dashboard, which tunnels the traffic through the Kubernetes API server.

In some cases, users may change the setting of the Istio Service to Load-Balancer, which exposes the Service, (i.e istio-ingressgateway in the namespace istio-system) to the internet. Microsoft said that some users preferred to do it for convenience to directly access the dashboard without the need of tunnelling through the Kubernetes API server.

By opening the service to the Internet, users may access the dashboard directly. But, this operation also enables insecure access to the Kubeflow dashboard, which permits anyone to execute operations in Kubeflow, also including the deployment of new containers in Kubeflow cluster.

“If attackers have access to the dashboard, they have multiple methods to deploy a backdoor container in the cluster,” according to Yossi Weizman Security Research Software Engineer, Azure Security Center.

The cyber attacker utilised an exposed dashboard (Kubeflow dashboard in this case) for obtaining access to the cluster. The attacker headed to move laterally and deploy the container using the mounted service account. The attacker impacted the cluster by deploying a cryptocurrency miner. 

Kubernetes is fast rising to become the new standard for deploying and managing containerised software in the cloud. According to experts, only a few professionals have in-depth experience, and therefore security can be ignored. Consequently, Kubernetes environment needs to be configured properly to be secure, assuring nothing is exposed for cyber attackers. 

Microsoft has released instructions on how to check if your cluster is impacted and what security measures can be taken in future to prevent such attacks.

The post How Kubeflow Clusters Were Targeted Recently By Hackers For Crypto-Mining appeared first on Analytics India Magazine.