Kubernetes Security

Strong Kubernetes security is important to keep out attackers looking to use compromised containers for malicious activities yet another story about cloud assets that aren’t well-protected. RedLock found that hackers were mining bitcoin within at least one of the compromised containers, essentially turning the cloud instances into bots. Kubernetes security had little to do with the compromise itself; Kubernetes was simply hijacked and used to leverage container instances. The biggest problem was a lack of sound configuration management, coupled with minimal monitoring and alerting.

Aside from isolating and controlling access to Kubernetes consoles, there are a number of additional steps organizations can take to improve Kubernetes security.

Implement the strongest authentication to Kubernetes that you can, ideally using a separate directory service and an additional factor, like certificates. Kubernetes offers authentication options ranging from basic username/password flat files to more sophisticated federation schemes using OpenID Connect. More information on Kubernetes authentication can be found on the Kubernetes website.

Use only trusted images that meet best practices for configuration controls, and continually reassess the state of any running images using scanners and local tools. There are many commercial and open source tools available for assessing containers. In the compromised systems described by RedLock, admins failed to detect changes made to container instances when bitcoin mining tools were uploaded and executed in the environment. This demonstrates a lack of configuration control, as well as a lack of monitoring within containers.

Implement administrative boundaries in containers with namespaces. This enables admins to hide resources from different groups, and then apply different authorization controls to the namespaces to control and limit access as needed. Similarly, pods and containers should have a security context assigned to them as defined in the deployment configuration that restricts privileges as much as possible for operation.

Define resource quotas for CPU and memory that are attached to namespaces. By default, all Kubernetes containers have unlimited CPU and memory request capabilities, which can easily lead to abuse in a denial-of-service or bitcoin mining scenario.

Define network segmentation policies between Kubernetes pods, services and containers where appropriate.

Log everything in the environment. Kubernetes and other container technologies, like Docker, support robust logging at the node, cluster and administrative levels. Security, operations and development teams can all benefit from comprehensive logs describing activities and events that occur in the environment at all levels, and most of these logs are either natively Syslog-compatible or output in formats like JSON, which can be parsed easily and ingested into many log and event management systems.

In addition to all these Kubernetes security recommendations, cloud users should configure billing thresholds that match normal usage patterns, and alerts should be sent when usage exceeds the threshold that has been set. Even if a cloud asset is compromised, there’s no excuse for high intensity activities like bitcoin mining to occur for any length of time without someone being alerted. Most major cloud providers readily support many ways to monitor the environment as a whole.