Kubernetes Annotation Security Risks in AWS

Misconfiguring just one word in Kubernetes can expose your AWS environment to the internet, putting your data and applications at serious risk. Kubernetes and AWS are essential tools for managing scalable applications, yet their complexity can sometimes lead to critical misconfigurations. As DevOps and security professionals, we understand the importance of staying ahead of these challenges and ensuring our systems are both robust and secure, but with the introduction of LLMs (Large Language Models) the challenges have grown.

There’s no question that LLMs are redefining how we perform tasks across the organization. From marketing to software engineering, there’s not a single aspect that hasn’t been impacted by the proliferation of tools like ChatGPT and Gemini.

However, along with the obvious boons come certain risks that must be managed. Thanks to the powers of LLMs, what was previously not an issue may now become one.

One example of a misconfiguration that can arise is managing Security Groups for AWS Load Balancers using Kubernetes annotations, specifically Classic Load Balancers (CLB) and Network Load Balancers (NLB).

What we’ve found in our research is that Chat GPT may sometimes suggest one annotation:

service.beta.kubernetes.io/aws-load-balancer-extra-security-groups

and other times the other:

service.beta.kubernetes.io/aws-load-balancer-security-groups

The difference between the annotations that the LLM will suggest will depend on how you prompt the model, and the wrong choice can dramatically amplify your exposure. In this post, we’ll dive into this issue with Kubernetes annotations, understand what can go wrong, and figure out how to avoid this pitfall.

Potential Scenario: Load Balancer Security Misconfiguration

Consider a scenario where a developer wants to set up a simple test environment using an AWS Classic Load Balancer in Kubernetes with the help of ChatGPT. To quickly get started, the developer creates a Security Group that allows traffic from their home IP address , 203.0.113.1/32 in our example, with the following Cloud Formation configurations:


YAML
AWSTemplateFormatVersion: "2010-09-09"
Resources:
MySecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: "Allow HTTP from specific IP"
VpcId: "vpc-12345678"
SecurityGroupIngress:
- IpProtocol: "tcp"
FromPort: 80
ToPort: 80
CidrIp: "203.0.113.1/32"
SecurityGroupEgress:
- IpProtocol: "-1”
CidrIp: "0.0.0.0/0"


After creating the Security Group, it receives the following Security Group ID: « sg-0123456789abcdef0 ».

Then, the developer asks ChatGPT to create a load balancer that sets their Security Group. ChatGPT provides a YAML with the following configuration:

apiVersion: v1
kind: Service
metadata:
name: my-loadbalancer-service
annotations: service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: "sg-0123456789abcdef0"
spec:
selector:
app: my-app
ports:
- protocol: TCP
port: 80
targetPort: 80
type: LoadBalancer

Unaware of the implications, contrary to the intention, the developer unknowingly exposes the load balancer to the entire internet, creating a significant security risk.

So what happened?

When AWS EKS creates a load balancer (for example, through a Kubernetes Service of type LoadBalancer), it also creates a default Security Group that allows access from IP addresses anywhere in the world (0.0.0.0/0) on the port specified in the Service configuration. In the example above, this port is 80.

While the developer created a restricted Security Group to allow traffic only from a specific IP address, the issue arises with the word « extra » in the annotation. Instead of replacing the default Security Group, this annotation adds the developer’s specified Security Group on-top of the default one. That means that the Load Balancer allows traffic from everywhere (0.0.0.0/0), even though the developer meant to configure only the Security Group which allows traffic from the specific IP. In short, this one word can leave your load balancer open to the world.

It’s not a bug; it’s a feature.

Understanding the Differences Between Key Annotations

aws-load-balancer-security-groups vs. aws-load-balancer-extra-security-groups

When configuring AWS Classic/Network Load Balancers in Kubernetes, there are two kinds of annotations developers can use for managing Security Groups:

1. service.beta.kubernetes.io/aws-load-balancer-security-group
   • Purpose: Specifies the primary Security Groups for the load balancer and replaces the default Security Group.
   • When to Use: Use this annotation when you need to define a new set of Security Groups to replace the default Security Group created by AWS for your load balancer.
2. service.beta.kubernetes.io/aws-load-balancer-extra-security-groups
   • Purpose: Allows you to add additional Security Groups to the load balancer in addition to the default Security Group.
   • When to Use: Use this annotation when you want to retain the default Security Group and simply add more Security Groups for additional control and security policies.

3 Types of Annotations in Kubernetes

There are three types of annotations for AWS Load Balancers in Kubernetes that relates to our use case:

3 Types of Annotations in Kubernetes

Unlike the CLB and NLB that we discussed in the above scenario, using AWS Application Load Balancer (ALB) is straightforward since there’s only one annotation you can use:

alb.ingress.kubernetes.io/security-groups

• Purpose: Specifies the Security Groups for the ALB. When you use this annotation, the default Security Group created by AWS for the load balancer is replaced by the specified Security Groups. There isn’t a separate annotation for extra Security Groups; simply list multiple Security Groups within this annotation to configure your ALB’s Security Groups.
• When to Use: Use this annotation for ALB to define one or more Security Groups that will replace the default Security Group for enhanced security management.

Best Practices for Avoiding Security Groups Misconfigurations

There are numerous practices, some of which you might already be familiar with, to avoid this misconfiguration:

• Specify Primary Security Groups: It’s generally safer to use service.beta.kubernetes.io/aws-load-balancer-security-groups. But you should choose the one that best fits your use case.
• Review Security Groups Rules: Regularly audit and adjust AWS’s Security Group rule configurations to ensure they meet your security standards.
• Automate Audits: Use tools like AWS Config to automatically check for and alert on Security Group misconfigurations.
• Stay Informed: Continuously learn and stay updated with AWS and Kubernetes best practices to avoid common pitfalls.

The Risks of Not Securing Your Configurations

Understanding the distinction between Security Groups annotations is crucial for securing your AWS Load Balancers in Kubernetes. The risks of not configuring these annotations correctly can be critical and include:

1. Data Breaches: Overly permissive Security Groups or misconfigurations can expose sensitive data to unauthorized access, leading to data theft, privacy violations, and financial loss.
2. Service Disruption: Improper security configurations can leave your applications vulnerable to attacks, resulting in downtime and loss of availability.
3. Compliance Violations: Failing to secure your infrastructure properly may lead to non-compliance with industry regulations, potentially causing legal penalties and reputational damage.

Final Thoughts

It’s essential to validate your code—especially if you’re copying it from sources like Google or ChatGPT. Even seemingly small changes in Kubernetes annotations can expose your environment to unnecessary risks. It’s important to familiarize yourself with the different kinds of annotations, what each one is doing, and the implications of even small changes. The other key factor is understanding how LLMs are changing the game. LLMs are becoming increasingly popular across various aspects of work, and this Kubernetes annotation scenario is just one of many emerging with the widespread use of these models. While LLMs provide useful suggestions, may also suggest something that might not fit the situation, and could introduce an unnecessary security risk you didn’t count on.

Stay driven, keep learning, and stay caffeinated.