Kubernetes Deep Dive: From Auto Scaling to Service Mesh with Istio
Week 13: Advanced Kubernetes Concepts
Introduction
Hello readers 👋! Sorry for the delay with last week's article. Week 13 of my DevOps learning journey has been quite eventful. For the past two weeks, I've been diving deep into different Kubernetes concepts, and this week I continued on that path. After completing the fundamentals of Kubernetes last week, I covered some more advanced topics including Auto Scaling, Role-Based Access Control, Custom Resource Definitions, Helm, and Service Mesh with Istio.
Auto Scaling (HPA and VPA)
When I was learning about creating pods using manifest files, I discovered that we could scale the number of pods using the scale command like this: kubectl scale deployment random-deployment -n random --replicas=3. At that time, I thought scaling pods in Kubernetes was really straightforward, but as I learned more about production environments where application demands fluctuate throughout the day, I realized that manually scaling pods up or down isn't practical.
One of Kubernetes' main features is its ability to automatically scale resources according to demand and workload. Kubernetes provides two complementary autoscaling mechanisms: HPA and VPA.
Horizontal Pod Autoscaling (HPA) works by having the HPA controller automatically scale the number of pod replicas in a Deployment, ReplicaSet, or StatefulSet based on observed metrics like CPU utilization, memory usage, or custom metrics. I learned how to create HPA manifest files and tested them in practice. Some common use cases for HPA that I discovered include:
Web applications with varying traffic patterns
API services experiencing fluctuating request loads
Processing workloads with queue-based scaling
Vertical Pod Autoscaling (VPA) automatically adjusts the CPU and memory requests/limits for containers in a pod based on historical and current resource usage patterns. VPA consists of three components:
Recommender: Analyzes resource usage and provides recommendations
Updater: Decides which pods need updates and triggers eviction
Admission Controller: Applies recommended resources to new/restarted pods
Some use cases for VPA include:
Batch processing jobs with unpredictable resource needs
Applications with evolving resource requirements over time
Cost optimization by eliminating resource over-provisioning
In summary, HPA increases the number of pods based on resource utilization, whereas VPA increases or decreases the resource limits of containers.
Role-Based Access Control (RBAC)
In an organization using Kubernetes, there can be many users who need to work on different aspects of the platform. Each user may have different roles and responsibilities. To manage all these users and service accounts so they don't accidentally or intentionally cause trouble with the cluster and deployments, Role-Based Access Control is used. Without proper access controls, any user could potentially delete critical workloads, access secrets, or modify cluster configurations.
RBAC consists of three core components:
Subjects: Who is requesting access (Users, Groups, ServiceAccounts)
Resources: What they want to access (Pods, Services, Secrets, etc.)
Verbs: What actions they want to perform (get, list, create, delete, etc.)
To create RBAC rules, we first need to create Roles or ClusterRoles. A Role defines permissions within a specific namespace, whereas a ClusterRole defines permissions across the entire cluster. After defining the rules for resources and what actions can be performed on those resources, RoleBinding and ClusterRoleBinding are created to define which subjects these rules will be applied to.
For example:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: apache-manager
namespace: apache
rules:
- apiGroups: ["*"]
resources: ["deployments", "pods", "services"]
verbs: ["get", "apply", "delete", "watch", "create", "patch"]
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: apache-manager-role-binding
namespace: apache
subjects:
- kind: User
name: apache-user
namespace: rbac.authorization.k8s.io
roleRef:
kind: Role
name: apache-manager
apiGroup: rbac.authorization.k8s.io
I also learned about important commands such as kubectl auth can-i get pods -n apache --as=<user> to check whether a user has permission to execute specified commands, which I found really useful while learning and testing. I also noted several best practices for RBAC policies, such as the principle of least privilege, namespace isolation, and service account strategy.
Custom Resource Definitions (CRDs)
Until now, I had been learning and experimenting with the built-in resources that Kubernetes provides, such as pods and services. But CRDs allow us to extend the Kubernetes API with our own custom resources. I can already imagine how useful CRDs can be as the complexity of modern applications grows rapidly. With CRDs, we can define application-specific resources that the Kubernetes API can manage natively. CRDs are essentially like custom data structures.
To create a CRD, we need to define the structure and validation rules of our custom resource. For example, I created a CRD for a simple database:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: databases.example.com
spec:
group: example.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
engine:
type: string
enum: ["mysql", "postgres"]
version:
type: string
storage:
type: string
required: ["engine", "version"]
status:
type: object
properties:
phase:
type: string
scope: Namespaced
names:
plural: databases
singular: database
kind: Database
After defining and applying this resource definition, we can use these resources in the same way as native Kubernetes resources and perform actions such as get, describe, etc.
apiVersion: example.com/v1
kind: Database
metadata:
name: my-app-db
namespace: production
spec:
engine: postgres
version: "1.4"
storage: "100Gi"
Helm
I already knew that Kubernetes had multiple interconnected components, but what I didn't realize was just how many components there could be. So many that it can really become a hassle to create and manage all the YAML files for these components. This problem is solved by Helm. Helm addresses the critical problem of managing complex application deployments with multiple interconnected components. Instead of manually creating dozens of YAML files and managing their dependencies, Helm packages everything into reusable, versioned "charts." This idea immediately struck me as a genius solution, especially after seeing how lengthy and confusing these YAML files can become during my learning process.
Helm Charts
A Helm chart is a collection of files that describe a related set of Kubernetes resources. I think of it like a recipe book - it contains all the instructions (templates) and ingredients (values) needed to create a complete meal (application deployment). I installed Helm on my EC2 instance using the Helm installation guide and observed the structure of a Helm chart.
Chart Structure:
mychart/
├── Chart.yaml # Chart metadata
├── values.yaml # Default configuration values
├── templates/ # Kubernetes manifests templates
│ ├── deployment.yaml
│ ├── service.yaml
│ └── ingress.yaml
└── charts/ # Chart dependencies
The real power lies in templating - instead of static YAML files, Helm uses Go templates that dynamically generate manifests based on provided values.
We can either create a Helm chart from scratch or install charts from repositories that contain multiple charts. Chart repositories function like Docker registries - centralized locations for sharing and distributing packaged applications. Through these charts, we can manage multiple components for our Kubernetes cluster. The process followed for a Helm chart is as follows:
1. CREATE CHART
↓
helm create myapp
↓
2. CUSTOMIZE TEMPLATES
↓
Edit templates/*.yaml files
Edit values.yaml
↓
3. VALIDATE & TEST
↓
helm lint ./myapp
helm template myapp ./myapp --debug
↓
4. PACKAGE CHART
↓
helm package ./myapp
↓
5. DEPLOY TO CLUSTER
↓
helm install myrelease ./myapp
A chart is a bundle that contains all the necessary resources for an application. Helm provides versioning and rollback capabilities, which are essential for maintaining application stability. If an update causes issues, teams can easily revert to a previous stable state using the helm rollback command. This feature is particularly valuable in CI/CD pipelines, where automated deployments and rollbacks are critical.
Service Mesh - Istio
After getting familiar with Helm and gaining hands-on experience with it, I learned about a new challenge in microservices architecture: managing communication between dozens or hundreds of microservices. In complex microservices architectures, cross-cutting concerns like security, observability, and traffic management become scattered across application code. The traditional solutions to this problem were either application-level (embedding networking logic into each service) or network-level (using traditional load balancers and firewalls). However, these approaches aren't suitable for the modern era. The solution to these challenges is Service Mesh.
A Service Mesh is a dedicated infrastructure layer for managing communication between services in a microservice-based system. It provides a centralized, dedicated infrastructure layer that handles the intricacies of service-to-service communication. Istio is a popular open-source service mesh tool.
Istio has two main core components:
Data Plane - Envoy Sidecars: Istio injects Envoy proxy sidecars alongside each application container. These proxies intercept all network traffic, providing:
Transparent service discovery
Load balancing and health checking
Security policy enforcement
Metrics collection and distributed tracing
Control Plane - Istiod: The unified control plane manages configuration and policy distribution:
Pilot: Service discovery and traffic management
Citadel: Certificate authority for mTLS
Galley: Configuration validation and distribution
At first, it seemed as complicated as it sounds, but by slowly understanding the infrastructure and working with Istio to deploy a sample application, I gained some confidence. I installed Istio and after deploying the sample application, I learned to visualize the service mesh and network using the Kiali dashboard.
Other Concepts
Apart from these major concepts, I also explored some other smaller but interesting and useful topics such as Node Affinity, Taints and Tolerations, Init containers, and Sidecar containers. I learned how these features contribute to the Kubernetes environment and did hands-on mini exercises for these services, like using Init containers for database readiness checks and sidecar containers for metric collection.
Resources I used-
Challenges I Faced
1️⃣ Service unavailable for K8s dashboard with Kind cluster
This was quite frustrating initially as I couldn't access the Kubernetes dashboard that I had set up. The dashboard service wasn't accessible through the usual methods, and I spent some time troubleshooting network connectivity issues.
Solution: I resolved this by using port forwarding with the command: kubectl -n kubernetes-dashboard port-forward service/kubernetes-dashboard 8443:443 --address=0.0.0.0. This allowed me to forward the dashboard service to my local machine and access it through the browser.
2️⃣ Couldn't connect to Istio Kiali Dashboard
After successfully installing Istio and deploying sample applications, I couldn't access the Kiali dashboard to visualize the service mesh. This was particularly challenging because the dashboard is crucial for understanding how services communicate within the mesh.
Solution: I used local port forwarding by opening an SSH tunnel that forwards local port 20001 to the EC2's port 20001. This allowed me to securely access the Kiali dashboard running on my EC2 instance from my local machine.
What's Next
After covering these many topics and concepts about Kubernetes, I'm planning to work on a comprehensive project that uses all these concepts and involves different third-party components and services such as Prometheus and Grafana. This should help me consolidate my learning and gain practical experience with real-world scenarios.
Let's Connect!
If you have any recommended resources, better approaches to my challenges, or insights, I'd love to hear them! Drop your thoughts in the comments.
Have a wonderful day!
