Took a long break from writing. Didn't plan to, just happened. But I'm back, and I figured what better way to return than tearing apart something I actually built. I wanted a real project to learn Spring Boot properly. Not a tutorial. Not a todo app. Something with enough moving parts that I'd actually have to figure things out — auth, background jobs, file storage, the whole thing. So I built Time Capsule: a web app where you can create sealed digital capsules with messages and photos, lock them, and have them automatically delivered on a future date you set.

The idea is simple. The implementation was not.

What It Actually Does

You create a capsule, add content to it — text, images, files — set an unlock date, and that's it. The capsule is sealed. Nobody (including you) can see the contents until that date hits. You can invite other people as contributors or viewers. When the unlock date arrives, everyone gets an email notification and the capsule opens.

The features list sounds clean. Getting there was... a process.

The Stack

Frontend: Next.js 14 (App Router), Tailwind CSS, Radix UI, Zustand for state, React Hook Form + Zod for validation.

Backend: Spring Boot 3.x, Java 21, PostgreSQL, Spring Security + JWT, Spring Data JPA, Flyway for migrations, Amazon S3 for file storage, Resend for email notifications, and Spring's @Scheduled for background jobs.

I picked this stack specifically because I was weak on the backend side. I knew React well enough. Spring Boot was the gap I wanted to close.

What Broke Me: JWT Auth

I'll be honest — I underestimated this completely.

The filter chain, the OncePerRequestFilter, plugging everything into SecurityFilterChain correctly — it took way longer than it should have. Here's the actual filter I ended up with:

@Component
@RequiredArgsConstructor
@Slf4j
public class JwtAuthFilter extends OncePerRequestFilter {
    private final JwtUtil jwtUtil;
    private final UserDetailsServiceImpl userDetailsService;

    @Override
    protected boolean shouldNotFilter(HttpServletRequest request) {
        String servletPath = request.getServletPath();
        return servletPath.startsWith("/api/v1/oauth2/") ||
                servletPath.startsWith("/api/v1/login/oauth2/") ||
                servletPath.startsWith("/api/v1/auth/");
    }

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
            throws ServletException, IOException {
        String authHeader = request.getHeader("Authorization");

        if (authHeader == null || !authHeader.startsWith("Bearer ")) {
            filterChain.doFilter(request, response);
            return;
        }

        String token = authHeader.substring(7);

        try {
            if (jwtUtil.isTokenValid(token)) {
                String email = jwtUtil.extractEmail(token);
                UserDetails principal = userDetailsService.loadUserByUsername(email);

                var authToken = new UsernamePasswordAuthenticationToken(
                        principal, null, principal.getAuthorities()
                );
                authToken.setDetails(new WebAuthenticationDetailsSource().buildDetails(request));
                SecurityContextHolder.getContext().setAuthentication(authToken);
            }
        } catch (JwtException e) {
            request.setAttribute("jwt_exception", e);
            SecurityContextHolder.clearContext();
            return;
        } catch (UsernameNotFoundException e) {
            request.setAttribute("username_not_found_exception", e);
            SecurityContextHolder.clearContext();
            return;
        }

        filterChain.doFilter(request, response);
    }
}

The filter itself wasn't the hard part. The surprise was that exceptions thrown here never reach your @ControllerAdvice / GlobalExceptionHandler — because this filter runs before the request ever enters the controller layer. I kept wondering why my exception handler was doing nothing.

The fix was a CustomAuthenticationEntryPoint registered directly in SecurityConfig:

.addFilterBefore(jwtAuthFilter, UsernamePasswordAuthenticationFilter.class)
.exceptionHandling(exception -> exception
        .authenticationEntryPoint(authenticationEntryPoint));

One line, but it took me an embarrassingly long time to figure out why it was needed. Spring Security's error handling is a different world from what you're used to coming from Express or Next.js API routes.

Role-based access was its own thing. Three roles: OWNER, CONTRIBUTOR, and VIEWER. Enforcing this at the service layer rather than just the controller made the code cleaner — but also meant more places for something to silently fail.

What Else Broke Me: S3

File uploads felt straightforward on paper. In practice, I had three separate issues:

CORS. When a capsule unlocks and the frontend tries to render images via presigned S3 URLs, the browser blocks the request if your S3 bucket CORS policy doesn't allow your origin. The error message is useless — it took me a while to trace it back to a missing CORS rule in the bucket config rather than anything in my own code.

File size limits. Spring Boot has a default multipart upload limit of 1MB. My app is supposed to support "high-quality images." I kept getting MaxUploadSizeExceededException until I added this to application.properties:

spring.servlet.multipart.max-file-size=10MB
spring.servlet.multipart.max-request-size=10MB

Presigned URLs for access control. Uploads go frontend → Spring Boot → S3 — the server handles all file storage. But the more interesting part is how files are served back. The S3 bucket stays private by default; nothing is publicly accessible. When a capsule unlocks, the backend generates a presigned URL for each file: a temporary, expiring S3 link valid for a limited window. That URL gets handed to the frontend, which renders the image. The file was sitting in S3 the whole time, completely unreachable until unlock. It's a clean way to enforce the sealed capsule mechanic at the storage layer, not just the app layer.

The Part I Actually Liked: Scheduled Jobs

The core mechanic of the whole app is that capsules unlock automatically. No cron job on the server, no manual trigger. Spring's @Scheduled annotation makes this almost embarrassingly easy.

@Transactional
@Scheduled(fixedDelay = 60000)      // runs every 60 secs
    public void unlockDueCapsules(){
        List<Capsule> dueCapsules = capsuleRepository
                .findAllDueCapsulesWithDetails(Instant.now());


        for(Capsule capsule : dueCapsules){
            capsule.setStatus(CapsuleStatus.UNLOCKED);
            capsuleRepository.save(capsule);

            log.info("Capsule: {}, unlocked from scheduler at: {}", capsule.getSlug(), Instant.now());

            // Send emails to capsule members
            resendEmailService.sendUnlockNotification(capsule);
        }
    }

Every 60 seconds, it checks for capsules whose unlock date has passed, flips their status, and fires off email notifications. That's it. I used Resend for the emails — clean API, took maybe 30 minutes to integrate.

This was the part of the backend I felt most confident building, which says something about how the auth and S3 sections went.

Architecture

Here's the full system diagram:

Two layers — client and server — with clear separation between them.

Client layer: The user hits the Next.js frontend over HTTPS. All API calls go to Spring Boot with a JWT attached in the Authorization header. That's it for the frontend's job.

Server layer: Spring Boot is the hub. It handles three distinct flows:

1. Data flow — REST requests from the frontend hit the API, which reads/writes capsule data to PostgreSQL via JPA. Standard CRUD, nothing unusual here.

2. File flow — When a user attaches a file to a capsule, it routes frontend → Spring Boot → S3. The bucket is private by default. When a capsule unlocks, the backend generates presigned URLs — temporary, expiring S3 links — so the frontend can render the files without ever making them publicly accessible.

3. Scheduler flow — The Task Scheduler (Spring's @Scheduled) runs in the background independently of any HTTP request. When a capsule's unlock date passes, it updates the capsule state in PostgreSQL and triggers the SMTP mail server to send email notifications out to all participants.

That last flow — the async, time-driven one — is what makes the app actually work. Everything else is just a web app. The scheduler is what gives it its whole point.

What I'd Do Differently

The app works. It's deployed at timecapsule.akshanshsingh.com. But if I started over:

I'd plan the data model first. I made a few schema decisions early that I had to undo later. Flyway migrations saved me here, but the back-and-forth was avoidable.

I'd handle errors more consistently on the frontend. Some API errors surface nicely. Others fail silently. Users deserve better than that.

I'd containerize earlier. Docker is on the roadmap but I kept pushing it off. Starting with a docker-compose.yml would've made the local dev setup way less painful to explain to anyone who wanted to run it.

I'd generate presigned URLs more granularly. Right now they're generated at unlock time for all files in a capsule. In hindsight I'd scope them tighter — shorter expiry, per-request generation — so there's no window where a URL stays valid longer than it should.

What's Next

• Full Dockerization for one-click deployment

• In-app audio player for sound-based memories

• Search and filter for public vaults

The core feature set is solid. The polish is what's left.

This was the most complete full-stack project I've shipped so far. Spring Boot clicked for me around week three. Java stopped feeling verbose and started feeling structured. The backend patterns — services, repositories, controllers — make a lot of sense once you build something non-trivial with them.

If you're coming from a JavaScript background and considering Spring Boot: the learning curve is real, but it's worth it. Just expect JWT to humble you first.

What part of a full-stack project do you find hardest to get right — auth, storage, or something else entirely?

Hello readers👋! Past 3 weeks have been hectic to be honest and I wasn't able to get a lot done but still managed to take some time to learn and create something. I continued where I left off from my previous Terraform article and dove deep into two crucial concepts that every Terraform practitioner needs to master - State Management and Workspaces. And to practice and get some hands-on experience of using Terraform, I've built a mini project and a real-life industry standard project to enhance my understanding and workflow of Terraform.

After understanding the basics of Terraform in my previous article, I realized that knowing how to write .tf files is just the beginning. The real challenge comes when you need to manage infrastructure across different environments, collaborate with teams, and ensure that your infrastructure state is consistent and secure.

State Management and Workspaces

Understanding Terraform State

Before diving into advanced state management, I had to understand what exactly the state is in Terraform. The state is basically Terraform's way of keeping track of the real-world resources it manages. When you run terraform apply, Terraform doesn't just create resources - it also records information about those resources in a state file (terraform.tfstate).

This state file contains:

• Resource metadata - IDs, current configuration, and dependencies

• Resource mappings - How your .tf configuration maps to real-world resources

• Performance optimization - Instead of querying all resources every time, Terraform uses state for faster operations

The problem I learned about is that by default, Terraform stores state locally, which creates several challenges:

• Collaboration issues - Multiple team members can't work on the same infrastructure

• State loss - If your local machine crashes, you lose track of your infrastructure

• Security risks - State files contain sensitive information and shouldn't be stored in version control

Remote State Management

To solve these problems, I learned about remote state backends. A backend in Terraform determines where and how state is stored and accessed. The most commonly used backend for AWS environments is S3 with DynamoDB.

Here's how I configured remote state in my projects:

terraform {
  backend "s3" {
    bucket         = "terraform-state-bucket-akshansh029"
    key            = "dev/terraform.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "terraform-state-lock"
    encrypt        = true
  }
}

The S3 bucket stores the actual state file, while DynamoDB provides state locking to prevent multiple people from running Terraform simultaneously on the same infrastructure. The encrypt = true ensures that the state file is encrypted at rest.

Terraform Workspaces

After understanding state management, I learned about Workspaces - a feature that allows you to manage multiple environments (dev, staging, prod) using the same Terraform configuration.

Think of workspaces as separate "instances" of your infrastructure. Each workspace has its own state file, so you can have identical infrastructure setups for different environments without them interfering with each other.

Key workspace commands I learned:

• terraform workspace list - Shows all available workspaces

• terraform workspace new <name> - Creates a new workspace

• terraform workspace select <name> - Switches to a specific workspace

• terraform workspace show - Shows current workspace

What's really cool about workspaces is that you can use the terraform.workspace variable in your configurations to make environment-specific decisions:

resource "aws_instance" "example" {
  instance_type = terraform.workspace == "prod" ? "t3.medium" : "t2.micro"

  tags = {
    Name = "${terraform.workspace}-server"
    Environment = terraform.workspace
  }
}

Mini Project: Dev and Prod Infrastructure

To practice these concepts, I created a mini project that demonstrates how to use workspaces to manage identical infrastructure across different environments.

Project Structure

I organized my project with the following structure:

DevOps-Learning/terraform-modules-app/
├── main.tf
├── provider.tf
├── terraform.tf
├── app
        ├── dynamo.tf
        ├── ec2.tf
        ├── s3.tf
        ├── variables.tf

Environment-Specific Configuration

The importance of this project was seeing how the same Terraform configuration could create different infrastructure based on the workspace. Here's how I implemented environment-specific logic:

# Different instance types for different environments
resource "aws_instance" "web_server" {
  ami           = var.ami_id
  instance_type = terraform.workspace == "prod" ? var.prod_instance_type : var.dev_instance_type
  key_name      = aws_key_pair.deployer.key_name

  # Different storage sizes
  root_block_device {
    volume_size = terraform.workspace == "prod" ? 20 : 8
    volume_type = "gp3"
  }

  tags = {
    Name = "${terraform.workspace}-web-server"
    Environment = terraform.workspace
  }
}

Workflow Implementation

The workflow I followed was:

1. Create Dev Environment:

terraform workspace new dev
terraform plan
terraform apply

1. Create Prod Environment:

terraform workspace new prod
terraform plan
terraform apply

Each environment got its own EC2 instance with appropriate sizing, security groups, but they were completely isolated from each other thanks to workspaces.

2-Tier-AWS-Infrastructure-Terraform Project

After getting comfortable with workspaces, I decided to tackle a more comprehensive project - a 2-tier AWS infrastructure that follows industry standards and best practices.

Project Architecture

This project implements a typical web application architecture with:

Presentation Tier (Web Layer):

• Application Load Balancer for traffic distribution

• Auto Scaling Group with EC2 instances in multiple Availability Zones

• Launch Template with user data for automatic application setup

Data Tier (Database Layer):

• RDS MySQL database with Multi-AZ deployment

• Private subnets for database security

• Database subnet group for proper placement

Infrastructure Deep Dive

The 2-tier architecture I implemented consists of two main layers that work together to create a scalable and secure web application infrastructure:

Web Tier (Presentation Layer): This is the front-facing layer that handles all user requests. I created an Application Load Balancer that distributes incoming traffic across multiple EC2 instances running in different Availability Zones. The EC2 instances are part of an Auto Scaling Group, which means they can automatically scale up or down based on traffic demand. Each instance runs a web server (I used Apache with a simple HTML page) that serves the application content to users.

Database Tier (Data Layer): The second tier consists of an RDS MySQL database that stores all the application data. What's really important here is that the database is placed in private subnets, meaning it's not directly accessible from the internet. Only the web servers can communicate with the database through internal network routing.

Implementation Process and Workflow

Phase 1: Network Foundation I started by building the network infrastructure - creating a custom VPC with both public and private subnets across multiple Availability Zones. The public subnets host the web servers and load balancer, while the private subnets contain the database. An Internet Gateway provides internet access to the public subnets, and a NAT Gateway allows the private subnets to download updates and patches while remaining secure.

Phase 2: Compute Layer Setup Next, I implemented the compute resources using a Launch Template that defines the EC2 instance configuration. The Launch Template includes the AMI, instance type, security groups, and a user data script that automatically installs and configures Apache web server when instances launch. The Auto Scaling Group uses this template to maintain the desired number of healthy instances and can automatically replace failed instances.

Phase 3: Database Implementation For the database layer, I created an RDS MySQL instance with Multi-AZ deployment for high availability. The database is configured with automated backups, encryption at rest, and is placed in a database subnet group that spans multiple Availability Zones. I also implemented parameter groups to optimize database performance.

Phase 4: Load Balancing and Traffic Management The Application Load Balancer sits in the public subnets and routes traffic to healthy web server instances based on configured health checks. I set up target groups that define which instances should receive traffic and configured the load balancer to distribute requests evenly across all available instances.

Security Architecture

Security was implemented at multiple layers throughout the infrastructure:

Network Security: I created separate security groups for each tier - the load balancer security group allows HTTP/HTTPS traffic from the internet, the web server security group only accepts traffic from the load balancer, and the database security group only allows MySQL connections from the web servers. This creates a secure communication path where each layer only accepts traffic from the layer above it.

Database Security: The RDS instance is completely isolated in private subnets with no direct internet access. Database credentials are managed securely, and I enabled encryption both at rest and in transit. The database is also configured with automated backups and point-in-time recovery.

Access Control: I used IAM roles instead of hardcoded credentials, and implemented the principle of least privilege throughout the infrastructure. The EC2 instances have just enough permissions to function but can't access other AWS services unnecessarily.

Project Flow

The beauty of this infrastructure is how all the components work together automatically:

1. Traffic Flow: When users access the application, their requests hit the Application Load Balancer, which checks the health of all web server instances and routes the request to a healthy server.

2. Auto Scaling: If traffic increases, the Auto Scaling Group automatically launches new EC2 instances using the Launch Template. These new instances automatically register with the load balancer target group and start receiving traffic.

3. Database Connectivity: The web servers connect to the RDS database using the internal DNS endpoint, ensuring all data operations happen securely within the private network.

4. Failure Recovery: If an EC2 instance fails, the Auto Scaling Group automatically terminates it and launches a replacement. If the primary database fails, RDS automatically fails over to the standby instance in another Availability Zone.

The best thing was seeing this entire infrastructure come to life with just a few Terraform commands. Running terraform apply creates dozens of resources in the correct order, with all the dependencies handled automatically by Terraform's dependency graph.

GitHub repo of this project- https://github.com/Akshansh029/2-Tier-AWS-Infrastructure-Terraform

Challenges I Faced

1️⃣ RDS Cluster Engine Version Problem

When I initially tried to create an RDS cluster, I encountered an error related to the engine version. I had specified engine_version = "8.0" but AWS was expecting a more specific version like 8.0.35.

Solution: I learned to check available engine versions using the AWS CLI. Then I updated my configuration to use a specific version that was available in my region.

2️⃣ State Lock Already Engaged

This was frustrating! I was working on the project and my terraform apply command got interrupted. When I tried to run it again, I got a "state lock" error saying the state was already locked by a previous operation.

Solution: I had to use the force-unlock command:

terraform force-unlock <LOCK_ID>

3️⃣ Unable to Select Which Fields Are Required and Which Were Not in AWS Resource Creation

While creating AWS resources, I was confused about which arguments were required and which were optional. The Terraform documentation sometimes wasn't clear, and I kept getting errors about missing required arguments.

Solution: I had to rely more heavily on the official Terraform AWS provider documentation, search for better examples and use terraform plan extensively to catch missing required arguments before applying.

Resources I Used

1. Terraform Official Documentation

2. DevOps Projects | NotHarshhaa

What's Next

After understanding the creation of cloud infrastructure using Terraform, my next plan is to learn configuration management of the infrastructure using Ansible. I want to understand how Terraform and Ansible work together’

Let's Connect!

🔗 My LinkedIn

🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I'd love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Infrastructure as Code

Before getting started with Terraform, one has to understand Infrastructure as Code, and I also started with the same question - What is IaC? Infrastructure as Code can be broken down into two terms - Infrastructure and Code.

Infrastructure refers to the underlying resources required to support applications and services, such as servers, networking components, operating systems, storage, databases, and security configurations. These resources are hidden from the user's eyes, but they are as important as the code of the application. Without these resources, an application cannot be deployed, maintained, and served to the audience.

Traditionally, physical/on-premises servers were used to deploy and maintain any application, with system administrators continuously monitoring everything. When the application's demand increased, the scaling process was done manually, and we all can imagine how painful and time-consuming that sounds.

But then the cloud revolution began. The evolution of cloud resources and providers such as AWS, GCP, Azure, Oracle, etc., came up with a solution to the traditional way of maintaining infrastructure for applications. The solution was to shift the focus from managing physical hardware to leveraging on-demand, scalable, and managed services over the internet. So now, instead of purchasing and configuring physical infrastructure, businesses can provision resources through cloud providers on a pay-per-use basis, significantly reducing upfront capital expenditures and operational costs.

However, with time, managing infrastructure through these cloud providers was becoming a pain too. Scaling up and down, replicating and tearing down infrastructure according to demand was becoming too time-consuming and was not fully automated.

Terraform Origin

AWS introduced their own IaC tool called AWS CloudFormation, which enables the automation and consistent provisioning of AWS resources through Infrastructure as Code. But this was only limited to AWS, forcing users and teams to use multiple tools for multi-cloud environments. This problem was highlighted by the co-founder of HashiCorp in 2011. Later in 2014, HashiCorp announced Terraform - an open-source Infrastructure as Code tool that could provide consistent workflows regardless of the cloud provider.

Terraform uses HCL (HashiCorp Configuration Language) to declare and provision resources across cloud providers. Terraform solves the complexity and fragility of managing infrastructure by turning it into code — making creation, change, and teardown of cloud (and some on-premises) resources repeatable, auditable, and automatable.

Some benefits of Terraform I learned:

• Declarative: You describe what you want, not how to build it.

• Idempotent & reproducible: Re-running the same configuration brings your infrastructure to the same desired state.

• Versionable: Configuration files live in Git like application code (review, branch, rollback).

• Multi-cloud & provider ecosystem: Single tool to manage many providers.

• Plan/Apply safety: Previews changes before they happen.

• Modularity & reuse: Modules let you package infrastructure patterns.

Terraform Basics

After thoroughly understanding the concept of IaC and its necessity, I focused on learning the core concepts of Terraform. I started by installing Terraform using Chocolatey on my laptop.

The first concept was Provider - it is a plugin that knows how to create resources on a platform (e.g., AWS, Google). It allows users to interact with different APIs and services. Resource is a cloud object or service of a provider that the user manages, such as EC2 instances, S3 buckets, etc. These are provisioned with the help of Terraform, which communicates with the provider to create, configure, or delete the resources.

Then I learned HCL (HashiCorp Configuration Language), which is used to write Terraform files (.tf file extension). It is a structured configuration language created by HashiCorp. An HCL file contains blocks, arguments, and expressions.

block_type "label1" "label2" {
  argument = value
}

Arguments are key-value pairs inside blocks where the configurations are written about the resource, and expressions are the values assigned to the arguments, which can be variables, references, or functions. I also learned about variables, comments, and other basic syntax.

Then it was time to learn about the core and important commands of Terraform:

• terraform init - Initializes a Terraform working directory and downloads provider plugins and modules.

• terraform validate - Validates syntax and checks if the configuration is structurally correct.

• terraform fmt - Formats .tf files into a canonical style and ensures consistent formatting.

• terraform plan - Shows the execution plan of changes and compares current state vs desired state.

• terraform apply - Executes the changes shown in the plan.

• terraform destroy - Removes all managed infrastructure.

These are the commands that are used most frequently in the Terraform workflow. Upon completing these basic but essential concepts, I decided to try creating some local and AWS resources using Terraform.

Terraform Implementation

At first, I created a simple text file in the same repository with the help of the "local_file" resource like this:

resource "local_file" "my_file" {
  filename = "example.txt"
  content  = "First file created with Terraform"
}

After this, I tried to create an S3 bucket with a simple HCL file like this:

resource "aws_s3_bucket" "terraform_bucket" {
  bucket = "terraform-bucket-akshansh029"
}

After being able to create these simple resources with the help of Terraform's official documentation, I continued to learn advanced concepts like loops (count, for_each) and conditional expressions. Loops help users create multiple resources from a single block instead of duplicating them. The count argument creates N copies of the specified resource, whereas the for_each argument creates resources from a map or set of values. Along with these, I learned about interpolation arguments like user_data.

With these concepts, I then created multiple EC2 instances along with key-pairs, security groups, VPC, and installed Nginx on them by using a custom script in user_data like this:

# Key pair for EC2 instance
resource "aws_key_pair" "deployer" {
  key_name   = "terraform-ec2-key"
  public_key = file("terraform-ec2-key.pub")
}

# VPC for EC2 instance
resource "aws_default_vpc" "default" {}

# Security group for EC2 instance
resource "aws_security_group" "allow_tls" {
  name        = "terraform-ec2-sg"
  description = "Allow TLS inbound traffic and all outbound traffic"
  vpc_id      = aws_default_vpc.default.id # Interpolation

  # inbound rules
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "SSH from anywhere"
  }
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "Access to HTTP from anywhere"
  }

  # outbound rules
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "allow_tls"
  }
}

# EC2 Instance
resource "aws_instance" "terraform_ec2" {
  # count = 2 # meta argument
  for_each = tomap({
    terraform-ec2-instance-t2 = "t2.micro"
    terraform-ec2-instance-t3 = "t3.micro"
  })
  ami             = var.ubuntu_ec2_ami_id # Ubuntu 24
  instance_type   = each.value
  key_name        = aws_key_pair.deployer.key_name
  security_groups = [aws_security_group.allow_tls.name]
  root_block_device {
    volume_size = var.env == "prd" ? var.aws_root_storage_size : var.aws_default_root_storage_size
    volume_type = "gp3"
  }
  user_data = file("install_nginx.sh") # Run install_nginx script

  tags = {
    Name = each.key
  }
}

Challenges I Faced

1. Didn't know which files not to push to GitHub

While maintaining the GitHub repository for my Terraform learning, I committed and pushed terraform.tfstate to GitHub. Later, I learned that state files should never be pushed to GitHub because they contain sensitive information about your infrastructure and can lead to security vulnerabilities. The state file also creates conflicts when multiple team members are working on the same infrastructure. I immediately removed it from the repository and added it to .gitignore.

2. Connection error during terraform init

While trying to initialize Terraform using terraform init, I encountered this error:

│ Error: Failed to install provider │ │ Error while installing hashicorp/aws v6.11.0: │ releases.hashicorp.com: read tcp │ 172.20.146.187:54611->108.159.15.48:443: wsarecv: A │ connection attempt failed because the connected party │ did not properly respond after a period of time, or │ established connection failed because connected host │ has failed to respond.

This was a network connectivity issue where my college WiFi was blocking or limiting connections to HashiCorp's servers. The college network likely had firewall restrictions or proxy settings that prevented Terraform from downloading the required provider plugins. I solved this by switching to my personal mobile hotspot.

Resources I Used

1. Terraform Official Documentation

2. TWS | Terraform Course

3. Kubesimplify | Terraform Crash Course

What’s Next

I will continue to learn the advanced topics of Terraform and then to practice these, I will try to create an infrastructure using Terraform as a project.

Let's Connect!

🔗 My LinkedIn

🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I'd love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Hello there 👋! Sorry for the delay with last week's article - exams kept me busy. The past couple of weeks of my DevOps learning journey have been incredibly rewarding. After completing my deep dive into Kubernetes concepts in the previous weeks, I decided it was time to put all that theoretical knowledge to the test. This week, I focused on building two comprehensive Kubernetes projects that would help me understand how all these concepts work together in real-world scenarios.

Project 1: Full-Stack Chat Application on MiniKube

The first project I tackled was deploying a three-tier chat application consisting of a React.js frontend, Node.js backend, and MongoDB database. What I found interesting about this project was how it brought together so many Kubernetes concepts I had learned - deployments, services, persistent volumes, secrets, and ingress controllers.

Architecture and Components

The application architecture was straightforward but comprehensive:

• Frontend: React.js application for the user interface

• Backend: Node.js API server handling chat logic

• Database: MongoDB for storing chat messages and user data

  

Key Kubernetes Concepts Applied

Persistent Storage: One of the most important aspects I learned was implementing persistent storage for MongoDB. I created a Persistent Volume and Persistent Volume Claim to ensure that chat data wouldn't be lost when pods were restarted. For MiniKube, I used hostPath which stores data directly on the host machine.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongodb-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/mongodb

Secrets Management: I learned how critical it is to properly handle sensitive information. The application required a JWT secret key and a MongoDB connection string, which I stored as Kubernetes secrets with Base64 encoding. This was my first real experience with securely managing credentials in Kubernetes.

Service Discovery: While learning Kubernetes, what impressed me while learning Kubernetes was seeing how services could connect to each other simply using service names. When building this project, seeing the backend connect with MongoDB using just mongodb:27017 demonstrated the automatic service discovery within the cluster perfectly.

Ingress Controller: Instead of using port-forwarding for access, I configured an ingress controller to route traffic based on hostnames. I had to add chats.aks.com to my local hosts file to access the application through a custom domain.

Docker Hub Integration

Before deploying to Kubernetes, I had to build and push Docker images for both frontend and backend to Docker Hub. This step taught me about the importance of having images accessible to the cluster.

Project 2: Voting Application with GitOps using ArgoCD

The second project was significantly more complex and introduced me to the world of GitOps. I deployed a microservices voting application using ArgoCD on a multi-node Kubernetes cluster running on AWS EC2.

Infrastructure Setup

I started by setting up a Kind (Kubernetes in Docker) cluster on an AWS EC2 instance. This is how I created a multi-node cluster configuration:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.30.0
- role: worker
  image: kindest/node:v1.30.0
- role: worker
  image: kindest/node:v1.30.0

This gave me hands-on experience with managing a cluster that closely resembles a production environment with separate control plane and worker nodes.

ArgoCD and GitOps Implementation

ArgoCD Installation: Installing ArgoCD was my first deep dive into GitOps tooling. I learned how to create dedicated namespaces, patch services to expose them externally, and manage RBAC for secure access.

GitOps Workflow: The most fascinating part was experiencing true GitOps in action. I configured ArgoCD to:

• Automatically sync applications from my GitHub repository

• Monitor changes in the k8s-specifications folder

• Apply updates to the cluster whenever I pushed changes to GitHub

Real-time Synchronization: I tested the GitOps functionality by changing replica counts directly in GitHub. Within minutes, ArgoCD detected the changes and automatically updated the cluster. Seeing pods scale up and down based on Git commits was incredibly satisfying and showed me the power of GitOps for production deployments.

Microservices Architecture

The voting application consisted of multiple microservices:

• Vote Service (Python) - Frontend for casting votes

• Result Service (Node.js) - Dashboard showing vote results

• Worker Service (.NET) - Processes votes from Redis to PostgreSQL

• Redis - Message queue for vote processing

• PostgreSQL - Database for storing vote results

This project taught me how different technologies can work together seamlessly in a Kubernetes environment, with each service handling its specific responsibility.

Monitoring with Prometheus and Grafana

I also implemented a Prometheus-Grafana monitoring stack using Helm charts to monitor and visualize the voting application. This was my first hands-on experience with Helm charts in a real project, and it perfectly demonstrated why Helm is considered the package manager for Kubernetes.

I used the official Prometheus community Helm chart which automatically deployed Prometheus server, Alertmanager, and Grafana with pre-configured dashboards. Setting up the entire monitoring stack with a single helm install command was incredibly efficient compared to manually creating dozens of YAML files.

The Grafana dashboards provided real-time insights into cluster metrics, pod resource usage, and application performance. I could visualize CPU and memory consumption across all microservices, monitor the health of PostgreSQL and Redis, and track request rates to the vote and result services. This monitoring setup gave me a clear understanding of how the voting application was performing under load and helped me identify potential bottlenecks.

What I Learned from These Projects

Real-world Application Architecture: Both projects showed me how theoretical Kubernetes concepts translate into practical applications. Understanding how frontend services communicate with backend APIs, and how those APIs connect to databases, gave me a complete picture of modern application deployment.

Security Best Practices: Working with secrets, RBAC, and service accounts taught me the importance of security in Kubernetes. I learned that proper access control and secret management aren't optional - they're fundamental requirements.

GitOps Philosophy: The ArgoCD project introduced me to the GitOps approach, where Git becomes the single source of truth for infrastructure and applications. This paradigm shift from push-based to pull-based deployments feels much more reliable and auditable.

Networking Understanding: Configuring services, ingress controllers, and port forwarding gave me practical experience with Kubernetes networking. I now understand how traffic flows within a cluster and how to expose applications externally.

Persistent Storage: Managing stateful applications like databases taught me about persistent volumes, storage classes, and the importance of data persistence in containerized environments.

Challenges I Faced

1️⃣ MiniKube Ingress Controller Issues

Initially, the ingress controller wasn't routing traffic properly to my chat application. The frontend was accessible, but API calls to the backend were failing.

Solution: I discovered that I needed to enable the MiniKube ingress addon with minikube addons enable ingress and ensure that the ingress resource was properly configured with the correct service names and ports.

2️⃣ ArgoCD UI Access Problems

After installing ArgoCD, I couldn't access the web interface even though the pods were running. The service was only accessible within the cluster.

Solution: I had to patch the ArgoCD server service to change its type from ClusterIP to NodePort, then use port forwarding with the --address 0.0.0.0 flag to make it accessible from my browser. I also had to add the appropriate inbound rules to the EC2 security group.

3️⃣ Docker Permission Issues on EC2

When setting up Kind on the EC2 instance, I kept getting "permission denied" errors when trying to run Docker commands.

Solution: I needed to add the ubuntu user to the Docker group with sudo usermod -aG docker ubuntu and then refresh the group membership with newgrp docker. This allowed me to run Docker commands without sudo.

4️⃣ JWT Token Authentication Issues in Chat Application

Even after properly setting up the JWT secret in Kubernetes secrets, the frontend was continuously logging users out of the chat application. Users could register and login initially, but within seconds they would be automatically logged out, making the application unusable.

Solution: I discovered that the issue was with how the JWT secret was being referenced in the backend deployment. The environment variable name in the deployment YAML had to exactly match what the backend was expecting. I also learned that the base64 encoded secret in Kubernetes gets automatically decoded when injected as an environment variable, so the backend was receiving the correct plaintext value.

Resources I Used

1. TrainWithShubham | K8s ArgoCD Project

2. Kubernetes Official Documentation

3. ArgoCD Documentation

Let's Connect!

🔗 My LinkedIn

🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I'd love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Hello readers 👋! Sorry for the delay with last week's article. Week 13 of my DevOps learning journey has been quite eventful. For the past two weeks, I've been diving deep into different Kubernetes concepts, and this week I continued on that path. After completing the fundamentals of Kubernetes last week, I covered some more advanced topics including Auto Scaling, Role-Based Access Control, Custom Resource Definitions, Helm, and Service Mesh with Istio.

Auto Scaling (HPA and VPA)

When I was learning about creating pods using manifest files, I discovered that we could scale the number of pods using the scale command like this: kubectl scale deployment random-deployment -n random --replicas=3. At that time, I thought scaling pods in Kubernetes was really straightforward, but as I learned more about production environments where application demands fluctuate throughout the day, I realized that manually scaling pods up or down isn't practical.

One of Kubernetes' main features is its ability to automatically scale resources according to demand and workload. Kubernetes provides two complementary autoscaling mechanisms: HPA and VPA.

Horizontal Pod Autoscaling (HPA) works by having the HPA controller automatically scale the number of pod replicas in a Deployment, ReplicaSet, or StatefulSet based on observed metrics like CPU utilization, memory usage, or custom metrics. I learned how to create HPA manifest files and tested them in practice. Some common use cases for HPA that I discovered include:

• Web applications with varying traffic patterns

• API services experiencing fluctuating request loads

• Processing workloads with queue-based scaling

Vertical Pod Autoscaling (VPA) automatically adjusts the CPU and memory requests/limits for containers in a pod based on historical and current resource usage patterns. VPA consists of three components:

1. Recommender: Analyzes resource usage and provides recommendations

2. Updater: Decides which pods need updates and triggers eviction

3. Admission Controller: Applies recommended resources to new/restarted pods

Some use cases for VPA include:

• Batch processing jobs with unpredictable resource needs

• Applications with evolving resource requirements over time

• Cost optimization by eliminating resource over-provisioning

In summary, HPA increases the number of pods based on resource utilization, whereas VPA increases or decreases the resource limits of containers.

Role-Based Access Control (RBAC)

In an organization using Kubernetes, there can be many users who need to work on different aspects of the platform. Each user may have different roles and responsibilities. To manage all these users and service accounts so they don't accidentally or intentionally cause trouble with the cluster and deployments, Role-Based Access Control is used. Without proper access controls, any user could potentially delete critical workloads, access secrets, or modify cluster configurations.

RBAC consists of three core components:

• Subjects: Who is requesting access (Users, Groups, ServiceAccounts)

• Resources: What they want to access (Pods, Services, Secrets, etc.)

• Verbs: What actions they want to perform (get, list, create, delete, etc.)

To create RBAC rules, we first need to create Roles or ClusterRoles. A Role defines permissions within a specific namespace, whereas a ClusterRole defines permissions across the entire cluster. After defining the rules for resources and what actions can be performed on those resources, RoleBinding and ClusterRoleBinding are created to define which subjects these rules will be applied to.

For example:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: apache-manager
  namespace: apache
rules:
  - apiGroups: ["*"]
    resources: ["deployments", "pods", "services"]
    verbs: ["get", "apply", "delete", "watch", "create", "patch"]

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: apache-manager-role-binding
  namespace: apache

subjects:
  - kind: User
    name: apache-user
    namespace: rbac.authorization.k8s.io

roleRef:
  kind: Role
  name: apache-manager
  apiGroup: rbac.authorization.k8s.io

I also learned about important commands such as kubectl auth can-i get pods -n apache --as=<user> to check whether a user has permission to execute specified commands, which I found really useful while learning and testing. I also noted several best practices for RBAC policies, such as the principle of least privilege, namespace isolation, and service account strategy.

Custom Resource Definitions (CRDs)

Until now, I had been learning and experimenting with the built-in resources that Kubernetes provides, such as pods and services. But CRDs allow us to extend the Kubernetes API with our own custom resources. I can already imagine how useful CRDs can be as the complexity of modern applications grows rapidly. With CRDs, we can define application-specific resources that the Kubernetes API can manage natively. CRDs are essentially like custom data structures.

To create a CRD, we need to define the structure and validation rules of our custom resource. For example, I created a CRD for a simple database:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              engine:
                type: string
                enum: ["mysql", "postgres"]
              version:
                type: string
              storage:
                type: string
            required: ["engine", "version"]
          status:
            type: object
            properties:
              phase:
                type: string
  scope: Namespaced
  names:
    plural: databases
    singular: database
    kind: Database

After defining and applying this resource definition, we can use these resources in the same way as native Kubernetes resources and perform actions such as get, describe, etc.

apiVersion: example.com/v1
kind: Database
metadata:
  name: my-app-db
  namespace: production
spec:
  engine: postgres
  version: "1.4"
  storage: "100Gi"

Helm

I already knew that Kubernetes had multiple interconnected components, but what I didn't realize was just how many components there could be. So many that it can really become a hassle to create and manage all the YAML files for these components. This problem is solved by Helm. Helm addresses the critical problem of managing complex application deployments with multiple interconnected components. Instead of manually creating dozens of YAML files and managing their dependencies, Helm packages everything into reusable, versioned "charts." This idea immediately struck me as a genius solution, especially after seeing how lengthy and confusing these YAML files can become during my learning process.

Helm Charts

A Helm chart is a collection of files that describe a related set of Kubernetes resources. I think of it like a recipe book - it contains all the instructions (templates) and ingredients (values) needed to create a complete meal (application deployment). I installed Helm on my EC2 instance using the Helm installation guide and observed the structure of a Helm chart.

Chart Structure:

mychart/
├── Chart.yaml          # Chart metadata
├── values.yaml         # Default configuration values
├── templates/          # Kubernetes manifests templates
│   ├── deployment.yaml
│   ├── service.yaml
│   └── ingress.yaml
└── charts/            # Chart dependencies

The real power lies in templating - instead of static YAML files, Helm uses Go templates that dynamically generate manifests based on provided values.

We can either create a Helm chart from scratch or install charts from repositories that contain multiple charts. Chart repositories function like Docker registries - centralized locations for sharing and distributing packaged applications. Through these charts, we can manage multiple components for our Kubernetes cluster. The process followed for a Helm chart is as follows:

1. CREATE CHART
   ↓
   helm create myapp
   ↓
2. CUSTOMIZE TEMPLATES
   ↓
   Edit templates/*.yaml files
   Edit values.yaml
   ↓
3. VALIDATE & TEST
   ↓
   helm lint ./myapp
   helm template myapp ./myapp --debug
   ↓
4. PACKAGE CHART
   ↓
   helm package ./myapp
   ↓
5. DEPLOY TO CLUSTER
   ↓
   helm install myrelease ./myapp

A chart is a bundle that contains all the necessary resources for an application. Helm provides versioning and rollback capabilities, which are essential for maintaining application stability. If an update causes issues, teams can easily revert to a previous stable state using the helm rollback command. This feature is particularly valuable in CI/CD pipelines, where automated deployments and rollbacks are critical.

Service Mesh - Istio

After getting familiar with Helm and gaining hands-on experience with it, I learned about a new challenge in microservices architecture: managing communication between dozens or hundreds of microservices. In complex microservices architectures, cross-cutting concerns like security, observability, and traffic management become scattered across application code. The traditional solutions to this problem were either application-level (embedding networking logic into each service) or network-level (using traditional load balancers and firewalls). However, these approaches aren't suitable for the modern era. The solution to these challenges is Service Mesh.

A Service Mesh is a dedicated infrastructure layer for managing communication between services in a microservice-based system. It provides a centralized, dedicated infrastructure layer that handles the intricacies of service-to-service communication. Istio is a popular open-source service mesh tool.

Istio has two main core components:

1. Data Plane - Envoy Sidecars: Istio injects Envoy proxy sidecars alongside each application container. These proxies intercept all network traffic, providing:

   • Transparent service discovery

   • Load balancing and health checking

   • Security policy enforcement

   • Metrics collection and distributed tracing

2. Control Plane - Istiod: The unified control plane manages configuration and policy distribution:

   • Pilot: Service discovery and traffic management

   • Citadel: Certificate authority for mTLS

   • Galley: Configuration validation and distribution

At first, it seemed as complicated as it sounds, but by slowly understanding the infrastructure and working with Istio to deploy a sample application, I gained some confidence. I installed Istio and after deploying the sample application, I learned to visualize the service mesh and network using the Kiali dashboard.

Other Concepts

Apart from these major concepts, I also explored some other smaller but interesting and useful topics such as Node Affinity, Taints and Tolerations, Init containers, and Sidecar containers. I learned how these features contribute to the Kubernetes environment and did hands-on mini exercises for these services, like using Init containers for database readiness checks and sidecar containers for metric collection.

Resources I used-

1. Kubernetes Official Docs

2. Kubernetes | TrainWithShubham

3. Istio Documentation

Challenges I Faced

1️⃣ Service unavailable for K8s dashboard with Kind cluster

This was quite frustrating initially as I couldn't access the Kubernetes dashboard that I had set up. The dashboard service wasn't accessible through the usual methods, and I spent some time troubleshooting network connectivity issues.

Solution: I resolved this by using port forwarding with the command: kubectl -n kubernetes-dashboard port-forward service/kubernetes-dashboard 8443:443 --address=0.0.0.0. This allowed me to forward the dashboard service to my local machine and access it through the browser.

2️⃣ Couldn't connect to Istio Kiali Dashboard

After successfully installing Istio and deploying sample applications, I couldn't access the Kiali dashboard to visualize the service mesh. This was particularly challenging because the dashboard is crucial for understanding how services communicate within the mesh.

Solution: I used local port forwarding by opening an SSH tunnel that forwards local port 20001 to the EC2's port 20001. This allowed me to securely access the Kiali dashboard running on my EC2 instance from my local machine.

What's Next

After covering these many topics and concepts about Kubernetes, I'm planning to work on a comprehensive project that uses all these concepts and involves different third-party components and services such as Prometheus and Grafana. This should help me consolidate my learning and gain practical experience with real-world scenarios.

Let's Connect!

🔗 My LinkedIn

🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I'd love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Hello there to all 👋! This week has been one of the most overwhelming ones yet. I have been studying and learning Kubernetes for the past 8-9 days, and I have to say it hasn't been easy. The amount of information one has to absorb while learning and understanding Kubernetes is truly overwhelming. Last week, I got started with Kubernetes and learned what Kubernetes is about, what problems it solves, and its architecture. This week I continued on that journey and learned many important concepts such as Namespaces, Pods, Deployments, Services, Jobs, storage, and more.

Workload Resources

After going through and understanding the core concepts and architecture, I moved on to the next topic, which was Workload resources. Workload resources comprise Pods, Controllers, and Configuration management. Once again, to practice while learning, I created an EC2 instance of type t2.medium and learned to install Kubernetes through both Minikube and Kind. Once the K8s setup was done, I got started with the first topic, which was Namespaces.

Namespaces are virtual clusters within a physical Kubernetes cluster. Namespaces are Kubernetes' way of organizing and isolating resources within a cluster.

We know Kubernetes is a container orchestration tool, and all those containers are wrapped with an abstract layer called Pods.

Pods are the fundamental building blocks of Kubernetes, serving as the smallest deployable unit that can be created and managed within the system. The purpose of pods is to wrap containers with shared networking and storage.

I came to know that pods are ephemeral, which means they can fail or be destroyed easily. So as part of K8s's auto-healing process, K8s uses controllers like Deployment to maintain the desired state and re-create or restart pods to fulfill the desired number of pods.

Deployment manages multiple identical pods and ensures the desired number of pod replicas are running. The same functionality is provided by ReplicaSets, but Deployment has a few important advantages over it, such as rolling updates and rollbacks. Since these pods run inside the cluster, to access the applications running inside the pods, we use Services.

Service allows for stable and permanent networking for pods, ensuring that pods remain accessible even as they are updated or scaled. Service has 4 types: ClusterIP (internal), NodePort (external), LoadBalancer (cloud), and ExternalName.

While learning and practicing these concepts, I was creating these resources using commands, and then I came to know about manifest files. Manifest files are YAML or JSON files that describe the desired state of resources such as Pods, Deployments, and Services, and can be applied directly using the kubectl tool. This approach was much more efficient than using commands to create, update, or delete resources. Examples of all the manifest files for different resources are well documented in the Kubernetes official documentation, which has been a great help.

Workload Controllers

Like Deployment and ReplicaSets, there are more controllers that ensure the state of K8s is equal to the desired state, such as StatefulSet and DaemonSet.

StatefulSet manages stateful applications that need stable identities and persistent storage, such as databases, message queues, etc. Unlike Deployments, which are used for stateless applications, StatefulSets ensure that each Pod has a unique, persistent identity, including a stable hostname and volume.

DaemonSet ensures exactly one pod runs on every node (or selected nodes). DaemonSets are perfect for system-level services that need to run on every node, such as log collectors, monitoring agents, etc.

While these controllers are used to ensure the pods are running on the desired worker nodes and in the desired number, Jobs and CronJobs handle workloads that need to run to completion rather than continuously. While Deployments and StatefulSets keep applications running forever, Jobs are designed to run to completion and then stop. Jobs are perfect for one-time tasks, batch processing, and migration tasks. While Jobs run to completion for a single time, CronJobs run on a schedule. These can be used for scheduled maintenance, periodic backups, and regular data processing. I practiced creating Jobs and CronJobs by creating a simple CronJob to log a sentence.

Storage Resources

I have learned that Pods are ephemeral, which means if or when they die, their data disappears. But for stateful applications like databases, any team or company cannot afford to lose their data. So, the solution to this problem is Persistent Volume and Persistent Volume Claim, which provides storage that outlives individual pods.

Persistent Volume is a piece of storage in the cluster that has been provisioned by the administrator or dynamically provisioned. It represents actual storage like disk space or cloud storage, and this storage is not dependent on any pod.

Persistent Volume Claim is the request to use storage by an application. When an application requests storage (Persistent Volume), the volume binds to the application. The process of binding is as follows:

1. Administrator provisions PV (or dynamic provisioning creates it)

2. User creates PVC specifying storage requirements

3. Kubernetes matches PVC to suitable PV (binding process)

4. Pod uses PVC to mount persistent storage

5. Data persists even if pod is deleted

Networking - Ingress

Up to this point, I learned about internal networking for pods and how to access them from inside the container. But when deploying an application, we want users to be able to access and route through the application without any errors. For that purpose, Kubernetes Ingress is used to provide a centralized way to manage external access to services within a cluster. It allows for the routing of HTTP and HTTPS traffic to different services based on criteria such as URL paths and hostnames. Ingress is an important and essential part of the Kubernetes ecosystem.

Ingress comprises 3 main components:

1. Ingress Resource: The configuration object defining routing rules

2. Ingress Controller: The actual implementation that processes Ingress rules (nginx, Traefik, AWS ALB, etc.)

3. Backend Services: The destination services that receive routed traffic

Ingress has different routing strategies, such as path-based routing like /api, /app, etc., and hostname-based routing such as api.company.com and blog.company.com. Ingress is a complex and deep topic, so even though I learned to configure and implement Ingress rules in demo applications, I still need to dive deeper into this topic in the future.

Resource Configuration

So far, I had learned the fundamental yet important topics of K8s such as storage, networking, etc., but learning about resource configuration is as important as them. Resource configuration consists of concepts like ConfigMaps, Secrets, Resource Quotas, Limits, and much more. But for initial understanding, I decided to focus on these four first. By creating multiple manifest files for the creation and deletion of resources like pods, deployments, and services, I discovered that configuration should be stored in a separate environment. Just like we use .env to store configurations and secrets of the application, in K8s we use ConfigMaps and Secrets.

ConfigMap stores configuration data as key-value pairs that pods can consume as environment variables, command-line arguments, or mounted files. ConfigMap stores non-sensitive data like database URLs, settings, etc.

Secrets store sensitive data like passwords, tokens, and keys. They're similar to ConfigMaps but with additional security features. Data in Secrets configuration is usually stored in base64 format to encode binary data into text format for systems that can't handle binary data well.

These two are used to manage configuration data and sensitive information, whereas Resource Quotas and Limits are part of a resource management strategy that ensures fair distribution of resources. Resource Quotas and Limits work together to control how much CPU, memory, and other resources can be consumed at both the namespace and pod levels.

These are used to set maximum resources a container can use. If exceeded, the container is throttled (CPU) or killed (memory). Managing resources inside the cluster is critical to ensure that applications run smoothly while simultaneously optimizing costs.

Resources I Used

1. Kubernetes Official Docs

2. Kubernetes | TrainWithShubham

Challenges I Faced

1️⃣ K8s pod's status - CrashLoopBackoff when creating deployment

This was frustrating because the pods kept restarting in an endless loop. After debugging for a while, I discovered that I was working on the wrong Git branch, which had different configuration files than what I expected. The pod was trying to run with incorrect configurations, causing it to crash repeatedly.

Solution: Switched to the correct Git branch with the proper configuration files, and the deployment worked perfectly.

2️⃣ Unexpected Ingress error - Validating Admission Webhook for the NGINX Ingress Controller

This error appeared when I was trying to create Ingress resources. The webhook was rejecting my Ingress configurations, and despite trying various troubleshooting approaches like checking the webhook configuration and validating my manifest files, I couldn't resolve the issue.

Solution: After spending considerable time debugging, I decided to create a fresh cluster, which resolved the issue completely.

3️⃣ Wasn't able to update previously written manifest files

When trying to edit YAML files using vim, I kept getting permission denied errors. This was particularly annoying when I needed to make quick changes to my configurations during testing.

Solution: Had to use sudo with the vim command to get the necessary permissions to edit the files.

What's Next

The next topics for me in Kubernetes are Auto-scaling (HPA and VPA) and Taints and Tolerations, so I will continue with my Kubernetes learning journey.

Let's Connect!

🔗 My LinkedIn

🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I'd love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Hello there👋! This past week was both exciting and intense. After understanding Docker, I wanted to explore it more practically, so I decided to create a couple of mini projects. I also got started with Kubernetes, and to say it's confusing at first would be an understatement. In this article, I'll talk about my experience with the Docker projects and my first impressions of Kubernetes.

Mini Project 1: Web App with Nginx + MySQL

After getting the hang of Docker and its different components and concepts, I was ready to experiment by implementing what I learned to deploy something real. I decided to choose a simple Notes application with CRUD operations using Django for the backend, Nginx as a proxy, and MySQL as the database. I started with an EC2 instance and installed Docker and Docker Compose. Then I cloned the selected repository into the EC2 instance.

The repository was divided into frontend, backend, and nginx sections, and it had pre-existing Dockerfiles which I double-checked for reference. I created a fresh docker-compose.yml file with 3 services: Nginx, MySQL, and Django.

Here, the Nginx service depends on Django, and Django depends on the database service since the database needs to initialize first. For each service, I configured the ports, network, container name, and build image. Since the project had Django and MySQL, to connect Django and MySQL, we need to use the migrate command to create the database and tables, and Gunicorn is used to serve the Django application. All the database credentials were passed through the environment file.

To ensure proper initialization of services and proper connection, I used health checks in both the database service and Django app service. I had some problems with the containers not connecting to each other due to networking issues, but after some troubleshooting, I was able to initialize and connect the services properly.

Mini Project 2: Spring Boot Application

As part of my second mini project, I chose a Spring Boot application containing 3 components: Thymeleaf (UI), Spring Boot (backend), and MySQL (database), with Maven as the build tool. In the Dockerfile, I chose to go with a multi-stage build - the first stage to install all dependencies and build the application as an artifact, and the second stage with a smaller base image to run the application.

After completing the Dockerfile, it was time to configure the docker-compose file. This project had two main services: the main application and the database (MySQL). I followed the same process as the previous project - configuring container name, base image, ports, creating a separate network and volume, and setting up health checks to ensure proper initialization and connection.

The difference was in configuring environment variables. Using the application.properties file, I noted all the required parameters and provided them in the docker-compose file using the environment object. After some trial and error, I got the containers running and checked the application on the instance's public IP.

Getting Started with Kubernetes

After completing the projects using Docker, it was time to start something new. After Docker, I decided to start learning one of the most important tools in the DevOps field - Kubernetes. Kubernetes sits at the heart of modern cloud-native application deployment and management.

I started with "What is Kubernetes?" The answer I found through Kubernetes official documentation is that Kubernetes (often abbreviated as "K8s") is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. It's like the "operating system for the cloud" - just as your computer's OS manages programs and resources, Kubernetes manages containers and compute resources across multiple machines.

Then I wanted to know what problems Kubernetes actually solves. K8s solves problems like:

1. Manual scaling

2. Service discovery

3. Load distribution

4. Health monitoring

5. Resource optimization

While solving all of these critical problems, K8s also delivers concrete business value such as faster release cycles, cost efficiency, reliability, high scalability, and developer productivity. With the rise of containers, the moving trend from monolith to microservices, and the rising demand to manage hundreds of containers, container orchestration tools like Kubernetes have become an essential part of the software development and deployment lifecycle.

K8s Architecture

Understanding the Kubernetes architecture is essential as it provides the fundamental framework for managing containerized applications effectively. Kubernetes follows a master-worker pattern (also called control plane and data plane).

The key components of a Kubernetes cluster are:

• Pod: The smallest deployable unit in Kubernetes, grouping one or more containers that share storage, network, and a specification.

• kube-api-server: The central "front door" that receives and processes all REST requests for the cluster.

• etcd: A lightweight, distributed key-value store that persistently holds all cluster configuration and state.

• kube-scheduler: Watches for new pods with no assigned node and assigns them to the most appropriate node.

• kube-controller-manager: Runs background controllers (like replication and endpoint controllers) to ensure the cluster's desired state matches the actual state.

• cloud-controller-manager: Integrates Kubernetes with your cloud provider's API to manage cloud-specific resources (e.g., load balancers, volumes).

• kubelet: An agent running on each node that ensures containers described in PodSpecs are up and healthy.

• kube-proxy: Maintains network rules on nodes to allow communication to your pods from inside or outside the cluster.

• kubectl: The command-line tool used to send commands to the kube-api-server and manage your cluster.

The architecture follows a declarative workflow:

1. You submit a request (via kubectl) → API Server receives it

2. API Server validates and stores → Information goes to etcd

3. Scheduler assigns work → Decides which node should run the pod

4. kubelet executes → Node agent creates and manages containers

5. Controllers monitor → Ensure everything stays as desired

6. kube-proxy handles networking → Manages traffic routing

Through this, I understood how the components work together. I also learned about different installation methods for Kubernetes such as Minikube, Kubeadm, and KinD. As I got to know more about K8s, the more confusing it became for me, and I expected that since Kubernetes is not easy to learn and is a big and complex tool. But slowly and steadily, I hope I can learn K8s effectively and use it for practical and real-world implementations.

Resources I Used

1. Kubernetes Crash Course | Techworld with Nana

2. Kubernetes | TrainWithShubham

3. Docker | TrainWithShubham

4. Kubernetes Official Docs

Challenges I Faced

1️⃣ Containers couldn't connect to network

This was frustrating because the containers were running individually, but they couldn't communicate with each other. I spent quite some time debugging this issue before realizing the root cause.

Solution: I hadn't configured the network properly between the services in the docker-compose file. I needed to explicitly define a custom network and ensure all services were part of the same network to enable inter-container communication.

2️⃣ Application redirecting to Nginx welcome page instead of the notes app

When I opened the application on port 80, it kept showing the default Nginx welcome page instead of my notes application. This was confusing because the containers seemed to be running fine.

Solution: The issue was in the Nginx configuration. I had to change the container name to match the service name in the default.conf file in Nginx. This allowed Nginx to properly proxy requests to the Django application container.

3️⃣ Difficulty understanding the key components of K8s and its flow

Kubernetes felt overwhelming at first. There are so many components and concepts, and understanding how they all work together was challenging. The official documentation, while comprehensive, felt too technical for a beginner.

Solution: I referred to a Kubernetes crash course video to better understand the big picture and components. Visual explanations and practical examples helped me grasp the concepts much better than just reading documentation.

What's Next?

As I said, Kubernetes is a complex and difficult tool to master, so I will continue to learn and understand the basics first and build a strong foundation before moving on to advanced concepts. I plan to set up a local Kubernetes cluster and deploy some simple applications to get hands-on experience.

Let's Connect!

🔗 My LinkedIn 🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I'd love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Hello there! I was unable to write any article for the past 2 weeks due to a few reasons, but the learning process didn't stop. For the past two weeks, I have been revising and exploring the capabilities of Python and understanding Docker in a more comprehensive way, which is one of the most important tools in the modern software development era. So in this article, I will be discussing the experiments I have been doing with Python and my experience learning Docker.

Python Revision and Scripting

Python is widely used in DevOps due to its versatility. It is used for automation, CI/CD pipelines, infrastructure as code (IaC), and more. One of the most common applications of Python is scripting, and the extensive library support makes Python an easy choice for engineers when it comes to automation.

In my first semester, Python was part of our curriculum, but it wasn't taught extensively. So first, I revised the basic syntax and fundamental concepts of Python such as lists, sets, dictionaries, functions, etc., and practiced them using simple programs.

After revising the basic syntax and concepts, it was time to learn important stuff like modules, automation, and scripting. The libraries I learned about that are really important and useful in scripting are os, subprocess, and argparse. These modules are essential for automating Linux system tasks. I learned different functions of these modules such as os.system(), os.mkdir(), subprocess.run([]), etc. To implement these functions in a real-world script, I created an Ubuntu VM and wrote a user management script to perform addition and removal of users and groups.

#!/usr/bin/python3

import argparse
import subprocess
import logging
import getpass

logging.basicConfig(
    filename="logs/actions.log",
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s"
)

# Main command runner
def run_cmd(cmd):
    try:
        result = subprocess.run(cmd, check=True, text=True, capture_output=True)
        logging.info(f"Command succeeded: {' '.join(cmd)}")
        print(result.stdout.strip())
    except subprocess.CalledProcessError as e:
        logging.error(f"Command failed: {' '.join(cmd)}\n{e.stderr}")
        print(f"Error: {e.stderr.strip()}")

# Argument parser
parser = argparse.ArgumentParser(description="Linux User & Group Management")

parser.add_argument('--add-user', help="Add a new user")
parser.add_argument('--group', help="Assign user to group (with --add-user)")
parser.add_argument('--del-user', help="Delete a user")
parser.add_argument('--add-group', help="Create a new group")
parser.add_argument('--del-group', help="Delete a group")
parser.add_argument('--list-users', action='store_true', help="List all system users")
parser.add_argument('--change-pass', help="Change password of a user")

args = parser.parse_args()

# Actions
if args.add_user:
    cmd = ["sudo", "useradd", args.add_user]
    if args.group:
        cmd.extend(["-G", args.group])
    run_cmd(cmd)

if args.del_user:
    run_cmd(["sudo", "userdel", "-r", args.del_user])

if args.add_group:
    run_cmd(["sudo", "groupadd", args.add_group])

if args.del_group:
    run_cmd(["sudo", "groupdel", args.del_group])

if args.list_users:
    run_cmd(["cut", "-d:", "-f1", "/etc/passwd"])

if args.change_pass:
    password = getpass.getpass("Enter new password: ")
    confirm = getpass.getpass("Confirm new password: ")
    if password == confirm:
        proc = subprocess.Popen(["sudo", "passwd", args.change_pass], stdin=subprocess.PIPE)
        proc.communicate(input=f"{password}\n{password}\n".encode())
        logging.info(f"Password changed for {args.change_pass}")
    else:
        print("Passwords do not match.")
        logging.warning(f"Password mismatch for user {args.change_pass}")

# Help command if no arguments are passed
if len(vars(args)) == 0:
    parser.print_help()

With the help of ChatGPT, I enhanced the script using the getpass module for the change password action, and it was working as expected.

Python Automation

One of the main use cases of Python in DevOps is automation. I came to know about two libraries that are primarily used for automation: Boto3 and Fabric. Boto3 is the official SDK that is used to interact with AWS services and simplifies the process of creating, deleting, and configuring different AWS services. While learning how to use Boto3, it became very clear to me that to make use of boto3 or even any automation tool, we first need to fully understand the manual process that we are trying to automate. So the knowledge of creating EC2 instances, uploading files to S3 buckets, and creating RDS instances using the AWS console with all the steps was crucial in order to write automation scripts using Boto3.

Using the Boto3 documentation, I created a few scripts to create EC2 instances, upload files to S3, and perform health checks on running EC2 instances. Understanding the responses of each call like ec2.describe_instances() was tricky at first as there was so much information to first understand and then filter out for what you actually want, such as instance ID and status. Formatting the responses using the pprint library helped in this process.

Docker Deep Dive

Earlier, to learn and work with different technologies like Jenkins and AWS, I learned about Docker but not in depth. But for DevOps, saying Docker is an essential tool would be an understatement, so learning Docker extensively is a must. So first, I checked what the main topics to learn Docker were, and Claude provided me with a list:

• Docker Overview (bigger picture)

• Docker Architecture

• Images

• Containers

• Dockerfile

• Docker Networking

• Volumes and Storage

• Docker Compose

• Docker Registry

Docker solves the "But it works on my machine" problem that was repeated millions of times in the past. The main reason for this problem is inconsistencies in development environments such as differences in operating systems, application dependency issues, etc. To overcome this issue, containerization is used—a lightweight form of virtualization that packages an application and all its dependencies into a portable, executable unit called a container.

Docker operates on a client-server architecture with three primary components:

1. Docker Engine

2. Docker Client

3. Docker Registry

I learned in-depth about important terminologies like Dockerfile, Images, and Containers. I also learned about the flow of Docker: Dockerfile is used to build read-only templates which are called Images that contain everything needed to run the application. These images are used to create Containers in which applications run.

I practiced Docker on an EC2 instance by cloning a few simple projects and trying to containerize them. I wrote Dockerfiles for different projects such as Java applications and Python applications—getting base images, creating working directories, copying source code, and installing dependencies. But the main challenge was when I tried to create a Dockerfile for my own Node project. When I was able to build its image successfully and deploy its container and could access my project, I was delighted.

I proceeded to learn more important concepts such as networking in Docker, how Docker containers communicate with each other and with external networks using network drivers such as Bridge, Host, Overlay, and None network. After deploying a container for any project, when I was trying to restart the container, the data was being lost from the application, and then I came to know about the importance of Volumes and Storage. Volumes and Storage solve the challenge of data persistence in containerized applications. By creating and mounting the containers to named volumes, even if we stop, remove, or restart the container, the data will be stored in the attached volume.

While learning these topics, I noticed one thing: this is a lot of manual work for such simple projects—pulling images, creating Dockerfiles, building application images, running containers, all actions through manual commands. While commands are great for understanding the fundamental work, for real work at companies, this would be too time-consuming to configure hundreds of containers. The solution to this problem is Docker Compose. Docker Compose is a tool for defining and running multi-container Docker applications using YAML configuration files. I learned about the syntax and different components of the docker-compose.yml file such as services, networks, and volumes. To make use of it, I first removed all the containers from the previous projects and then configured a docker-compose.yml file to build a two-tier Flask application with a Flask backend and MySQL database.

At last, I understood Docker Registry—a centralized repository for storing, managing, and distributing Docker images. After logging into DockerHub using docker login, I learned to tag images and to push and pull from DockerHub. I pushed my two-tier-flask-app image to DockerHub, which I can use in the future to practice more.

Resources I Used

1. Techworld With Nana | Python Tutorial

2. Boto3 documentation

3. TrainWithShubham | Docker

Challenges I Faced

1️⃣ Python interpreter not working in Git Bash

• The Python interpreter wasn't working properly in Git Bash when trying to run my automation scripts due to path recognition and environment variable issues.

  Solution: Switched to PowerShell which resolved the issue completely.

2️⃣ Wrong path error after deploying Docker container for canvas-assignment application

• Encountered path errors that prevented the React application from loading correctly in the containerized environment.

  Solution: Replaced "homepage" line in package.json to "." and forced the PUBLIC_URL environment variable in Dockerfile.

3️⃣ Difficulty in creating health check in docker-compose.yml file

• Had trouble understanding the syntax and getting the health check commands right for multi-container application monitoring.

  Solution: Referred to Docker Compose documentation and configured proper health checks that monitor both application status and database connectivity.

What's Next?

For the next week, I am planning to write some more automation scripts using Python to understand more about scripting. I am also planning to combine all the Docker knowledge and use all the components to create a comprehensive project and then revise all these concepts.

Let's Connect!

🔗 My LinkedIn 🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I'd love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Hello there! 👋 The past week was the 9th week of my DevOps learning journey, and I have to say that this was one of the toughest ones so far. Last week, I got started with CI/CD in Jenkins, and this week I continued to dive deeper into it. I learned and understood the Master-Slave Architecture in Jenkins, different types of build triggers and how they work, versioning of artifacts, Pipeline as Code, and learned to build an end-to-end CI/CD pipeline using different technologies.

Master-Slave Architecture

One of the most important concepts in Jenkins is understanding the Master-Slave Architecture. This architecture is used to distribute and organise the workload of running multiple jobs across different machines (physical, virtual, or cloud-based).

Jenkins Master is the central server that hosts the Jenkins web UI, manages job configurations, schedules builds, and coordinates agents (slaves).

Jenkins Slaves are remote machines that connect to the master and execute build jobs or specific stages (e.g., CPU-intensive tests, platform-specific builds) and then send the reports of the build back to the master.

There are many benefits of this architecture:

• Scalability - Distribute workloads across multiple machines; the master stays responsive.

• Isolation - Run builds in dedicated environments (different OS, JDK versions, or isolated containers).

• Platform diversity - Agents can run on Linux, Windows, macOS, or even containerised environments. It is useful when you need to test on multiple OSes or hardware architectures.

• Security - Master is protected (usually behind a firewall); only agents need SSH/JNLP access.

• Maintenance and upgrades - We can reboot or upgrade agents with minimal disruption, leaving the master unaffected.

I also learned about the prerequisites required, such as OS considerations, requirements of Java, JRE and JDK, networking so master and slave agents can connect to each other, user permissions and working directories, etc. All this was theoretical knowledge needed to understand the benefits and working of Jenkins, so after this I learned to create slave agents using Jenkins and EC2 instances.

I created an Ubuntu EC2 instance with SSH access from my IP address and also from Jenkins Master. Then I installed Java, created the jenkins-agent user, and gave it the required permissions for the workspace. In the Jenkins Master, I added a new Node with remote root directory and the label. In Jenkins, we can assign jobs to different slaves or groups of slaves using labels, so it becomes important to carefully group the slaves which will be used for jobs for a specific project or task. To make use of labels, we have to select "Only build jobs with label expressions matching this node" in the Usage dropdown.

For credentials, I created a credential of "SSH Username with private key" kind with an ID which will be used to select it in the future. I copied the private key from the .pem file and pasted it in the private key section. After saving the configuration, I checked in the slave agent's workspace directory if there was a "remote" directory. If yes, the slave agent has been connected successfully to the master and is ready to be assigned jobs.

Pipeline as Code

A job in Jenkins can be of two types - Freestyle and Pipeline as Code. In freestyle jobs, we manually configure the job settings and build steps in the Freestyle job dashboard. Whereas in Pipeline as Code, it allows you to define the entire CI/CD process in a declarative or scripted syntax inside a Jenkinsfile. This file lives in the source code repository, enabling version control of the build process just like your application code.

Pipeline as Code is the standard for industry-level CI/CD pipelines as it helps in automation of the workflows, enables storing pipeline logic in source control, and provides advanced features like parallel execution, artifact management, and more.

A pipeline is of two types:

1. Declarative - Structured & opinionated

2. Scripted - Groovy-based & flexible

I understood the key components/blocks of a Jenkinsfile, such as pipeline, agent, environment, options, stages and stage, steps, when, and post block. Each type of block serves a different purpose and is useful in creating a pipeline. The Jenkinsfile is placed in the root directory of the source code repository so that Jenkins can automatically detect and run it when the pipeline job is configured properly. I created a few build pipelines with simple steps such as checking out code from GitHub, building the application using Maven, and running tests.

CI/CD Pipeline

After I learned and understood the basic working and management of Jenkins, it was time to put it all together and create something to gain hands-on knowledge. A CI/CD pipeline consists of checking out the code, running tests, building the artifact, running code analysis to check for bugs and vulnerabilities, storing the artifact in an artifact repository if it passes all the quality gates, and deploying it in the dev or staging environment.

At first, I decided to create a Continuous Integration pipeline only, which later became a complete CI/CD pipeline. The tools I decided to use for the CI pipeline:

• Build tool - Maven

• Code Analysis - Sonar Scanner

• Storing the artifact - Nexus repository

• Notification - Slack Integration

Steps for creating the CI pipeline:

1. Jenkins setup

• Installing and configuring required JDK and Maven version

• EC2 instance (Ubuntu, t2.medium)

• Security group rules:

  • SSH, port 22, my IP

  • Custom TCP, port 8080, my IP (Jenkins webpage access)

  • HTTP, port 80, Anywhere

  • Custom TCP, port 8080, Sonarqube-sg (Sonarqube communication)

2. Nexus setup

• EC2 instance (AWS Linux, t2.medium)

• Configuring the setup through a script

• Security group rules:

  • SSH, port 22, my IP

  • Custom TCP, port 8081, my IP (Nexus webpage access)

  • Custom TCP, port 8081, Jenkins-sg (Publish artifact from Jenkins)

3. SonarQube setup

• EC2 instance (Ubuntu, t2.medium)

• Configuring the setup through a script

• Security group rules:

  • SSH, port 22, my IP

  • HTTP, port 80, my IP (SonarQube webpage access)

  • HTTP, port 80, Jenkins-sg (Allow Jenkins to send reports)

4. Installation of required plugins

• Nexus artifact uploader

• Sonar Scanner

• Pipeline Maven integration

• Build timestamp

• Pipeline utility steps

5. Integration of

• Nexus

• SonarQube

6. Write Pipeline script

7. Set a notification using Slack

After launching EC2 instances for Nexus and SonarQube, the real challenge was to integrate each of them with Jenkins. I won't be talking about these steps in-depth, but the high-level steps were to install the required plugins for these tools, configure the tools in Manage Jenkins > Configure System, create authentication tokens for each of them so Jenkins can connect and communicate to them, provide it in Jenkins configuration, integrate custom quality gates for SonarQube, and configure the pipeline script with code analysis and upload artifact to Nexus steps.

After fixing a few typos and configuration errors, I was able to run the job successfully.

After creating this CI pipeline, the next step was to create a complete CI/CD pipeline using Docker, Docker Hub, and AWS ECR (Elastic Container Registry) and ECS (Elastic Container Service). For deploying the Docker image through ECS, I replaced the Nexus repository with AWS ECR.

Steps for integrating Docker, AWS ECR and ECS-

1. Docker Engine in Jenkins

• Installed Docker on the Jenkins server so it can build container images

• Required for using docker CLI commands in the pipeline

2. Add Jenkins User to Docker Group & Reboot

• Allows the Jenkins user to run Docker commands without sudo

3. Install AWS CLI

• Enables Jenkins to interact with AWS services like ECR from the command line

• Used in the pipeline for commands like aws ecr login

4. Create IAM User

• A dedicated IAM user with programmatic access and permissions (AmazonEC2ContainerRegistryFullAccess and AmazonECS_FullAccess)

• Ensures secure and scoped access to your AWS ECR for Jenkins

5. Create ECR Repository

• Create an Elastic Container Registry to store and version Docker images

• The destination for built images to be pushed from Jenkins

6. Plugins: ECR, Docker Pipeline, AWS SDK

• ECR Plugin – Helps interact with ECR directly from Jenkins

• Docker Pipeline Plugin and CloudBees Docker – Enables Jenkins to run Docker build/push commands in a pipeline

• AWS SDK Plugin – Enables Jenkins to store and use AWS credentials securely

7. Store IAM Credentials in Jenkins

• Use Secret Text or AWS Credentials type under "Manage Credentials"

• Required for Jenkins to authenticate with AWS in your pipeline securely

8. Create ECS Cluster and Task Definition

• ECS Cluster is a logical grouping of tasks (Fargate or EC2) and services

• Task Definition is like a blueprint for your container: image location, CPU/memory requirements, port mappings, and log settings

9. Create a Load Balancer and ECS service

• Required to distribute traffic across multiple Fargate tasks and expose your service to the internet

• Attach the ECS Service to Load Balancer which ensures that a specified number of tasks (containers) are always running

10. Run Pipeline

• Executes stages like Docker build, login to ECR, and push image

• Completes CI/CD by deploying containerised artifacts to the cloud registry

The hardest part was learning and configuring AWS ECR and especially ECS. I did all these steps one at a time, and in between, I ran the job multiple times to test whether the steps were working fine. Again, each of these is much more elaborate than I mentioned, but I don't want to stretch the article that long. After writing the complete pipeline and checking beforehand for errors in the script, I created a new Pipeline Job, and seeing it build perfectly was such a rewarding feeling.

For creating a CI/CD pipeline from scratch, it is very important to understand the flow of the pipeline and plan each step accordingly. Planning is as important as configuring and executing those steps.

At last, I learned while exploring Jenkins how Authentication and Authorization work. Jenkins allows the admin to control exactly who can do what - like accessing the dashboard, reading or editing jobs, configuring pipelines, or deleting them. I learned how large companies manage multiple users working on different projects at the same time using Jenkins. They create roles that group users based on their responsibilities, and then assign the same set of permissions to everyone in that role. This not only makes user management easier but also saves a lot of time and reduces the chances of mistakes.

Resources I used-

1. Docker Installation

2. Slack Marketplace

3. DevOps Course | Udemy

Challenges I faced-

1️⃣ Disk space issue in the Jenkins server

• 8 GB volume was not enough to run multiple build jobs in the Jenkins server and the jobs kept crashing.

Solution: Increased the EBS volume from 8 GB to 20 GB for the Jenkins EC2 instance.

2️⃣ URL issues for different servers

• For accessing the web pages of Jenkins, Nexus, and SonarQube, when I pasted the public IPs for each instance in the browser, I wasn't able to access the web pages.

Solution: For some reason, the URL was using HTTPS for these public IPs, so I had to manually type http://<INSTANCE_PUBLIC_IP> and then I was able to access the web pages.

3️⃣ Unable to connect the Slave agent with Master

• When adding a new Node in Jenkins Master, I was using the public IP of the slave EC2 instance in the "Host" input and it wasn't connecting to master for that configuration.

Solution: I replaced the public IP address with the private IP address of the slave's EC2 instance, which worked for me.

What's Next?

To be honest, it was one of the toughest things to learn so far as there were too many things to learn in Jenkins, and along with that, I learned about integration with SonarQube, Nexus, AWS ECR, and AWS ECS. So firstly, I would revise all the concepts of Jenkins once again, go through all the steps of creating a pipeline to fully understand the flow, and then next week get started with Python. It was a long article as I had to cover a lot of stuff - sorry about that, and thank you if you still read it till here! :)

Let's Connect!

🔗 My LinkedIn

🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I'd love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Hello there! 👋 It's been a while since I last posted because of exams and a week of relaxation after coming back to my hometown for vacation. But in the meantime, I managed to do some useful work, and this past week, I got back on track with my DevOps learning. So for the past few weeks, I had been working on an AI GitHub SaaS project, and this week I got started with CI/CD processes using GitHub Actions and Jenkins.

GitHub SaaS Project

Previously, after finishing any of my projects, I wanted to document them for GitHub. I used to either create the README for that project on GitHub manually or give the context of what my project does to ChatGPT and just hope it would do a good job understanding the idea and context of the project. So then I thought, what if I create my own app to generate READMEs for my projects by using all of the context of my project directly from GitHub? From then on, I started researching about the GitHub API, how to get the source code of any project, and give it to any LLM to generate a perfect README for me.

Through the Octokit library, we can interact with the GitHub API and get the source code, but what if the source code for each file goes beyond the context limit of Gemini? So to solve this problem, I came to know about a method called "Prompt Chaining." In this method, first, we ignore the irrelevant files like package.json, node_modules, yarn.lock, bun.lock, .env, etc. Then we generate a summary for each relevant file. We store the summaries of all the relevant files in our database and then pass all those summaries as context to create the README.

While researching about this method, I was discussing with one of my friends about this, and he gave me a suggestion that if I can get the context of any project efficiently, I can do whatever I want with that data. And that sparked another idea - we can implement a Q&A-based feature where users can query anything about their codebase. But there's a problem with that too: how can we answer specific queries about anything from the codebase? How can we know which files are relevant for the query? The answer to that was Text Embedding.

We can create vector embeddings of the queries and the files of the codebase, and through matching them, we will get the relevant files and the required answer to the query using AI. I learned a few necessary things about Text Embedding and Vector databases and how I can use them in my project. For that, along with the generation of the summary of each file, we would also need to generate the embedding of the summary of each file and store it. Through them, we will get the answer to the query asked about the codebase. This could be very useful for junior developers or interns joining a new project or for open source projects where contributors want to have a thorough understanding of the codebase before contributing.

The tech stack I used for this project:

🔹 NextJS

🔹 TypeScript

🔹 TRPC

🔹 TailwindCSS

🔹 Shadcn

🔹 Clerk

🔹 Google Gemini API

🔹 Razorpay

🔹 PostgreSQL

🔹 Prisma

Feel free to check out the project and GitHub repository. Also, if you have any suggestions to improve the website, please do let me know.

GitHub link of the project: https://github.com/Akshansh029/Nebulo

Project link: https://nebulo-zeta.vercel.app

Getting Started with CI/CD

CI/CD with GitHub Actions

After completing the project and deploying it, this week I got back on track with DevOps. It was time to start one of the most important topics of DevOps - Continuous Integration and Continuous Delivery (CI/CD). As I had already studied about CI/CD and the principles used in it, I did a quick recap and wanted to start with the implementation part. I started with GitHub Actions.

GitHub Actions is a built-in CI/CD automation tool in GitHub. It lets you run workflows directly in your repository to automate tasks like:

• Building code

• Running tests

• Deploying to production

It works using YAML configuration files inside your repository.

The workflows are defined inside:

.github/
└── workflows/
    ├── ci.yml
    └── deploy.yml

Key components of a workflow are:

The steps defined in the .yml file are executed in virtual or physical machines called Runners. There are two types of runners which GitHub provides:

1. GitHub Hosted Runners (Shared) - These are provided by GitHub with pre-configured environments.

2. Self-hosted Runners - Users install and manage them and have complete control over hardware and OS.

I learned different concepts and components of GitHub Actions and how to use them in a workflow, such as using multiple jobs, environment variables, job concurrency, matrix strategy, etc. As practice, I wrote and used two workflows - one to build and test the source code of a Node application and another to build the Docker Image and store it in DockerHub.

integration.yml:

name: Node.js Build

on:
  push:
    branches: ["main"]
  pull_request:
    branches: ["main"]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [18.x, 20.x, 22.x]

    steps:
      - uses: actions/checkout@v4
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
          cache: "npm"
      - name: Install dependencies
        run: npm install

      - name: Build command
        run: npm run build

  unit-tests:
    runs-on: ubuntu-latest
    needs: build
    strategy:
      matrix:
        node-version: [18.x, 20.x, 22.x]

    steps:
      - uses: actions/checkout@v4
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
          cache: "npm"
      - name: Install dependencies
        run: npm install

      - name: Running unit test cases
        run: npm run test

deploy.yml:

name: Deploy on DockerHub

on:
  push:
    branches: ["main"]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Docker Build process
        run: docker build . -t akshansh29/next-js-app:latest

      - name: DockerHub login
        run: echo "${{secrets.DOCKERHUB_PASSWORD}}" | docker login -u ${{secrets.DOCKERHUB_USERNAME}} --password-stdin

      - name: Pushing image to DockerHub
        run: docker push akshansh29/next-js-app:latest

CI/CD with Jenkins

Jenkins is one of the most used DevOps tools to automate CI/CD workflows. It is an open-source automation server used to automate software development processes. Some core features of Jenkins are:

• Pipeline as Code using Jenkinsfile

• Extensible plugin ecosystem (e.g., Git, Maven, Docker, Slack, Kubernetes)

• Distributed builds using master-agent architecture

• Supports polling SCM or webhooks for triggering builds

• Can schedule jobs (cron-like syntax)

• Provides a dashboard and build history

To get started with Jenkins, I referred to the Jenkins docs for the installation guide. I created an EC2 instance to run the Jenkins server on it, SSH into it, and I copied the Linux commands to install Jenkins from the docs and set up the Jenkins server. I learned about the home directory of Jenkins, i.e., /var/lib/jenkins, where all the files of Jenkins are stored. Using the instance's IP address and port number, I accessed the Jenkins web page.

I also learned about Jobs - A Job in Jenkins is a task or project that Jenkins performs, such as pulling code from a repository, building a project, running tests, etc. It is of two types:

1. Freestyle project - A simple, GUI-based job configuration which is ideal for small or basic automation tasks.

2. Pipeline as Code - Jenkins Pipeline is a suite of plugins that support pipeline as code using a Jenkinsfile, which is a plain-text file checked into the source repository, enabling complex automation scenarios with full control.

Till now, I have learned how to create Freestyle projects and how to install plugins and tools in Jenkins. I learned how I can configure a freestyle project to install different tools like Git and Maven using shell commands and also by using the "Build Environment" section and selecting required tools. I configured a build job in which it should fetch the code from a GitHub repository, run the required tests, and build the artifact. All the relevant files and artifacts are stored in the workspace in the build section.

Resources I Used

1. GitHub Actions | KodeKloud

2. CI/CD Explained | TechWorld with Nana

3. Installation guide | Jenkins

4. DevOps Course | Udemy

Challenges I Faced

1️⃣ Too many requests to GitHub API

Solution: Collected only relevant files by filtering out using file extensions and implemented Exponential Backoff with retries, which is a strategy where you wait progressively longer before retrying a failed request. The delay increases exponentially (e.g., 500ms, 1000ms, 2000ms).

2️⃣ Error in testing the Vprofile using Maven due to a different JDK version

Solution: Installed the required JDK version.

3️⃣ Jenkins webpage access issue

Solution: Changed the inbound rule from my IP to every IPv4 address for testing and rebooted the instance.

What's Next?

I'll continue learning Jenkins, which is an essential tool that I want to master. I will learn about Pipeline as Code, practice by creating different pipelines, and make a project to create an end-to-end CI/CD pipeline.

Let's Connect!

🔗 My LinkedIn

🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I'd love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Hello there! 👋 The past week marked Week 7 of my DevOps journey, and this time, I started by revisiting concepts of Computer Networks, as it’s been a while since I looked back at them. I continued my AWS learning by exploring services like EFS, Auto Scaling Groups, S3, and RDS. To revise and implement my previous learnings, I built a project called Vprofile, where I used AWS services to deploy a Java-based application using EC2, S3, Load Balancers, Route 53, and Auto Scaling Groups.

Exploring AWS Storage Services

Elastic File System (EFS)

AWS has many options for storage, and as a DevOps learner, I should be proficient in using these storage services. The first one is EFS or Elastic File System. It is a fully managed, scalable NFS file system that can simultaneously be mounted on multiple EC2 instances. EFS is like a virtual hard disk which can store and protect our data from EC2 instances, Lambda, ECS, EKS, etc. Some of its key features are

• Shared file storage for Linux workloads

• Elastic & Scalable: Automatically grows/shrinks as files are added/removed

• Supports POSIX-compliant permissions (Linux-style)

• Accessible across multiple AZS

• Use Cases: Web servers, CMS, containers (EKS, ECS), CI/CD, ML, shared file access

After covering some theoretical knowledge of EFS, I jumped into how to create and mount one in an EC2 instance. I followed the following high-level steps to accomplish this-

1. Create an EFS File System in the AWS Management Console

2. Create Access Point for EFS (Recommended)

3. Configure the Security Group to allow an NFS type inbound rule for port 2049

4. Configure EC2 Instance

   For Amazon Linux 2 / Amazon Linux AMI

    sudo yum update -y
    sudo yum install -y amazon-efs-utils
   

   For Debian/Ubuntu

    sudo apt update
    sudo apt install -y amazon-efs-utils
   

5. Create a Mount Directory inside the EC2 instance (eg, /mnt/efs)

6. Mount EFS Temporarily (Test)

    sudo mount -t efs -o tls,accesspoint=access_point_id efs_file_system_id:/ /mnt/efs
   

7. Verify using:

    df -h
   

8. Mount Persistently Using /etc/fstab

9. Test and check using:

    sudo mount -a
    df -h
   

Simple Storage Service (S3)

After EFS, I moved to Simple Storage Service (S3) . It is a service that stores files of different types, like Photos, Audio, and Videos, as Objects, providing more scalability and security. The main components of S3 are Bucket, Key, and Object. S3 has many use cases, like-

• Static website hosting

• Backup and restore

• Data archiving

• Big data analytics

• Disaster recovery

and many more. The thing I liked about S3 is that it is really simple to get started with S3. There’s not much hassle in using S3. I learned about different classes of S3.

I also learned about versioning, access control using bucket policies, IAM policies, ACLs and Lifecycle policies, which is how we can configure the transition of objects between storage classes based on how old they are. I also learned to host a static website using S3 buckets and how disaster management is done using replication rules.

Relational Database Service (RDS)

The next storage service was RDS or Relational Database Service and unlike S3, it wasn’t easy for me to grasp. It is a managed Relational Database Service supporting engines such as Mysql, Postgresql, Mariadb, Oracle, SQL Server, and Amazon Aurora. Some of its key features I came to know-

• Automated Backups: Point‑in‑time recovery with daily snapshots and transaction logs

• Multi‑AZ Deployments: Synchronous standby in a separate Availability Zone for failover

• Read Replicas: Scale read workloads and offload reporting

• Storage Auto Scaling: Automatically increase storage when you approach capacity limits

• Encryption at Rest & In Transit: AWS KMS–managed keys or customer‑managed keys

• Monitoring & Metrics: Integration with CloudWatch, enhanced monitoring, Performance Insights

• Maintenance Windows: Automatic minor version upgrades during a defined time window

It has many use cases in real-world applications like web and mobile applications that require relational schema, analytics and reporting with read-heavy queries, etc. I learned how to create an RDS instance, how to configure it, how to delete it and learned about a few best practices for RDS.

Project Highlight: Vprofile Deployment on AWS

And to practice and implement a few services which I learned, I made a project to deploy a multi-tier Java application. Building a project is always the fun part of learning (till you encounter an unsolvable error and lose all your motivation 🙂). The services I used for this project are-

1. EC2 Instances - VM for Tomcat, RabbitMQ, MemCache, MySQL

2. Elastic Load Balancer (ELB) - Nginx Load Balancer replacement

3. Auto Scaling Groups - Automation for VM scaling

4. S3/EFS Storage - Shared storage

5. Route 53 - Private DNS service

6. IAM - To create a user and a role for S3 full access

The architecture of the project-

The user accesses the application via a custom domain registered on GoDaddy, which routes traffic through an HTTPS-enabled Elastic Load Balancer (ELB). The ELB forwards requests on port 8080 to an Auto Scaling Group of EC2 instances, ensuring high availability and performance. These instances interact with backend services, including Rabbitmq for messaging, Mysql for database operations, and Memcached for caching, all hosted on separate EC2 instances within a secure group. The setup also integrates Route 53 for DNS management (with Private Zones) and uses S3 buckets to store application artifacts, making the deployment both scalable and modular.

While there's a lot more depth to how each service was integrated and managed, for now, I’ll focus on outlining the key steps of the overall process to give a clear high-level view.

Steps involved-

1. Set Up Security Groups

   Created separate security groups for the Load Balancer, Tomcat app server, and backend services (Mysql, Memcache, Rabbitmq). Each group had tightly scoped rules- like allowing port 8080 access from the ELB SG or enabling internal backend communication via private IPs and ports.

2. Create a Key Pair for SSH Access

   Generated an AWS key pair to securely SSH into EC2 instances during setup and troubleshooting.

3. Launch EC2 Instances for Backend Services

   Used Amazon Linux AMI to launch instances for Mysql, Memcached, and Rabbitmq. Applied startup scripts from the vprofile repo during instance creation and attached the appropriate backend security group.

4. Configure Private DNS with Route 53

   Created a Private Hosted Zone in Route 53 to map backend service hostnames (e.g., db01.vprofile.in) to their private IPs. This made service-to-service communication IP-independent.

5. Test DNS Resolution

   Verified hostname resolution by SSHing into the Tomcat instance and pinging other backend services using their DNS names (e.g., ping -c 4 db01.vprofile.in).

6. Build the Java Artifact Using Maven

   Cloned the vprofile source code locally, built the .war file using mvn install, and verified the build in the target directory.

7. Set Up S3 for Artifact Storage

   Created an S3 bucket to store the built .war file. Used AWS CLI to upload the artifact. An IAM user with S3 access was configured locally for this step.

8. Assign IAM Role to Tomcat EC2 Instance

   Created an IAM role with full S3 access and attached it to the Tomcat instance so it could pull the .war file directly during deployment without AWS credentials.

9. Deploy the Application on Tomcat

   Pulled the .war from S3 into the Tomcat EC2 instance, replaced the default webapps directory with the new artifact, and restarted the Tomcat service to reflect the changes.

10. Set Up Application Load Balancer (ALB)

    Created a target group pointing to the Tomcat instance on port 8080 and attached it to a newly created ALB. Enabled both HTTP/HTTPS (if SSL cert is present). The ALB DNS name served as the public entry point.

11. Test Application via Load Balancer

    Verified successful deployment by accessing the app through the ELB DNS URL. Ensured all backend services were communicating correctly through configured hostnames.

12. Configure Auto Scaling with Launch Template

    Created an AMI from the configured Tomcat instance, set up a launch template with required settings and IAM role, and finally created an Auto Scaling Group to automatically manage Tomcat app server instances.

For now, I skipped purchasing a domain name and configuring the ELB DNS name with it. Doing so would have given this project a more realistic approach, but that can be done in future projects.

Resources I used-

1. Storage Services | AWS Documentation

2. RDS Instance | geekforgeeks

3. DevOps Course | Udemy

Challenges I faced-

1️⃣ Difficulty in understanding RDS instances - I couldn’t grasp the concept of RDS instantly. I learned how to create an RDS instance and how to modify/delete it, but I have to figure out how to implement it.

Solution - I will try to create a mini project to implement and understand the flow of how RDS is used.

2️⃣ SSL-related error mid-upload when using the AWS CLI - When I was trying to copy the artifact from the local system to the S3 bucket using AWS CLI, I was getting an SSL-related error, and my connection with the S3 bucket kept getting interrupted. I thought it was a Git Bash issue as I was using Bash. I tried using PowerShell but kept getting the same error. Solution- As the connection was getting interrupted during the artifact upload, I came to know that it could be a network-related issue. So I changed the wifi from my hostel wifi to my personal hotspot, and now there was no error, and the artifact was uploaded successfully.

What’s Next?

From next week, my end-of-semester exams are going to be held, so it would be difficult to continue learning new services and publish articles. So, I will go through all the AWS services I have learnt till now and revise them

Let’s Connect!

🔗 My LinkedIn 🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I’d love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Hello there! 👋 The past week marked Week 6 of my DevOps journey, and it was the beginning of something more hands-on and real-world focused. After completing most of the fundamental concepts, I was about to start learning Docker and Kubernetes, but a YouTube video changed the course a bit.

It is recommended learning cloud computing first, then explore containerization and deployment tools in the context of the cloud. That clicked for me. So, this week, I started exploring containerization concepts, got familiar with microservices architecture, and kicked off with AWS Fundamentals. Here's how the week unfolded:

Understanding Containerization and Docker

I began with a simple question: "What is a container?"

Containers are lightweight, portable environments that package an application and all its dependencies together.

Advantages of Containers:

• Consistency across development, testing, and production.

• Portability across platforms (Docker, Podman).

• Scalability for service expansion and contraction.

• Efficiency compared to VMs.

The most widely used tool for containerization, as we all know, is Docker. So, I learned about Docker, a platform for developing, shipping, and running applications in containers.
I understood the flow for creating a Docker container -

• Write a Dockerfile with commands like FROM, RUN, CMD, etc.

• Build a Docker Image from the Dockerfile.

• Use that image to run a Docker Container.

Docker Architecture

Docker follows a Client-Server model:

• Docker Client → CLI or GUI for user interaction

• Docker Daemon → Manages containers in the background

• Docker Registry → Stores Docker images (e.g., DockerHub)

Scenario: Nginx Web Server in Docker

I noted a simple scenario for reference: running an Nginx Web Server in Docker. It involves the following steps -

1. Install Docker on your system.

2. Pull the Nginx image** from DockerHub

    docker pull nginx
    docker run -d -p 8080:80 --name my-nginx nginx
   

   Visit http://localhost:8080 to see it in action.

3. Container Management

    docker ps
    docker stop my-nginx
    docker rm my-nginx
   

To get a hold of how Docker containers are created and run inside a VM, I set up Docker inside a VM:

1. Create a VM using Vagrant and configure it using Vagrantfile.

2. Run the VM using vagrant up, log in to it using SSH and switch to the root user.

3. Basic Commands

    systemctl status docker
    docker run hello-world
   
    docker images
    docker ps
    docker ps -a
   

4. Run a container

5. Building an image

    mkdir images
    cd images/
    vim Dockerfile
   

    FROM ubuntu:latest AS BUILD_IMAGE
    RUN apt update && apt install wget unzip -y
    RUN wget https://www.tooplate.com/zip-templates/2128_tween_agency.zip
    RUN unzip 2128_tween_agency.zip && cd 2128_tween_agency && tar -czf tween.tgz * && mv tween.tgz /root/tween.tgz
   
    FROM ubuntu:latest
    LABEL "project"="Marketing"
    ENV DEBIAN_FRONTEND=noninteractive
   
    RUN apt update && apt install apache2 git wget -y
    COPY --from=BUILD_IMAGE /root/tween.tgz /var/www/html/
    RUN cd /var/www/html/ && tar xzf tween.tgz
    CMD ["/usr/sbin/apache2ctl", "-D", "FOREGROUND"]
    VOLUME /var/log/apache2
    WORKDIR /var/www/html/
    EXPOSE 80
   

    docker build -t tesimg .
    docker images
    # Run container from the Image
    docker run -P -d tesimg
    docker ps
   
    ip addr show
    docker ps
   

6. Go to the browser and enter IP_Addr:HostPort

7. Cleaning up containers and images. This sample Dockerfile I pulled from the internet to test the creation of a Docker container.

Learning Microservices Architecture

I also learned about Microservices - what Microservices architecture is, how it is different from the previously used Monolithic architecture. The ideology of Microservices is breaking down a monolith into several smaller, self-contained services, each responsible for a specific business functionality. This introduces isolation, loose coupling and flexibility in choosing technology.

Communication Methods:

• Synchronous (APIs)

• Asynchronous (Message brokers like RabbitMQ)

• Service Mesh

I also explored:

• Monorepo vs Polyrepo code strategies

Although I stayed on a theoretical level for now, I plan to go deeper with projects soon.

Getting Started with AWS

Since cloud was the next big milestone, I turned to AWS Fundamentals. Thankfully, I had some theoretical exposure from a course I took last semester, which helped me understand things quickly.

Important AWS Services for DevOps:

➡️ Compute: EC2, Lambda, Beanstalk

➡️ Storage: S3, EBS

➡️ Identity: IAM, KMS

➡️ Monitoring: CloudWatch, CloudTrail, X-Ray

➡️ Networking: VPC, ELB, Route53

➡️ Containers: ECS, EKS, ECR

➡️ CI/CD: CodeCommit, CodeBuild, CodeDeploy, CodePipeline

Deep Dive into EC2 Instances

I created multiple EC2 instances and:

• Played with Security Groups, Key Pairs, and Elastic IPs

• Hosted a static HTML page on an Ubuntu server using Apache2

  

AWS CLI:

Next, I learned about AWS CLI, which lets users interact with AWS services from the terminal. It also helps in automation, scripting, and managing resources via the terminal.

• Installed CLI on my machine

• Created an IAM user and configured it with Access Keys

• Practiced launching instances, creating key pairs, and security groups via CLI

EBS Volumes & Snapshots

Each EC2 instance comes with a block storage volume (EBS). I learned:

• Types of EBS volumes

• Creating, attaching, and configuring them

• Creating partitions, mounting and unmounting them persistently

• Snapshots for backup and restore

Load Balancing with ELB

I created multiple EC2 instances using:

• AMIs and Launch Templates

• Hosted a basic HTML page on each

• Created Target Groups and an Application Load Balancer (ALB)

• Configured listeners and verified load distribution

Monitoring with CloudWatch & SNS Alarms

I used CloudWatch to monitor CPU usage on an EC2 instance and:

• Created SNS Topic and Subscription

• Configured Alarms to notify me via email or SMS

Faced some issues with email alerts, but SMS notifications worked as expected.

Resources I Used

1. Docker Documentation

2. Microservices | Techworld with Nana

3. AWS | Cloud Computing

4. AWS CLI | Command References

Challenges I Faced

1️⃣ Microservices Mini Project Not Working

• After learning the basics of microservices, I tried building a small demo project using a sample docker-compose.yml file I found online. My goal was to spin up a few lightweight containers as independent services inside a VM. However, the setup consistently failed to build and run. Even after debugging the Dockerfile, checking for dependency issues, and validating port configurations, I couldn’t get the containers running inside the VM. I eventually paused the project and plan to revisit it later with a more hands-on guide.

2️⃣ Security Group Misconfiguration for SSH

• While working with EC2 instances, I created a new instance and configured the inbound rules via the security group to allow SSH access. However, every time I tried to log in using my key pair, I received a "Permission denied" error. I temporarily opened the rule to allow SSH from all IPs (0.0.0.0/0), and it worked. But that’s not a secure long-term solution. I terminated the instance, recreated it with the exact same rule settings and, surprisingly, this time, SSH worked perfectly. Still unsure what caused the initial misbehavior.

3️⃣ CloudWatch Alarm Emails Not Delivering

• I wanted to test CPU utilization monitoring and email alerts using AWS CloudWatch. After configuring everything, the alarm triggered correctly, but I wasn’t receiving emails. The SNS subscription for my email kept showing as "Pending Confirmation" despite confirming multiple times. After researching, I suspected my email address might be on AWS SES's suppression list. As a workaround, I created a new subscription using the SMS protocol and tested it using my mobile number - which worked flawlessly. Still need to confirm what happened to my email with AWS support, or try a different email next time.

What's Next?

Before exams hit, I plan to:

• Learn EFS, Auto Scaling Groups, and S3

• Revise previously covered concepts

Let’s Connect!

🔗 My LinkedIn

🔗 My GitHub

If you have any helpful resources or better ways to solve the challenges I faced, or just want to chat about DevOps, drop a comment!

Have a wonderful day!

Hello there! It's been a while. I'm really sorry for not uploading anything for the past 3 weeks — life got a little hectic due to a hackathon our team participated in, mid-semester exams, and some other personal stuff. But I’ve promised myself to try my best to stay more consistent from now on.

Despite the packed schedule, I didn’t completely pause my learning. I still managed to: ✅ Build a solo Web Development project

✅ Learn more about Build and Package Manager tools

✅ Dive deeper into Vagrant and Linux servers

✅ Get started with YAML

Building a Typing Speed Test Application

Just before the exams, I was casually testing my typing speed on MonkeyType, and as I was completing test after test, I couldn't help but wonder — How exactly is typing speed, raw speed, accuracy, and errors calculated?

While I had a rough idea about these parameters, I was curious about the logic behind implementing them. So, I asked ChatGPT how these parameters are calculated, and it gave me a clear explanation of the process.

Excited, I decided to put this into practice by building a simple Typing Speed Test as a single-page application. I implemented:

• Logic to generate random words.

• Display the generated words to the user.

• Allow the user to type in an input box.

• Check each character typed against the generated words.

• Highlight errors by displaying wrong characters in red.

• Calculate typing speed dynamically.

Improving the Application

While testing, I noticed that I had to refresh the page every time I wanted to try a new test, which was quite annoying. So, I added a restart button to reset the test without refreshing.

Then I thought — why stop here?

I went on to:

• Add raw speed and accuracy calculations.

• Integrate a results section with a graph.

• Create settings to change:

  • Font size

  • Font family

  • Enable/disable typing sounds

• Implement user authentication (which turned out to be challenging).

• Develop a leaderboard.

• Build an about page.

This turned into my first solo full-stack web development project after a long time!

Tech Stack Used:

• Next.js

• TailwindCSS

• NextAuth

• Prisma

• PostgreSQL

• Shadcn UI

• Vercel (for deployment)

➡️ Check out the GitHub repository here: TypeFlow Repo

Continuing DevOps Learning: Build & Package Manager Tools

After completing my web project, I resumed my DevOps journey.

Previously, I had learned to build Java applications using Gradle. This time, I expanded my knowledge by:

• Learning how to build Python projects using setuptools.

• Setting up a setup.py file.

• Understanding what a wheel (.whl) file is.

• Exploring NPM for JavaScript projects.

• Understanding the build process of JS applications.

• Differentiating between a Library, Package, and Framework.

Deep Dive into Vagrant & Linux Servers

I had earlier used Vagrant briefly to create a VM when I first started with Linux. This time, I took it a step further.

I learned how to:

• Configure Vagrant using a VagrantFile:

  • Choose which Vagrant Box (OS) to use.

  • Allocate RAM and CPUs.

  • Automate provisioning.

• Create Synced Folders:

  • Share files between the host machine and the virtual machine.

Small Projects to Practice:

1️⃣ Hosted a basic HTML template inside a VM.
2️⃣ Hosted a WordPress configuration setup for blogging inside a VM.

For both:

• First, I performed all steps manually to fully understand each action.

• Then, I automated the entire setup using VagrantFile so that when the VM boots, everything is ready automatically.

This helped me get hands-on with:

• Provisioning

• Service management

• Hosting applications

I also explored how to provision Multi-VM setups using a single VagrantFile. This was something I had been curious about, and accomplishing it felt rewarding.

Getting Started with YAML

Today, I finally got started with YAML — a human-readable data serialization language, just like XML and JSON.

What is Data Serialization?

Serialization is the process of converting data structures into a standard format so that they can be transferred between applications or services.

What I Learned:

• Basic syntax of YAML.

• Declaring various data types and structures.

• Writing real-world examples like an Nginx web server configuration.

Sample YAML Example:

apiVersion: apps/v1            # API version used
kind: Deployment               # The type of Kubernetes object
metadata:
  name: nginx-deployment       # Name of the deployment
  labels:
    app: nginx                 # Label assigned to this deployment
spec:
  replicas: 3                  # Number of pod replicas
  selector:
    matchLabels:
      app: nginx               # Must match pod labels
  template:
    metadata:
      labels:
        app: nginx             # Label assigned to pods created by this deployment
    spec:
      containers:
        - name: nginx          # Container name
          image: nginx:latest  # Docker image
          ports:
            - containerPort: 80 # Port exposed inside the container

Resources I Used

1. Install and Configure WordPress | Ubuntu

2. VagrantFile | Vagrant | HashiCorp Developer Docs

3. YAML Tutorial | TechWorld with Nana

(Note: This does not include resources I used for the web development project.)

Challenges I Faced

1️⃣ Authentication Struggle

Initially, I wanted to learn and use Supabase as my database and authentication system. However, I found it difficult to work with and eventually dropped it.

✅ Solution: Switched to NextAuth for authentication and used NeonDB instead of Supabase.

2️⃣ Provisioning WordPress Setup

I couldn't get the WordPress setup provisioned correctly at first.

✅ Solution: Restarted the whole process and fixed some typographical mistakes which solved the problem.

What’s Next?

• Build a project where I actually use YAML in a real-world scenario.

• Get started with Containers.

Let’s Connect!

🔗 My LinkedIn

🔗 My GitHub

If you have any recommended resources, better approaches to the challenges I faced, or just want to share some insights — I'd love to hear from you! Drop your thoughts in the comments.

Hello there! 👋 This past week marked the fourth week of my DevOps journey. While I was a bit inconsistent and sluggish, I still managed to learn some exciting and essential concepts. Initially, my plan was to start learning Python, but I realized it would be better to push it down the roadmap and focus on some other important tools first.

This week, I:
- Completed another mini Bash scripting project
- Learned about Makefiles
- Explored SSH login & SSH keys
- Dived into advanced Git features
- Revised API development in Next.js
- Started learning Build Tools, mainly Gradle

I'll share everything I learned, the resources I used, and the challenges I faced. Let's get started.

Automating AWS S3 Bucket Creation with Bash

Last week, I completed a mini project in which I wrote a script to create an EC2 instance on AWS and automate the AWS configuration. Although I modified the script and learned a lot, I wanted to build something similar but completely on my own.

So, I decided to write a Bash script to create an S3 bucket with default configurations on AWS. I reused some functions from my previous EC2 script, such as AWS CLI installation, and then searched for AWS CLI commands to create an S3 bucket.

Creating an S3 Bucket

The commands for creating an S3 bucket were simpler compared to EC2 instance creation. Here’s the function I wrote:

create_s3_bucket() {
    echo "Creating S3 bucket with default name as 'shell-scripting-demo'..."
    BUCKET_NAME="shell-scripting-demo"

    aws s3 mb s3://$BUCKET_NAME --region $AWS_DEFAULT_REGION

    if aws s3api head-bucket --bucket "$BUCKET_NAME" 2>/dev/null; then
        echo "S3 bucket created successfully!"
    else
        echo "There was an error creating S3 bucket, exiting the process..."
        exit 1
    fi
}

Copying an Item into the S3 Bucket

To add more challenge, I also wrote a function to upload a sample file to the newly created bucket:

copy_item_in_bucket() {
    echo "Copying sample item to S3 bucket..."
    local ITEM="/home/vagrant/DevOps-Learning/shell-scripting/script-projects/todo.txt"

    aws s3 cp $ITEM s3://$BUCKET_NAME

    local KEY=$(aws s3api list-objects-v2 --bucket $BUCKET_NAME --query "sort_by(Contents, &LastModified)[-1].Key" --output text)

    if [ -n "$KEY" ]; then
        echo "Item copied successfully! Bucket name: $BUCKET_NAME S3 item Key: $KEY"
    else
        echo "Item couldn't be copied to $BUCKET_NAME bucket"
        return 1
    fi
}

One tricky part was checking whether the item was successfully copied. Initially, I used the prefix keyword, but it was prone to errors as multiple files could have the same starting name (as mentioned on Stack Overflow). Using the last modified sorting provided a better approach.

Learning Makefiles

After completing my mini Bash scripting project, I started learning about Makefiles. I had briefly encountered them during my Linux fundamentals, but I hadn't explored them in depth.

What is a Makefile?

A Makefile is a simple text file that controls a project's build process. It is used to automate tasks in C, C++, and many other languages.

Why Use Makefiles?

• Automates repetitive tasks like compiling code, running tests, and installing dependencies.

• Simplifies command execution by defining custom rules.

• Speeds up builds by recompiling only changed files.

I explored the syntax and practiced by creating a simple Makefile to automate a small workflow.

Exploring SSH & SSH Keys

When I first learned Linux fundamentals, I didn’t fully grasp SSH (Secure Shell), so I revisited it this week. SSH is a protocol used for secure remote server access, file transfers, and system administration.

To practice SSH, I created an EC2 instance to simulate a remote server.

Methods to Login into a Remote Server I learned:

1️⃣ Password Authentication
2️⃣ SSH Keys

Generating an SSH Key Pair

The SSH key method doesn’t require passwords. Here’s how I generated an SSH key pair:

ssh-keygen -t rsa -b 4096

• This command creates two keys: a public key (stored on the server) and a private key (stored on the client).

• To log in, the public key is matched with the private key using:

ssh user@host_ip_addr

This ensures secure authentication without requiring passwords.

Getting Started with Gradle & Advanced Git Features

I took some time to understand what build tools are and why they are essential in software development. Build tools help automate the process of transforming source code into executable applications by handling tasks like:

• Compiling: Converting source code (C, Java, etc.) into machine code or bytecode.

• Linking: Combining multiple compiled files into a single executable.

• Dependency Management: Ensuring all required libraries are available.

• Testing & Packaging: Running unit tests and packaging the application for distribution.

Common Build Tools

After understanding these concepts, I started with one of the most popular build tools for Java—Gradle.

I started with Gradle, one of the popular build tools for Java, Android, and Kotlin projects. I explored:
→ Gradle Project Structure - Understanding files and directories in a basic project.
→ Gradle Tasks & Wrapper - How automation works within a Gradle project.
→ Running a small Java application using Gradle.

To practice, I got the code of a Java project that prints "Hello" or "Hola" based on an argument. I initially did all this on my Ubuntu dual-boot setup, but I ran into system crashes and memory issues, even though I had allocated 20GB of space. Eventually, I had to remove Ubuntu and switch back to Windows.

After tackling the basics of Gradle, I spent a few hours learning advanced Git features like:

• Interactive Rebase

• Reflog

• Search & Find

• Submodules

• Cherry-picking

Since I had been focused on DevOps, I hadn't done much Web Development recently. To refresh my skills, I revised API development in Next.js, covering:

• CRUD operations

• Query parameters

• Dynamic routes

• Middleware concepts

Resources I used this week-

1. AWS CLI Documentation

2. Awesome Make GitHub Repo

3. Learn Linux TV | OpenSSH Tutorial

4. Tom Gregory | Gradle Tutorial

5. freecodecamp Advanced Git Tutorial

6. freecodecamp | Rest APIs with Next.js 14

Challenges I Faced This Week

1️⃣ Checking if an Item Was Copied in an S3 Bucket

• Issue: Initial method using prefix keyword was unreliable.

• Solution: Used last modified sorting for accuracy.

2️⃣ SSH Login Issues

• Issue: ssh username@server_ip was failing despite enabling PasswordAuthentication.

• Solution: Couldn’t find a solution, so I switched to SSH keys, which worked perfectly.

3️⃣ Ubuntu Performance Issues

• Issue: Ubuntu wasn’t running smoothly, even with 20GB allocated space.

• Solution: Couldn’t find a way to increase disk space, so I removed Ubuntu and switched back to Windows.

What’s Next?

Next week, I plan to:
✅ Explore more Build Tools & Package Managers
✅ Learn about Artifacts & Artifact Repository Management

Let’s Connect!

🔗 My LinkedIn 🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I’d love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Hello there! 👋 This past week marked the third week of my DevOps journey, and it has been an exciting and challenging one. I dived deep into how DNS works, the TCP/IP model, Bash scripting, and built some projects along the way. I also faced my fair share of errors (as expected), but overcoming them was a great learning experience. Let's go through each topic in detail!

Exploring the Depths of Networking

Last week, I covered the basics of computer networks, including the OSI model and network topologies. This week, I explored:

1. How DNS Works

The Domain Name System (DNS) is what helps map human-readable domain names (like google.com) to IP addresses that machines understand. Learning about DNS helped me understand:

• The role of DNS resolvers, root servers, TLD servers, and authoritative name servers.

• How requests flow when a user enters a URL in the browser.

  

2. TCP/IP Model

Unlike the theoretical OSI model, the TCP/IP model is the real-world networking framework we use every day. It consists of the following layers: 1️⃣ Network Access Layer (Link Layer) – Deals with physical transmission of data.

2️⃣ Internet Layer – Responsible for addressing and routing packets using protocols like IP.

3️⃣ Transport Layer – Manages end-to-end communication using protocols like TCP and UDP.

4️⃣ Application Layer – Interfaces with user applications like HTTP, SMTP, and FTP.

Some resources describe the TCP/IP model as 5 layers, merging the Data Link Layer and Physical Layer into one. Learning about this model made it easier for me to grasp how data packets travel across networks and how routing works.

3. Forward and Reverse Proxy

I also explored proxy servers and how they act as intermediaries between clients and the internet:

• Forward Proxy: Sits between the client and the web, helping in hiding identity, caching content, and filtering traffic.

• Reverse Proxy: Sits between the web server and users, mainly used for load balancing, security, and caching.

A widely used example of a Reverse Proxy is Nginx, which I plan to explore in-depth soon.

Resources I Used for Networking:

1. Networking Basics - YouTube

2. GeeksforGeeks - DNS System

3. TCP/IP Explained - YouTube

Getting Started with Bash Scripting

After wrapping up Networking, I jumped into Bash scripting. The first thing I did was understand what Bash is and how it differs from other shells. Then, I got hands-on by writing my first script:

#!/bin/bash
# My first Bash script

echo "This is my first script as a DevOps Student"
echo "Lionel Andres Messi"
echo "GOAT"

A bit embarrassing, but let's move past that. 😅

I then learned about:

• Variables & Comments

• If statements & Loops

• Exit Codes & Functions

• Command-line arguments & Case statements

To practice Bash scripting, I built some small projects like:
✅ A simple to-do list script that adds tasks to a file.
✅ A basic calculator script for performing arithmetic operations.

First Project: Deploying a Django App using Docker & Nginx

Even though I had no prior experience with Docker and Nginx, I followed along with a tutorial where the instructor explained each script, ran it, and debugged errors. But while following the tutorial, I encountered an unexpected error:

error checking context: no permission to read from '/home/vagrant/DevOps-Learning/shell-scripting/script-projects/django-notes-app/data/mysql/db/#ib_16384_0.dblwr'
Failed to build and deploy the app.

Solution:

• The error occurred because Docker didn’t have permission to access the MySQL folder in the cloned repo.

• I used the chown command to give the current user ownership of the folder.

• Then, I modified the permissions of the cloned repo.

This allowed me to successfully deploy the app using Docker Compose & Nginx. However, since I was running my Ubuntu machine in a VM, I couldn't access localhost:8000 directly. Instead, I checked my VM’s IP using ip addr show and accessed the port via that IP.

Second Project: Creating an AWS EC2 Instance using Bash

This project aimed to automate the process of creating an AWS EC2 instance using Bash scripting. I was particularly excited about this project as it gave me a real-world perspective on automation and infrastructure provisioning.

Step 1: Checking AWS CLI Installation

Before proceeding with creating an instance, I needed to ensure that AWS CLI was installed. To check this, I wrote the following function:

check_awscli() {
    if ! command -v aws &> /dev/null; then
        echo "AWS CLI is not installed. Installing it now..." >&2
        return 1
    fi
}

If AWS CLI wasn’t installed, I wanted the script to handle the installation automatically.

Step 2: Installing AWS CLI

To automate the installation of AWS CLI on a Linux system, I wrote this function:

install_awscli() {
    echo "Installing AWS CLI v2 on Linux..."

    # Download and install AWS CLI v2
    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
    sudo apt-get install -y unzip &> /dev/null
    unzip awscliv2.zip
    sudo ./aws/install

    # Verify installation
    aws --version || {
        echo "AWS CLI installation failed. Please install manually."
        exit 1
    }

    # Clean up
    rm -rf awscliv2.zip ./aws

    echo "AWS CLI Installation Completed."
}

This function downloads and installs AWS CLI, verifies the installation, and cleans up unnecessary files.

Step 3: Creating the EC2 Instance

Once AWS CLI was installed, I wrote a function to create an EC2 instance using AWS CLI commands:

create_ec2_instance() {
    local ami_id="$1"
    local instance_type="$2"
    local key_name="$3"
    local subnet_id="$4"
    local security_group_ids="$5"
    local instance_name="$6"

    # Run AWS CLI command to create EC2 instance
    instance_id=$(aws ec2 run-instances \
        --image-id "$ami_id" \
        --instance-type "$instance_type" \
        --key-name "$key_name" \
        --subnet-id "$subnet_id" \
        --security-group-ids "$security_group_ids" \
        --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=$instance_name}]" \
        --query 'Instances[0].InstanceId' \
        --output text
    )

    if [[ -z "$instance_id" ]]; then
        echo "Failed to create EC2 instance." >&2
        exit 1
    fi

    echo "Instance $instance_id created successfully."

    # Wait for the instance to be in running state
    wait_for_instance "$instance_id"
}

This function takes multiple parameters such as AMI ID, instance type, key name, subnet ID, security group ID, and instance name. After executing the AWS CLI command to create an instance, it waits for the instance to be fully initialized.

Step 4: Waiting for Instance to Run

Once the EC2 instance is created, the script waits until the instance reaches the running state:

wait_for_instance() {
    local instance_id="$1"
    echo "Waiting for instance $instance_id to be in running state..."

    while true; do
        state=$(aws ec2 describe-instances --instance-ids "$instance_id" --query 'Reservations[0].Instances[0].State.Name' --output text)
        if [[ "$state" == "running" ]]; then
            echo "Instance $instance_id is now running."
            break
        fi
        sleep 10
    done
}

Step 5: Automating AWS Configuration

A key realization was that AWS CLI requires manual configuration before usage, which defeats the purpose of full automation. I discovered that AWS credentials are stored in .aws/config and .aws/credentials, so I automated this process:

configure_aws() {
    local aws_access_key="$AWS_ACCESS_KEY_ID"
    local aws_secret_key="$AWS_SECRET_ACCESS_KEY"
    local aws_region="$AWS_DEFAULT_REGION"

    # Ensuring the .aws directory exists
    mkdir -p ~/.aws

    cat > ~/.aws/credentials <<EOL
[default]
aws_access_key_id = $aws_access_key
aws_secret_access_key = $aws_secret_key
EOL

    cat > ~/.aws/config <<EOL
[default]
region = $aws_region
output = json
EOL

    echo "AWS CLI configured successfully."
}

This function pulls AWS credentials and region information from environment variables and writes them into the required configuration files. This ensures that AWS CLI is ready to use without manual intervention.

Step 6: Bringing It All Together

Finally, I wrapped everything into a main function and executed the script:

main() {
    check_awscli || install_awscli

    configure_aws || {
        echo "AWS Configuration failed, exiting..."
        exit 1
    }

    echo "Creating EC2 instance..."

    # Specify the parameters for creating the EC2 instance
    AMI_ID="ami-04b4f1a9cf54c11d0"
    INSTANCE_TYPE="t2.micro"
    KEY_NAME="$AWS_KEY_NAME"
    SUBNET_ID="$AWS_SUBNET_ID"
    SECURITY_GROUP_IDS="$AWS_SECURITY_GROUP_IDS"
    INSTANCE_NAME="Shell-Script-EC2-Demo"

    # Call the function to create the EC2 instance
    create_ec2_instance "$AMI_ID" "$INSTANCE_TYPE" "$KEY_NAME" "$SUBNET_ID" "$SECURITY_GROUP_IDS" "$INSTANCE_NAME"

    echo "EC2 instance creation completed."
}

main "$@"

After writing and testing this script, I was thrilled to see everything working smoothly on the first run! This was a huge confidence booster as I managed to:

• Automate the installation of AWS CLI

• Configure AWS CLI without manual input

• Create an EC2 instance automatically

• Ensure the instance reaches a running state

This project helped me understand how real-world automation works in DevOps.

Resources I Used for Bash Scripting:

1. Learn Linux TV | Bash Scripting

2. Shell Scripting in One Shot | Train With Shubham

Challenges I Faced

1️⃣ Understanding Data Flow in Networking

• Problem: Couldn't grasp how data flows across layers in networking models.

• Solution: Asked my roommate (who studied CN), and he explained it with real-world examples.

2️⃣ Errors in Django Deployment

• Solution: Used Stack Overflow & ChatGPT to debug errors, modifying file permissions & user ownership.

3️⃣ Automating AWS CLI Configuration

• Solution: Discovered where AWS stores credentials and wrote a script to automate the process.

What’s Next?

Next week, I plan to:
✅ Explore more Bash scripting projects
✅ Learn Makefile for automation
✅ Revise Python basics & explore advanced Python for DevOps

Let’s Connect!

LinkedIn: My LinkedIn Profile GitHub: My GitHub Profile
If you have any recommended resources, better approaches to my challenges, or insights, I’d love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Hello there! 👋 Last week marked Week 2 of my DevOps journey. Due to my sports commitments, I had limited time, but I still managed to learn and explore several key topics. This week, I completed advanced Linux commands, Git + GitHub, Computer Network fundamentals, and a bit about Open Source contributions. Though progress was a bit slow, learning something new always counts!

Wrapping Up Linux: Advanced Commands

In my previous week, I covered basic Linux commands, but a few advanced commands were still left. I focused on find, grep, awk, and sed:

➡️ grep - Used to search for a particular pattern in a file.
➡️ find → Searches for files and directories based on name, size, type, etc.
➡️ awk → A scripting language mainly used for text processing, filtering, and formatting structured data.
➡️ sed → A stream editor that works on character streams for pattern matching and replacements.

Now that I have a solid grasp of Linux basics, I plan to deepen my understanding by practising more with real-world assignments and tasks.

Deep Dive into Git and GitHub

As a web developer, I had experience using Git and GitHub, but for DevOps, I needed to expand my knowledge. I revised Git basics and moved to advanced topics like branches, pull requests, and resolving merge conflicts. Learning best practices for branch creation and conflict resolution helped me understand how teams collaborate efficiently in a DevOps environment.

While learning GitHub, I stumbled upon Open Source Contributions and decided to explore how Open Source projects work and how to contribute.

Getting Started with Open Source Contributions

I watched several videos and read about how to contribute to Open Source projects. Here’s the step-by-step process I learned:

1. What is Open Source?

Open source refers to any program whose source code is publicly available for modification, contribution, and improvement.

2. Finding a Repository to Contribute

Use GitHub/GitLab search to find a project related to your preferred tech stack.

3. Finding the Right Issue

Start with "good first issues" in the issues tab of a repository. These are beginner-friendly and help you get started.

4. Understand and Validate the Issue

Before contributing, ensure you understand the issue and check the discussion to validate its relevance.

5. Fork and Start Contributing

Since you don’t have access to the original repository, fork (copy) the repo to your account and start working on your own copy.

6. Create Relevant Branches

Never work in the main or master branch. Follow branch naming conventions:

• Bug fixes → fix/<bug_name>

• New Features → feat/<feature_name>

7. Test Before Creating a Pull Request (PR)

Before submitting a PR, ensure the changes work correctly and pass all required tests.

8. Link PR to Issue

Mention the issue number in your PR description:

• fixes: #issue_number

• resolves: #issue_number

9. Wait for the Maintainer’s Review

Be patient while the project maintainer reviews your PR. Engage in discussions if needed.

10. Repeat

Find another issue and continue contributing!

This was the step-by-step guide provided in the YouTube video.
I plan to find a suitable repository and make my first Open Source contribution in the coming days.

Resources I Used:

• Git/GitHub → Kunal Kushwaha - Complete Git and GitHub Tutorial

• Open Source Contributions → Piyush Garg (Open Source Bootcamp)

Learning Computer Networking Fundamentals

After exploring Open Source, I returned to my roadmap and focused on Computer Networking. I started with:

1. History of the Internet

Understanding why and how the internet was created helped me appreciate its evolution and role in modern computing.

2. Network Protocols

I explored different protocols, including HTTP, its methods, and status codes.

3. Network Architectures

I studied two primary architectures:

• Client-Server Architecture

• Peer-to-Peer Architecture

4. Network Topologies & Types

I learned about LAN, MAN, WAN and different network topologies used in enterprise environments.

5. OSI Model

One of the major topics I explored was the OSI (Open Systems Interconnection) Model, which consists of 7 layers:

• Application Layer

• Presentation Layer

• Session Layer

• Transport Layer

• Network Layer

• Data Link Layer

• Physical Layer

  

Each layer has specific responsibilities and protocols that ensure seamless data transfer over networks. I also studied how emails work and the different protocols used in email communication.

Resources I Used:

• Computer Networking → Kunal Kushwaha - Computer Networking Full Course

• GeeksforGeeks Networking Articles

Challenges I Faced & How I Overcame Them

1. Understanding Differences Between grep, awk, and sed

   🔹 Problem: I was confused about when to use each command.

   🔹 Solution: I searched for real-world examples and experimented with them.

   🔹 Resources: Baeldung - grep, sed, awk differences, Linode - awk, grep, sed differences

2. Getting Started with Open Source Contributions

   🔹 Problem: I can't determine a starting point for Open Source Contribution.

What’s Next?

For next week, my goals are:
✅ Understanding how DNS works
✅ Deep dive into each OSI model layer
✅ Getting started with Bash Scripting

Let’s Connect!

If you have any recommended resources, some better approaches to challenges I faced, insights, or tips, I’d love to hear from you! Drop them in the comments. Thank you for reading till the end!

Have a wonderful day.

Hello there! 👋 Last Sunday, I officially began my DevOps journey, and this past week has been an exciting deep dive into its fundamentals. I started by understanding what DevOps is, its ideology, core concepts, and terminologies. Alongside this, I ventured into the world of Linux—exploring installation methods, different Linux distributions, and, most importantly, learning essential commands. In this blog, I'll share my learnings, challenges, and how I overcame them.

Understanding DevOps: Beyond Just a Definition

When I first asked myself, "What is DevOps?", the common definition I found was:

DevOps is a set of practices that integrates software development (Dev) and IT operations (Ops) to shorten the development lifecycle while maintaining high software quality.

However, I quickly realized that DevOps is much more than a set of practices; it is a philosophy. It encourages collaboration between development and operations teams, reducing friction and accelerating code deployment into production.

The Software Development Lifecycle (SDLC)

I also came across the Software Development Life Cycle (SDLC), a structured process to develop software efficiently. It consists of:

• Planning → Identifying requirements and setting goals.

• Design → Creating architecture and technical specifications.

• Development → Writing the actual code.

• Testing → Ensuring the software works as expected.

• Deployment → Releasing the software into production.

• Operate & Monitor → Maintaining performance and addressing issues.

Continuous Integration & Continuous Deployment (CI/CD)

One of the most fascinating concepts I encountered was CI/CD:

➡️ Continuous Integration (CI) ensures that developers frequently merge their code into a shared repository, where it is automatically tested and built.

➡️ Continuous Deployment (CD) automates the process of releasing tested code, deploying it to production, and monitoring it for issues.

The main goal of CI/CD is to reduce human intervention, speed up deployment, and enhance code quality, thus minimizing the risk of large-scale failures.

Deployment Strategies

I also learned about different strategies used in production environments:

➡️ Rolling Deployment: Gradually replacing the old version of an application with a new one.

➡️ Blue/Green Deployment: Running two identical environments—one (blue) with the existing version and another (green) with the new version. Traffic is switched once testing is successful.

Other important concepts I explored include Virtual Machines (VMs), Containers, Linters, Log Aggregation, and Production Metrics.

Resources I Used for DevOps Basics:

• TechWorld with Nana: What is DevOps? REALLY Understand It | DevOps vs SRE

• freeCodeCamp.org: DevOps Engineering Course for Beginners

• GeeksforGeeks: What is DevOps?

After two days of DevOps basics, I moved on to Linux, and honestly, I was a bit nervous to start. But here’s how it went😅.

Getting Started with Linux

To begin my Linux journey, I first understood what Linux is:

Linux is an open-source operating system widely used for servers, development, and personal computing due to its security, flexibility, and stability.

I came across essential Linux components:

➡️ Kernel: The core of the OS, managing hardware-software interactions.

➡️ Bootloader: Loads the operating system into memory when the computer starts.

➡️ Shell: An interface to interact with the OS via commands.

Running Linux on My Windows System

Among various methods, I chose Virtual Machines (VMs). I installed VirtualBox and configured a Linux server. Later, I explored Vagrant, which automates VM creation and makes setup much easier.

Essential Linux Commands I Learned

I started with basic navigational commands:

➡️ pwd, ls, cd – for navigating the file system.

➡️ File management commands: touch, echo, cat, cp, mv, rm, ls.

➡️ System commands: df, du, top, ps, fuser, vmstat.

➡️ The fascinating nohup command—used to run processes even after system shutdown and store process logs.

Next, I explored User & Group Management:

➡️ Creating, deleting, switching users (useradd, userdel, su).

➡️ Managing groups (groupadd, groupdel, usermod).

➡️ File permissions using chmod and the importance of ****sudo.

I also tackled Network Commands, including ping, netstat, ip, traceroute, mtr, dig, curl, and wget. These were confusing at first, but practice made them clearer.

Resources I Used for Linux:

• TrainWithShubham: Linux For DevOps In One Shot | Beginners to Advanced

• Kunal Kushwaha: Introduction to Linux & Terminal Commands - Full Course

Challenges I Faced & Solutions

1. File Permission Issue in Vagrant

🔹 Problem: Initially, I couldn’t change file permissions inside the /vagrant directory. I learned that Vagrant uses VirtualBox's default settings, which override permission changes.

🔹 Solution: /vagrant is a shared folder between the host and VM, so permission changes were overridden. I either had to modify Vagrantfile settings or work in /home/vagrant, which resolved the issue.

2. Struggles with User & Group Management Commands

🔹 Problem: While I could handle basic file operations, user management commands were tricky.

🔹 Solution: I practised more real-world examples and used ChatGPT for additional scenarios to get a deeper understanding.

3. Network Commands Were Overwhelming

🔹 Problem: Too many commands at once felt confusing, and many seemed to have similar outputs.

🔹 Solution: I took a break, revised, and executed each command sequentially to observe differences. Practice is key!

What’s Next?

This past week has been exciting, challenging, and deeply rewarding. For the next week, I plan to:

✅ Complete advanced Linux commands (grep, awk, etc.)
✅ Dive deeper into Git and GitHub
✅ Begin learning Networking Fundamentals

I know the basics of Git and GitHub, but this time, I aim to go in-depth and explore real-world workflows.

Let’s Connect!

If you have any recommended resources, insights, or tips, I’d love to hear from you! Drop them in the comments. Thank you for staying with me till the last.

Have a wonderful day.

Hello there!👋 My name is Akshansh Singh, and I am a pre-final year engineering student at VIT Chennai. Over the past 2.5 to 3 years, I have been deeply involved in web development, building projects that not only enhanced my skills but also made the learning process enjoyable. Apart from web development, I have always been curious about new technologies and trends in the tech world, keeping myself updated through various articles and blogs. It was through web development that I first encountered the concept of deploying applications, which eventually led me to explore the vast domain of DevOps.

How I Discovered DevOps

Initially, I deployed my projects using platforms like Vercel, Netlify, and Render. While these platforms made deployment relatively easy, I faced a major challenge when deploying one of my projects, CodeChime—a WebSocket-based code editor. Everything worked perfectly in my local development environment, but as soon as I moved to production, things started breaking. It took me almost two to three days to identify and fix the issue. That was when I started wondering how large enterprises manage to deploy complex applications at scale without facing such problems.

To satisfy my curiosity, I began reading about how modern applications are built, tested, released, and deployed efficiently. That was my first real introduction to DevOps. As I delved deeper, I realized that DevOps is not just about deployment—it is an entire ecosystem of processes and tools designed to streamline software development and operations. I read more about its importance in the tech industry, the career opportunities it offers, and the skill set required to become a DevOps engineer. The more I explored, the more I became fascinated by the field.

My Approach to Learning DevOps

Once I decided to take DevOps seriously, my initial thought was to follow a structured course. I spent three to four days searching for online courses but was disappointed to find that free courses lacked depth, while the paid ones were too expensive. That’s when I decided to take a different approach—designing my own roadmap for learning DevOps.

Having previously followed roadmap.sh for web development, I turned to their DevOps roadmap. It gave me a broad overview of the necessary concepts and tools, helping me identify where to start. To supplement my learning, I spent an entire day watching YouTube videos and came across insightful content from Kunal Kushwaha and savinder puri. Their videos introduced me to the idea of learning in public—sharing my progress, experiences, and knowledge with others through blogging. This concept resonated with me, and I saw it as an excellent opportunity to document my journey while engaging with the DevOps community. ✨

Why Learn in Public?

The idea of learning in public appealed to me for several reasons:

✅ It helps reinforce my understanding of concepts by articulating them in my own words.

✅ It provides an opportunity to receive feedback from experienced professionals who can guide me in the right direction.

✅ It allows me to contribute to the DevOps community by sharing insights that might help other beginners like me.

Inspired by what I learned, I decided to start blogging on Hashnode as a way to track my progress and engage with the DevOps community.

My DevOps Learning Plan

To ensure a structured and practical approach to learning DevOps, I have outlined a plan that includes both theoretical understanding and hands-on practice. Here’s what I intend to do:

➡️ Understanding DevOps Fundamentals – Before diving into tools, I want to grasp the ideology behind DevOps. What problems does it solve? How does it improve the development lifecycle? What does a DevOps engineer do in real-world scenarios?

➡️ Learning Linux and Shell Scripting – Since Linux is the foundation of many DevOps tools, I will start with its basics, including important commands, file handling, process management, and shell scripting.

➡️ Version Control with Git – While I am already familiar with Git from web development, I will explore advanced workflows, branching strategies, and collaboration techniques.

➡️ Networking Fundamentals – Understanding the basics of networking, including protocols, DNS, load balancing, and security aspects.

➡️ Cloud Services Fundamentals – Learning the core services of AWS, including EC2, S3, IAM, and VPC to understand cloud infrastructure.

➡️ Containerization with Docker – Containers play a significant role in DevOps. Learning Docker will help me understand how to package applications efficiently.

➡️ CI/CD Pipelines – Setting up Continuous Integration and Continuous Deployment pipelines using tools like Jenkins, GitHub Actions, or GitLab CI/CD.

➡️ Infrastructure as Code (IaC) – Learning about Terraform and Ansible to automate infrastructure deployment and management.

➡️ Monitoring and Logging – Exploring tools like Prometheus, Grafana, and ELK stack to ensure system reliability and performance monitoring.

This roadmap will serve as my guide, but I also expect to adapt and refine it based on my experiences along the way.

What to Expect from My Blog

Starting today, I will be sharing my weekly learnings and experiences on Hashnode. My blogs will cover:

• What I learned – Concepts, tools, and best practices.

• Why I learned it – The importance of each topic in the DevOps workflow.

• How I learned it – Resources, challenges faced, and practical applications.

I also plan to work on small projects to solidify my understanding and validate my learning. These projects will help me implement DevOps principles in real-world scenarios and build a strong portfolio.

Join Me on This Journey

I believe this will be an exciting and rewarding journey, and I welcome anyone interested in DevOps to follow along. Whether you’re an experienced professional, a beginner like me, or someone curious about DevOps, your insights, feedback, and suggestions will be highly valuable.

Feel free to connect with me on LinkedIn or check out my GitHub profile. If you have any recommended resources, tips, or experiences to share, I would love to hear from you in the comments. 💡

Have a wonderful day.