Mastering CI/CD with Jenkins: Building and Deploying Applications with Efficiency
In this comprehensive article, we will dive into the world of Jenkins, covering everything from the fundamentals to advanced concepts, all while integrating Docker and Docker Compose for efficient and scalable application deployments. With step-by-step instructions and practical examples, you'll gain a better understanding of each topic.
We'll begin by exploring the core concepts, such as CI/CD pipelines and GitHub webhooks, ensuring you have a solid foundation to build upon. From there, we'll delve into the prerequisites of Jenkins, uncovering the requirements for successful project implementations. You'll learn how to create a GitHub access token and install Jenkins, Java, and Docker using detailed instructions and valuable reference URLs.
As we progress, we'll guide you through the process of creating freestyle projects on Jenkins, generating SSH key pairs, establishing connections to Jenkins servers, and setting up credentials for secure and seamless workflows. You'll also gain expertise in writing Dockerfiles and Jenkins scripts to automate the building and deployment of your applications.
One of the highlights of this article is our hands-on exploration of automating deployment using GitHub webhooks. Discover how to create webhooks and leverage their power to trigger Jenkins pipelines seamlessly. We'll guide you through running applications using Docker Compose on Jenkins servers, ensuring your deployments are efficient and reliable.
What is Jenkins and how does it work?
Jenkins is an open-source automation server used for continuous integration and continuous delivery (CI/CD) processes. It allows developers to automate the building, testing, and deployment of software applications. Jenkins works by integrating with version control systems, executing predefined build scripts or tasks, and providing feedback on the build status.
What is the CICD pipeline?
CI/CD (Continuous Integration/Continuous Deployment) pipeline is a set of automated processes that helps to integrate code changes, build, test, and deploy applications continuously. The primary goal of a CI/CD pipeline is to enable fast and reliable delivery of changes to production.
A typical CI/CD pipeline consists of several stages, including:
Code Integration: In this stage, developers integrate their code changes into a shared repository.
Build: In this stage, the CI system builds the code and runs any necessary tests.
Test: In this stage, the code is tested using various testing techniques, including unit tests, integration tests, and end-to-end tests.
Deployment: In this stage, the code is deployed to production or a staging environment.
Monitoring: In this stage, the system monitors the deployed application for performance and stability.
Github WebHooks
GitHub Webhooks are a feature of the GitHub platform that allow developers to receive notifications about events that occur in a GitHub repository.
Webhooks are HTTP callbacks that are triggered by specific events in a repository, such as a new commit, pull request, or issue being created or updated.
They provide a way to integrate external systems or services with GitHub and automate workflows based on repository activity.
When an event occurs, GitHub sends a POST request to a specified URL (endpoint) configured by the developer, containing information about the event.
Developers can secure their webhooks by using a secret token to verify the authenticity of incoming requests, ensuring that only valid requests are processed.
Tools we will be using in the project:
AWS EC2
GitHub
Docker
Jenkins
Below are the prerequisite →
Git Understanding
GitHub Account
EC2 instance on AWS
GIT Understanding
Docker understanding
Let's start with hands-on experience for better understanding...
Step-1. Fork Project to your GitHub account if you do not have your application.
You can use this todo app in case you do not have any projects.
https://github.com/nahidkishore/node-todo-cicd
Now, Let's create a token to access this repository. You can create a token just to follow the below steps;
Click on your GitHub Profile then click on Settings
Go to Developer Settings
Click on Personal Access Token
Click on Token(Classic)
Click on Generate New Token
Click on Token(Classic) again
It will ask Password , Put your password
Git Note description
Provide Expiration time as per your convenience
Check the Select scopes as per the requirement.
I checked scope : repo,workflow
Now click on Generate Token
You can see this type of message “ Make sure to copy your personal access token now. You won’t be able to see it again! “
So, Copy that token and save it at your safest place because you cannot find it again.
Step-2 : Let get EC2 Ready and Install Jenkins.
I hope you know how to create an EC2 instance so I am not explaining that steps here. Still need help then you can see my previous blogs, where i am share how to create ec2 instance.
After creating an Instance here we are connected to it via ssh.
Now we will install Jenkins on this EC2 instance and below are the commands list.
sudo apt update
sudo apt install openjdk-11-jre
java --version
curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins
Now we will install Jenkins on this EC2 instance and below are the commands.
sudo apt update
Jenkins needs the Java runtime as it is based on Java. Install open JDK 11 package using the following command.
sudo apt install openjdk-11-jre
Now let's check the Java version if it is installed or not.
java --version
Add the Jenkins Debian repo and add a Jenkins apt repository entry:
curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null
Note: Maybe when you come to my this blog above details may change so if you are facing any issues please refer to the official doc for the same.
You can install jenkins from officials docs: https://www.jenkins.io/doc/book/installing/linux/
Again update the system and install jenkins using the following command:
sudo apt-get update
sudo apt-get install jenkins
Start the Jenkins service & enable it for start during bootup.
sudo systemctl start jenkins
sudo systemctl enable jenkins
Now Let's check Jenkin's status active or not .
sudo systemctl status jenkins
Now to run Jenkins we need to expose port 8080 as Jenkins by default run on port 8080
Below are the steps to expose port 8080
Step 1 : Go to your server's security and click on the group link
Step 2: Click on Edit inbound rules
Step 3 : Click on Add Rule and expose port 8080 then Save
Now, go to the browser and hit your public:8080
Now, go to the browser and hit your public:8080
for example 18.212.251.220:8080
You will get below home page
You can Password location already indicated on the Home page as
/var/lib/jenkins/secrets/initialAdminPassword
Step 5 : Go to your terminal and type
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
It will show your password like below.
Click on Continue, You will see the below screen.
Click on Install Suggested Plugins . So that, Jenkins will install required plugins.
After installing all required plugins, put all the required information.
Then click save and continue
Now Jenkins is Ready!!!!
Now we are ready to create our CICD pipeline let's start.....
Click on New Item on left panel >Enter item name and select freestyle project options >click on Ok
Enter description > select checkbox Github project > enter github url of project
In Source Code Management --> select Git --> enter github url> In credentials
Now we have to create credentials for the Credentials Drop-down, currently, we have none.
We need to add the public key so that we can create a bridge between jenkins and Github for accessing the source code from the Github repo.
Go to your EC2 instance and type command
ssh-keygen
It will create 2 keys.
cd to .ssh and check the keys.
You can see message as "Your identification has been saved in /home/ubuntu/.ssh/id_rsa
Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub
The key fingerprint is:"
Now, navigate to .ssh directory
cd .ssh
You will see the below files here
authorized_keys id_rsa id_rsa.pub
Here id_rsa is the private key and id_rsa .pub is the public key
Cat id_rsa.pub and copy the key.
We need to add the Public key to Github so that we can create the bridge.
Go to Github -> setting -> SSH and GPG key ->Click on Generate SSH Key-->Add new key -> copy the pub key -> give a name to the key title -> paste the copied key.
You will be able to see your key in Authentication refer below.
Now come to your Jenkins and Click on Add+ button in your credentials and select Jenkins icon.
New window will open
Select Kind drop down value as SSH Username with private key. and add ID whatever you want.
Provide Description and user name of your instance and select Private key as Enter Directly and paste your.
Click on Add -> save it
Now, come back to Credentials on the same page , you will see your Description here which you mention.
Now click save button then click build now option . We see that code build successfully.
Go to instance and check if the repo is cloned there.
Here we see that port number is 8000
Expose port 8000 in your security group same as we have done for Jenkin's port 8080.
After Allowing port hit the IP:8000
You will get the result like below.
We will dockerize the application so that it can be accessed anywhere by anyone.
Go to the instance terminal using below commands
sudo apt-get install docker.io
Check docker is active or not using this following command
systemctl status docker
Add your current user to the Docker and jenkins group using the following command:
sudo usermod -aG docker jenkins
sudo reboot
Create your Dockerfile
vim Dockerfile
FROM node:12.2.0-alpine
WORKDIR app
COPY . .
RUN npm install
RUN npm run test
EXPOSE 8000
CMD ["node","app.js"]
Now build the image using this Dockerfile. Then create container from this image.
Now we will automate this process by adding the commands in the shell.
docker build . -t node-todo-app
echo "code build"
docker run -d -p 8000:8000 node-todo-app:latest
echo "code deployed"
Build is successsful. We can access the same on browser.
Now your application is up and running
Automating Deployment Using Github Webhook.
Now all the processes that we were doing manually will be performed automatically as we did the automation.
Now we will configure web-hook so that every time there is any updation, deletion on the repository the job should be automatically triggered and it should perform the upcoming processes.
Kill the existing container first.
Step 1 : Go to Manage Jenkins
Step 2 : Manage Plugins>>Available Plugins
Step 3 : GitHub Integration
Step 4 : Select Install without restart
Step 5 : Select Restart Jenkins when installation complete.
Step 6 : Relogin into Jenkins
Go to repository settings -> webhook -> Add webhook -> Payload URL add jenkins URL here -> (http://18.212.251.220:8080/github-webhook/)
Content type -> application.json
Add webhook.
Refresh the page untill you will get the Right sign as below.
Select GitHub hook trigger for GITScm polling then click Save button
Now, Let's change code in GitHub and as expected Application Deploy automatically.
Go to GitHub Project and make the in todo.js file then click Save button
So the build is successful.
Before update webpage text look like this→
After the updated webpage text look like this→
After configure web-hook, every time if there is any updation, deletion on the repository the job should be automatically triggered and it should perform the upcoming processes.
Thats the power of automation tools.
Now run the application using Docker compose
First we need to install docker compose using following command:
sudo apt-get install docker-compose
You will have to make a Docker Compose file for this Project (or Can be a good open source contribution)
In the job configuration page, scroll down to the Build section and click Add Build step.
Select Execute shell from the dropdown menu to add a shell command to the build. In the command field, enter the following command
In build steps -->select Execute shell --> Enter commands of docker compose and save the configuration
docker-compose down
docker-compose up -d --no-deps --build web
echo "code deployed successfully"
( docker-compose up -d –no-deps –build service name)
--no-deps - Don't start linked services.
--build - Build images before starting containers.
[Note : To rebuild the image before starting the container. It is not sufficient to simply start and stop the old container; it must rebuild the image and then only copy new files into the container.
Now make changes in code and push to Github main branch ,and refresh page so you will see changes
So the build is successful.
Now open the browser and verify the running application.
In conclusion, this comprehensive article has explored various aspects of Jenkins, providing in-depth knowledge and practical hands-on projects to enhance understanding. We covered a wide range of topics, starting with an introduction to CI/CD pipelines and GitHub webhooks, followed by step-by-step instructions on how to use webhooks effectively in conjunction with Jenkins.
We delved into the prerequisites of Jenkins, highlighting the requirements for successful Jenkins projects. Additionally, we provided detailed guidance on creating a GitHub access token and demonstrated the installation and configuration of Jenkins, Java, Docker, and other essential tools, accompanied by helpful reference URLs.
The article proceeded with a thorough walkthrough of creating freestyle projects in Jenkins, including key steps such as generating SSH keys, connecting Jenkins servers, and creating necessary credentials. We also explored writing Docker files and Jenkins scripts to build projects and automate deployment using GitHub webhooks.
Furthermore, we discussed the crucial process of building and deploying applications with Jenkins, emphasizing its significance in the software development life cycle. Through hands-on exercises and practical examples, you gained valuable experience in running applications using Docker Compose on Jenkins servers and verifying their successful execution.
By covering these topics in detail and providing a step-by-step approach, you were able to develop a solid understanding of Jenkins and its powerful capabilities for CI/CD automation. This knowledge empowers you to apply Jenkins effectively in your own projects, streamline development processes, and achieve efficient application deployment.
Thank you for reading this blog. If you found this blog helpful, please like, share, and follow me for more blog posts like this in the future.
— Happy Learning !!!