Automation using Jenkins Job DSL Plugin
Put your one more step towards Automation…
12 min readJul 12, 2020
This Article is for those who are still creating jobs in Jenkins manually but they want to see the another flavor of Jenkins’s Automation feature & want to create jobs Automatically on the fly.
As we know Plugins are the only way to enhance the functionality and Power of Jenkins.Jenkins Provides a Very Powerful Plugin : Job DSL. This Plugin allows you to create several jobs automatically from a main job called seed job.It is so called because it is the seed from which other jobs are created.The seed task basically consists of a file with a Groovy script where the tasks to be created automatically ,are defined.Even if you have never used Groovy you should not worry,the Jenkins provides you a JOB DSL API for syntax reference.
But Why , We are Using It ??
- For generated all the jobs using seed job , you only need to execute a single job it will automatically create all jobs for you. You can reduce this Extra Load of create jobs manually.
- In this way we avoid having to re-create the same jobs for each new project that requires similar tasks, being able to mount all the tasks for each project simply by executing the seed task. Not only that, if we modify the seed task and run it again Jenkins will update the jobs already created. In this way, we could update a single job (our seed task) and automatically update all projects that use the tasks generated by that seed. Fantastic, isn’t it?
- In the real words this groovy script which was used in seed job to run the jobs automatically are generally written by Developer, because how to deal with different jobs developer knows batter then operation guys , so developer can do this task for operation world for doing development & This is also the meaning of DevOps : when some task of operation Guys is performed by Developer & some task of developer performed by Operation team to get fast & agile development of a product.
so we are performing this following task using Jenkins Job DSL Plugin & with Integrating different tools of DevOps-World like Git,GitHub,Jenkins,K8s. For better understanding of this task please visit previous task done by me task-2 & task-3 .
OBJECTIVE :-
Perform this task with the help of Jenkins coding file ( called as jenkinsfile approach ) and perform the with following phases:
1. Create container image that’s has Jenkins installed using dockerfile Or You can use the Jenkins Server on RHEL 8/7
2. When we launch this image, it should automatically starts Jenkins service in the container.
3. Create a job chain of job1, job2, job3 and job4 using build pipeline plugin in Jenkins
4. Job2 ( Seed Job ) : Pull the Github repo automatically when some developers push repo to Github.
5. Further on jobs should be pipeline using written code using Groovy language by the developer
6. Job1 :
1. By looking at the code or program file, Jenkins should automatically start the respective language interpreter installed image container to deploy code on top of Kubernetes ( eg. If code is of PHP, then Jenkins should start the container that has PHP already installed )
2. Expose your pod so that testing team could perform the testing on the pod
3. Make the data to remain persistent using PVC ( If server collects some data like logs, other user information )
7. Job3 : Test your app if it is working or not.
8. Job4 : if app is not working , then send email to developer with error messages and redeploy the application after code is being edited by the developer
Shall we start?
- Let’s say developer write all the code for product/application & also Jenkins dSL script file in which all the groovy code is written that will generate the all jobs automatically in build pipeline.
job('github-pull-t6'){
scm{
github('Radhika299/task-6-jenkins-dsl')
}triggers{
upstream('seed-job-t6','SUCCESS')
}
steps{shell('if ls | grep .html; then sudo docker build -t radhu299/httpd-task6 . -f html.Dockerfile; sudo docker push radhu299/httpd-task6; elif ls | grep .php; then sudo docker build -t radhu299/php-task6 . -f php.Dockerfile; sudo docker push radhu299/php-task6; else echo "other files are present not needed to build & push image.."; fi')
}
}job('auto-deploy-t6'){
triggers{
upstream('github-pull-t6','SUCCESS')
}
steps{
shell('cd /var/lib/jenkins/workspace/seed-job-t6/; if ls | grep .html; then if sudo kubectl get deployment | grep web-html-deployment; then echo "deployment for html application already running"; kubectl rollout restart deployment/web-html-deployment; else sudo kubectl create -f /task6-jenkinsdsl/htmlpv.yml; sudo kubectl create -f /task6-jenkinsdsl/htmldeployment.yml; sudo kubectl create -f /task6-jenkinsdsl/htmlexpose.yml; fi; elif ls | grep .php; then if sudo kubectl get deployment | grep web-php-deployment; then echo "deployment for php application already running"; kubectl rollout restart deployment/web-php-deployment; else sudo kubectl create -f /task6-jenkinsdsl/phppv.yml; sudo kubectl create -f /task6-jenkinsdsl/phpdeployment.yml; sudo kubectl create -f /task6-jenkinsdsl/phpexpose.yml; fi; else echo "other files can not deploy automatically"; fi')
}}job('testing and email-sending-t6'){
triggers{
upstream('auto-deploy-t6','SUCCESS')
}
steps{
shell('if sudo kubectl get svc html-pod-expose ; then status1=$( curl -o /dev/null -s -w "%{http_code}" http://192.168.99.104:31000/index.html ); if [[ $status1 == 200 ]]; then exit 0; else exit 1; fi; elif sudo kubectl get svc php-pod-expose; then status1=$( curl -o /dev/null -s -w "%{http_code}" http://192.168.99.104:32000/index.php ); if [[ $status1 == 200 ]]; then exit 0; else exit 1; fi; else echo "No pod is exposed , so no need of testing"; fi')
}
publishers {
extendedEmail {
recipientList('radhey2900@gmail.com')
triggers {
failure {
sendTo {
recipientList()
}
}
}
}
}}buildPipelineView('task-6-build-pipelineview') {
title('task-6-build-pipelineview')
displayedBuilds(3)
selectedJob('github-pull-t6')
}
Above you can see the complete code written in groovy script for seed job.
Let’s see the Seed-job:
- In this job we are giving the name of the groovy file, It will be downloaded by Jenkins from the provided link of GitHub repository .
- This is the job which is configured by us only but code of groovy file will be pushed by developer in GitHub repository & This job use that code for generating jobs.
- But remember you have to approve this groovy script,the Jenkins has this option for security reasons to prevent the whole system from malicious code or virus so operation guys firstly have to approve this otherwise you will get the following error.
- The process of approval either can be done manually or automatic for automatic you have to write a code or program that will check either this file is correct or not, if it is OK then approve this file.
- we can also use Groovy Sandbox for approval , then we have to install Authorize project plugin & have to configure different authorization for this job for define the “job run as this particular user”& other things all, so just forget this confusing part because I am doing here manual approval..;)
- For approve the file you have to go in “manage jenkins -> In process Script Approval.”
& Here we Generated 3 different jobs in a chain using a single job ..
Now whatever I will show you further this will be automatically configured by this job, so let’s see what this job configure & create for us.
JOB-1(github-pull-t6)
- This job pull the code from GitHub repository & then build the Dockerfile & push it it into docker hub repository . Remember firstly It will see the code extension & after find which kind of code is provide ,it builds corresponding Dockerfile.
- For reference I wrote code for two type of file HTML& PHP , and in this case developer pushed the only html code so it will build the httpd Dockerfile.
code which was written in groovy script for this generating this job:
job('github-pull-t6'){
scm{
github('Radhika299/task-6-jenkins-dsl')
}triggers{
upstream('seed-job-t6','SUCCESS')
}
steps{shell('if ls | grep .html; then sudo docker build -t radhu299/httpd-task6 . -f html.Dockerfile; sudo docker push radhu299/httpd-task6; elif ls | grep .php; then sudo docker build -t radhu299/php-task6 . -f php.Dockerfile; sudo docker push radhu299/php-task6; else echo "other files are present not needed to build & push image.."; fi')
}
}
Now move to JOB-2(auto-deploy-t6):
- This job find the extension & type of code file and launch the application inside k8s usind different resources of k8s like Deployment,Pods,Services,Expose,PVC & PV.
- The interesting thing about this job that we are attaching- “the pods in which the container having our application is running “ with Persistent storage using a Powerful concept in k8s : PV[Persistent Volume] & PVC[Persistent Volume Claim].
Advantage of PV & PVC -
- We can mount the critical folder of container [The folder which has important data & we need that data in case Pod fails ] to these PV using PVC.
- One example of these kind of files are LOG FILES: Log files contains all the logs related to a web application & other things .we also know that the logs are very important because logs have been an essential part of troubleshooting application and infrastructure performance.
- so we don’t want to loose these log file if container fails or application fails , That’s why we have to make the data of log file permanent & one and only way for make them permanent to use PV & PVC so that even if pod fail we have log files’ data in our host system.
- Groovy code for this job:-
job('auto-deploy-t6'){
triggers{
upstream('github-pull-t6','SUCCESS')
}
steps{
shell('cd /var/lib/jenkins/workspace/seed-job-t6/; if ls | grep .html; then if sudo kubectl get deployment | grep web-html-deployment; then echo "deployment for html application already running"; kubectl rollout restart deployment/web-html-deployment; else sudo kubectl create -f /task6-jenkinsdsl/htmlpv.yml; sudo kubectl create -f /task6-jenkinsdsl/htmldeployment.yml; sudo kubectl create -f /task6-jenkinsdsl/htmlexpose.yml; fi; elif ls | grep .php; then if sudo kubectl get deployment | grep web-php-deployment; then echo "deployment for php application already running"; kubectl rollout restart deployment/web-php-deployment; else sudo kubectl create -f /task6-jenkinsdsl/phppv.yml; sudo kubectl create -f /task6-jenkinsdsl/phpdeployment.yml; sudo kubectl create -f /task6-jenkinsdsl/phpexpose.yml; fi; else echo "other files can not deploy automatically"; fi')
}}
All the important file written in yml to create deployment,expose,PV,PVC for launching application will be in our host system:
Code for these files: —
For HTML Application files:
---------------------htmldeployment.yml-----------------------------
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: web-html-pvc
labels:
app: web-html-app
spec:
accessModes:
- ReadWriteOnce
storageClassName: manual
resources:
requests:
storage: 1Gi---apiVersion: apps/v1
kind: Deployment
metadata:
name: web-html-deployment
labels:
app: web-html-app
spec:
selector:
matchLabels:
app: web-html-app
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: web-html-app
spec:
containers:
- image: radhu299/httpd-task6
name: web-html-container
ports:
- containerPort: 80
volumeMounts:
- name: web-html-volume
mountPath: "/var/log/httpd/"volumes:
- name: web-html-volume
persistentVolumeClaim:
claimName: web-html-pvc--------------------------htmlpv.yml--------------------------------
apiVersion: v1
kind: PersistentVolume
metadata:
name: html-pv-volume
labels:
app: web-html-app
type: local
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data/html/"---------------------------htmlexpose.yml---------------------------
apiVersion: v1
kind: Service
metadata:
name: html-pod-expose
spec:
type: NodePort
selector:
app: web-html-app
ports:
- port: 80
targetPort: 80
nodePort: 31000
For PHP application files:
---------------------------phpdeployment.yml------------------------apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: web-php-pvc
labels:
app: web-php-app
spec:
accessModes:
- ReadWriteOnce
storageClassName: manual
resources:
requests:
storage: 1Gi---apiVersion: apps/v1
kind: Deployment
metadata:
name: web-php-deployment
labels:
app: web-php-app
spec:
selector:
matchLabels:
app: web-php-app
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: web-php-app
spec:
containers:
- image: radhu299/php-task6
name: web-php-container
ports:
- containerPort: 80
volumeMounts:
- name: web-php-volume
mountPath: "/var/log/httpd/"volumes:
- name: web-php-volume
persistentVolumeClaim:
claimName: web-php-pvc--------------------------phppv.yml--------------------------------
apiVersion: v1
kind: PersistentVolume
metadata:
name: php-pv-volume
labels:
app: web-php-app
type: local
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data/php/"--------------------------phpexpose.yml-----------------------------
apiVersion: v1
kind: Service
metadata:
name: php-pod-expose
spec:
type: NodePort
selector:
app: web-php-app
ports:
- port: 80
targetPort: 80
nodePort: 32000
Now let’s see JOB-3(testing and email-sending-t6)
Groovy code for this job configuration & creation:
job('testing and email-sending-t6'){
triggers{
upstream('auto-deploy-t6','SUCCESS')
}
steps{
shell('if sudo kubectl get svc html-pod-expose ; then status1=$( curl -o /dev/null -s -w "%{http_code}" http://192.168.99.104:31000/index.html ); if [[ $status1 == 200 ]]; then exit 0; else exit 1; fi; elif sudo kubectl get svc php-pod-expose; then status1=$( curl -o /dev/null -s -w "%{http_code}" http://192.168.99.104:32000/index.php ); if [[ $status1 == 200 ]]; then exit 0; else exit 1; fi; else echo "No pod is exposed , so no need of testing"; fi')
}
publishers {
extendedEmail {
recipientList('radhey2900@gmail.com')
triggers {
failure {
sendTo {
recipientList()
}
}
}
}
}}
// For build pipeline viewbuildPipelineView('task-6-build-pipelineview') {
title('task-6-build-pipelineview')
displayedBuilds(3)
selectedJob('github-pull-t6')
}
- In this job we test our application which is running inside production environment ,here we our doing a very basic testing by checking status code which we get from curl command.
- If status code is 200, that means application is running fine & it will run command exit 0 [job built will be successful ] otherwise exit 1[job will be fail].
- If job fails then it will trigger email-sending option , for email-sending we are using here Email Extension Plugin.
Redeployment of Pods: —
- when developer receives the email , it is edited[remove bug from application , updating & other tasks] & committed by developer again and whole cycle of pushing ,triggering seed job is done automatically and thus redeployment of pods occurs.
Hurrah !! we done it….
PS: Here , I would like to show my gratitude & Respect for Vimal Daga sir, This task was given by them during DevOps Assembly Line training to all the trainees, I was one of them ,& Here I completed this task. This complete task is done by me under their guidance only.. Thank you sir :)