Day 28–101 Days of DevOps — Terraform Pipeline using Jenkins

Welcome to Day 28 of 101 Days of DevOps. The topic for today is building Terraform Pipeline using Jenkins.

On Day 13, you have learned how to automate the process of GitHub repository creation using Python https://www.101daysofdevops.com/courses/101-days-of-devops/lessons/day-13-github-api-using-python-and-using-pygithub-module/

On Day24 you have learned about Jenkinsfile https://www.101daysofdevops.com/courses/101-days-of-devops/lessons/day-24/

On Day 25 and Day 26, you have learned the basics of terraform.

101 Days of DevOps

101 Days of DevOps

To view the complete course, please check the below url.

101 Days of DevOps

For more info, register via the below link

Create an Account

YouTube Channel link

Prashant Lakhera

Let’s combine all these concepts and build a terraform Jenkins pipeline that looks like this

What is the Pipeline?

A pipeline is typically referred to as a part of Continuous Integration, and it’s mostly used to merge developer changes into code mainline.

Why do we need a pipeline?

1. Test changes before pushing to Production
2. Another pair of eyes in terms of approval
3. Logging to see who and when someone pushed the changes
4. Separate workspace/tfstate for Development and Production Environment

How Jenkins Terraform Pipeline Works?

• User pushes their code changes to GitHub.
• Code change trigger a Git Webhooks, which triggers the terraform pipeline

Setting up Jenkins

• Install Java

# yum -y install java-1.8.0-openjdk.x86_64

• Download the latest version of Jenkins rpm

# wget https://pkg.jenkins.io/redhat-stable/jenkins-2.164.1-1.1.noarch.rpm

RedHat Repository for Jenkins

• Install the Jenkins rpm

# rpm -ivh jenkins-2.164.1-1.1.noarch.rpm
warning: jenkins-2.164.1-1.1.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID d50582e6: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
1:jenkins-2.164.1-1.1              ################################# [100%]

• Start Jenkins

# systemctl status jenkins
● jenkins.service - LSB: Jenkins Automation Server
Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)
Active: active (running) since Sat 2019-03-16 18:59:54 UTC; 5s ago
Docs: man:systemd-sysv-generator(8)
Process: 4241 ExecStart=/etc/rc.d/init.d/jenkins start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/jenkins.service
└─4260 /etc/alternatives/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins....
Mar 16 18:59:53 ip-172-31-29-138.us-west-2.compute.internal systemd[1]: Starting LSB: Jenkins Automation Server...
Mar 16 18:59:53 ip-172-31-29-138.us-west-2.compute.internal runuser[4246]: pam_unix(runuser:session): session opened for user jenkins by (uid=0)
Mar 16 18:59:54 ip-172-31-29-138.us-west-2.compute.internal jenkins[4241]: Starting Jenkins [  OK  ]
Mar 16 18:59:54 ip-172-31-29-138.us-west-2.compute.internal systemd[1]: Started LSB: Jenkins Automation Server.

• Browse your Jenkins URL on port 8080; you will see something like this.

• Copy and Paste the password(from below location)and then Select the installed Plugins

# cat /var/lib/jenkins/secrets/initialAdminPassword

• Once Jenkins is up and running, click on Manage Jenkins and then Manage Plugins.

• Search for Terraform under the Available tab, Select the Plugin and then Click on Install without restart

• Go to Manage Jenkins but this Global Tool Configuration

* Name: This version must be consistent with the version of Jenkins we are going to refer in the Jenkins file
* Version: Use the same version

• Again go back to the main Jenkins console, click on Credentials, and then global.

• Add Credentials and then Secret text.

• Here we want to add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in Secret Text so that when Jenkins job update doesn’t show AWS Key and Secret in plain text.

• This is how Jenkinsfile looks like

pipeline {
    agent any
    tools {
        "org.jenkinsci.plugins.terraform.TerraformInstallation" "terraform"
    }
    parameters {
        string(name: 'WORKSPACE', defaultValue: 'development', description:'setting up workspace for terraform')
    }
    environment {
        TF_HOME = tool('terraform')
        TP_LOG = "WARN"
        PATH = "$TF_HOME:$PATH"
        ACCESS_KEY = credentials('AWS_ACCESS_KEY_ID')
        SECRET_KEY = credentials('AWS_SECRET_ACCESS_KEY')
    }
    stages {
            stage('TerraformInit'){
            steps {
                dir('jenkins-terraform-pipeline/ec2_pipeline/'){
                    sh "terraform init -input=false"
                    sh "echo \$PWD"
                    sh "whoami"
                }
            }
        }

        stage('TerraformFormat'){
            steps {
                dir('jenkins-terraform-pipeline/ec2_pipeline/'){
                    sh "terraform fmt -list=true -write=false -diff=true -check=true"
                }
            }
        }

        stage('TerraformValidate'){
            steps {
                dir('jenkins-terraform-pipeline/ec2_pipeline/'){
                    sh "terraform validate"
                }
            }
        }

        stage('TerraformPlan'){
            steps {
                dir('jenkins-terraform-pipeline/ec2_pipeline/'){
                    script {
                        try {
                            sh "terraform workspace new ${params.WORKSPACE}"
                        } catch (err) {
                            sh "terraform workspace select ${params.WORKSPACE}"
                        }
                        sh "terraform plan -var 'access_key=$ACCESS_KEY' -var 'secret_key=$SECRET_KEY' \
                        -out terraform.tfplan;echo \$? > status"
                        stash name: "terraform-plan", includes: "terraform.tfplan"
                    }
                }
            }
        }
        
        stage('TerraformApply'){
            steps {
                script{
                    def apply = false
                    try {
                        input message: 'Can you please confirm the apply', ok: 'Ready to Apply the Config'
                        apply = true
                    } catch (err) {
                        apply = false
                         currentBuild.result = 'UNSTABLE'
                    }
                    if(apply){
                        dir('jenkins-terraform-pipeline/ec2_pipeline/'){
                            unstash "terraform-plan"
                            sh 'terraform apply terraform.tfplan' 
                        }
                    }
                }
            }
        }
    }
}
* On the top of the pipeline, I am defining any agent that is available to run the pipeline
* Next step I am defining tools that is required to run the configuration and in this case terraform with version number that we setup during the global configuration.
* Next step we need some parameter that is required to run the pipeline, in this case workspace, Right now the only thing you need to know about workspace is that each workspace maintain seperate tfstate file, in this case I am putting development as default workspace

State: Workspaces - Terraform by HashiCorp

* Next step is to setting up some environment variable 
1: Where to find terraform
2: Setting up TF_IN_AUTOMATION = "true" will make terraform don't spit too much information
3: Access and Secret key to interact with AWS

• Various Stages of Terraform Pipeline

Terraform init

The first step is to check in the code from the GitHub repo

100daysofdevops/100daysofdevops

* second step is setting input=false which means we if terraform need input and its not present fail the command
* some extra command for informational purpose

Terraform Format

• In this case, I am checking the file checked in to the git is properly formatted if not simply failed at that point.

Terraform Validate

• Check if the syntax of the .tf file is proper or not

Terraform Plan

• Try to create a new workspace, and if it's already present, select it(default workspace is development)
• Then I am running a simple plan command, providing access and secret key and storing the output in terraform.tfplan
• The next step is I am stashing the output generated by the Previous command so that we can use it later

Terraform Apply

• Here I am giving a choice to the user whether he wants to apply this change; if he chooses Ready to Apply the config, then terraform will run with the plan created earlier.

OR

Creating Jenkins Pipeline

GitHub End

GitHub link: https://github.com/100daysofdevops/100daysofdevops/tree/master/jenkins-terraform-pipeline/ec2_pipeline

Please check this video if you want a step-by-step explanation of the various concepts discussed above.

If you like to dig deeper into the AWS concept, please feel free to check my book.

AWS for System Administrators: Build, automate, and manage your infrastructure on the most popular…

Looking forward to you guys joining this journey and spend a minimum of an hour every day for the next 101 days on DevOps work and post your progress using any of the below mediums.

• Website: https://101daysofdevops.com/
• Linkedin: https://www.linkedin.com/in/prashant-lakhera-696119b/
• Twitter: @100daysofdevops OR @lakhera2015
• Facebook: https://www.facebook.com/groups/795382630808645/
• Medium: https://medium.com/@devopslearning
• GitHub: https://github.com/100daysofdevops/100daysofdevops
• YouTube Channel: https://www.youtube.com/user/laprashant/videos
• Slack: https://join.slack.com/t/100daysofdevops/shared_invite/zt-au03logz-YfDUp_FJF4rAUeDEbgWmsg
• Reddit: r/101DaysofDevops
• Meetup: https://www.meetup.com/100daysofdevops/