DevOps Project to automate infrastructure on AWS using Terraform and GitLab CICD
Before commencing the project, make sure you have a basic understanding of the following topics, as they will simplify the implementation process.
Basic Terraform Knowledge
Understanding of CICD
GitLab CI Knowledge
PREREQUISITES:
1) Aws account creation
Check out the official site to create aws account Link
2) GitLab account
🔷Login to https://gitlab.com
🔷You can sign in via GitHub/Gmail
🔷Verify email and phone
🔷fill up the questionaries
🔷provide group name & project name as per your choice
3) Terraform Installed
Check out the official website to install terraform Here
4) AWS CLI Installed
Navigate to the IAM dashboard on AWS, then select "Users." Enter the username and proceed to the next step
Assign permissions by attaching policies directly, opting for "Administrator access," and then create the user.
Within the user settings, locate "Create access key," and choose the command line interface (CLI) option to generate an access key.
Upon creation, you can view or download the access key and secret access key either from the console or via CSV download.
Now go to your terminal and follow below steps:
sudo apt install unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws configure (input created accesskeyid and secret access key)
cat ~/.aws/config
cat ~/.aws/credentials
aws iam list-users (to list all IAM users in an AWS account)
5) Code editor (Vscode)
Download it from Here
Let's begin with the project. This project is divided in to two parts.
Part1:
Here, we write terraform code, run terraform commands and create infrastructure manually to ensure everything works fine before automating
Part2:
Create CICD pipeline script on Gitlab to automate terraform resource creation
Step1: Create a new folder named “cicdtf” and open it in vscode to start writing the code.
Step2: We will start writing our Terraform code in the “cicdtf” folder. The first step in writing Terraform code is to define a provider. To do this, we will create a file called provider.tf with the following content.
We will be deploying a VPC, a security group, a subnet and an EC2 instance as part of the initial phase.The folder structure is as follows:
1. VPC Module (vpc folder):
Files:
main.tf: Defines resources like VPC, subnets, and security groups.
variables.tf: Declares input variables for customization.
outputs.tf: Specifies outputs like VPC ID, subnet IDs, etc.
2.EC2 Module (web folder):
Files:
main.tf: Configures EC2 instance details, including AMI, instance type, and security groups.
variables.tf: Defines variables needed for EC2 instance customization.
outputs.tf: Outputs instance details like public IP, instance ID, etc.
Snap of folder structure:
Let’s start with defining vpc,
The below Terraform script (main.tf) sets up an AWS Virtual Private Cloud (VPC) with a CIDR block of 10.0.0.0/16, enabling DNS support and hostnames. It creates a subnet (10.0.1.0/24) in us-east-1a with public IP mapping. Additionally, it establishes a security group allowing inbound SSH (port 22) traffic from any IP address and permitting all outbound traffic from the instances within the VPC.
To know more about modules and different parameters being used in this project, check out the official documentation of Terraform Here
Make use of below repositories to check out the code.
https://gitlab.com/N4si/cicdtf
https://gitlab.com/Sakeena19/cicdtf
Step3:
we will create an EC2 instance in the web module and use the security group and subnet defined in the VPC module. This demonstrates how to share values between different modules in Terraform.
Main Module (Root Module): The main.tf file acts as the parent module
Child Modules: The VPC and web modules are child modules.
To share values from one child module to another, we follow these steps:
Define Outputs: Specify the values (e.g., subnet ID, security group ID) as outputs in the VPC module.
Use Variables: Reference these outputs as variables in the web module.
The script in main.tf file in web module is as follows:
Step4: define outputs.tf file in vpc module.
output "pb_sn": Defines an output variable named pb_sn.value = aws_subnet.pb_sn.id: This line assigns the ID of the subnet resource (aws_subnet.pb_sn) to the output variable. This allows other modules to access the subnet ID. Similar for security group as well.
Step5: Define variables.tf file in web module
These variables are used to pass the security group ID and subnet ID from the VPC module to the web module.
Step6: Now to start using these modules, we have to define both vpc and web in the root module(main.tf) as shown below.
source = "./vpc": Specifies the path to the VPC module directory. This imports the VPC module defined in the ./vpc folder.
source = "./web": Specifies the path to the web module directory. This imports the EC2 module defined in the ./web folder.
sn = module.vpc.pb_sn: Passes the subnet ID output (pb_sn) from the VPC module to the EC2 module, assigning it to the variable sn.
sg = module.vpc.sg: Passes the security group ID output (sg) from the VPC module to the EC2 module, assigning it to the variable sg.
Step7:
Now to check whether the code is working fine, lets run terraform commands. Make sure to connect aws with terraform (using aws configure) before running and save all the files if not done already.
To initialize terraform, use “terraform init” command which setups everything necessary for terraform to manage your infrastructure such as modules, plugins, backend config etc., as defined in your configuration files.
To check if our code is valid, use “terraform validate” command.
Run “terraform plan” command is used to create an execution plan to see what changes terraform will make to your infrastructure without actually applying those changes.
In below snap shown, it’s going to create 4 components: vpc, ec2 instance , subnet and security group will be created.
You can also run by checking “terraform apply -auto-approve” command which executes the terraform plan without requiring interactive communication and proceeds with deployment.
When we run apply, terraform. tfstate file will be created which is not a good practise to have it in local machine, we will setup backend in later steps to store on S3 using DynamoDB.
Also it will create vpc, subnet and ec2 instance as well which can be verified in your aws console.
Now that the code is working fine locally, we'll configure a backend on S3, push the code to GitLab, and proceed with the second part of the project: setting up a CI/CD pipeline to automate the infrastructure deployment tasks we previously performed manually.
before this, delete everything using “terraform destroy -auto-approve” to proceed with automation.
Step8: Set up a backend using S3 and Dynamo DB.
Follow below video or documentation mentioned which has a complete process on how to setup S3 bucket and DynamoDB in detail.
https://developer.hashicorp.com/terraform/language/settings/backends/s3
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket
[youtu.be/o04xfWEouKM?si=OGNj1c9R2iqe9TOM
](youtu.be/o04xfWEouKM?si=OGNj1c9R2iqe9TOMThe)The code for creating s3 and DynamoDB is as follows:Once the code has been written in a file, run below terraform commands to create s3 and DynamoDB table.
🔸terraform init (initialize your working directory)
🔸terraform plan (plan the changes)
🔸terraform apply (apply the changes)
This configuration will create an S3 bucket with versioning and server-side encryption enabled, as well as a DynamoDB table named state-lock with on-demand billing and a string primary key LOCKID
After applying the changes, you can verify whether the s3 bucket and DynamoDB table created in your aws console.
Now create an backend.tf file which will have your bucket details and dynamo dB table.
Run “terraform init” to initialize the backend.
To automate all the above actions, let’s move to part2, i.e., create GitLab repo, push code to repo and setup cicd pipeline.
Go to GitLab and create a new repository:
Click on new project -> create a blank project -> provide project name, visibility, enable readme -> create project.
To push the code, first step is to initialize the repository.
To use only necessary files and ignore other files, create .gitignore file which can be found Here
To connect with your GitLab repo, use
git remote add origin https://gitlab.com/Sakeena19/cicdtf.git
The next step is to create a branch called "dev" because we cannot directly push our code to the main branch. This allows us to make changes and test them safely before merging into the main branch which is the best practice.
To create a branch, use “git checkout -b dev” which will create a branch and switch at a time.
🔹git add . (Adds all changes from current working Dir to staging area)
🔹git commit -m “initial commit” (commits the staged changes to local repo)
🔹git push -u origin dev (pushes code from local to remote repo i.e., GitLab in the branch named dev)
The dev branch should be created in your GitLab repo from which you can create merge request to merge from dev to main.
After merging you can view your code available in main branch.
Now as the code is ready, lets write a CICD pipeline script in Gitlab.
The pipeline configuration file must be named "gitlab-ci.yml" for GitLab to recognize it as the CI/CD configuration file. This naming convention ensures that GitLab understands and processes the configuration defined within.
The main purpose of defining this file is to automate the terraform commands so that whenever a person makes any change in infrastructure the pipeline will trigger automatically.
As it’s not a best practise to hardcode the aws secret and access key in code, variables can be created for storing access keys and secret access keys in your GitLab repository.
Navigate to your project repo -> settings -> CICD ->Variables -> Add variables -> Add variables for your access key and secret access key.
Once the above changes done, the pipeline will start triggering automatically executing all the steps we scripted in .gitlab-ci.yml file.
cicd pipeline script explanation (.gitlab-ci.yml file):
This GitLab CI/CD pipeline script is designed to automate the deployment and management of infrastructure using Terraform. The script uses a Docker image that has Terraform installed and sets environment variables for AWS credentials and a GitLab token. It also caches Terraform plugins and modules to improve efficiency. The pipeline is divided into four stages: validate, plan, apply, and destroy.
In the validate stage, the script checks if the Terraform configuration files are correct. The plan stage then generates a Terraform execution plan and saves it as an artifact called planfile. The apply stage uses this plan to create or update the infrastructure, but this stage must be triggered manually to execute. Similarly, the destroy stage, which is also manually triggered, destroys the Terraform-managed resources automatically.
Before running these stages, the script outputs the Terraform version and initializes Terraform with a backend configuration specified in a tfstate.config file. By organizing the pipeline in this way, the script ensures that infrastructure changes are validated, planned, and applied in a controlled and orderly manner, with the option to manually control the application and destruction of infrastructure changes.
Whenever the pipeline executes, the validate and plan stages run automatically, while the apply and destroy stages require manual execution, as defined in the script. This approach follows industry best practices, allowing verification of changes and manual approval before they are applied or destroyed.
1) Logs from validate stage:
2) Logs from plan stage:
3) Logs from apply:
4) Logs from destroy:
All 4 states have been executed.
The pipeline performs the following steps:
Initializes Terraform with the specified backend configuration.
Applies the Terraform plan to create infrastructure resources (VPC, Subnet, Security Group, and EC2 instance).
Saves .terraform directory to cache for future use.
Cleans up the environment after the job is completed.