What is Infrastructure as Code(IaC)?
Infrastructure as Code (IaC) is an IT practice that automates infrastructure management by using configuration files. With IaC, the entire infrastructure is described using code, which makes it easier to deploy, update, and scale environments.Popular IaC tools include:
- AWS CloudFormation
- ARM templates
- Red Hat Ansible
- Chef, Puppet
- HashiCorp Terraform
Why Terraform?
Terraform is an infrastructure as code tool that lets you define both cloud and on-prem resources in human-readable configuration files that you can version, reuse, and share. Terraform uses a declarative configuration language known as HashiCorp Configuration Language, or optionally JSON. Terraform helps to deploy infrastructure across multi-cloud and certain areas of on-prem data centres.Key Features of Terraform:
-
- Declarative Language: Terraform uses the HashiCorp Configuration Language (HCL) to define infrastructure, which allows for reusable and shareable configurations.
- Modules: Terraform modules help organize configurations, encapsulate resources, and enable code reuse, consistency, and best practices.
What are Terraform Modules?
A Terraform module is a collection of standard configuration files in a dedicated directory. Modules are intended for code reusability. In other words a module allows you to group resources together and reuse this group later, possibly many times.- Modules helps in organising configuration
- Encapsulation Configuration
- Re- use configuration
- Provide consistency and best practices
Why Use Remote State in Terraform?
By default, Terraform stores information about the infrastructure in a state file (terraform.tfstate) in the local filesystem. Use of a local file makes Terraform usage complicated because each user must make sure they always have the latest state data before running Terraform and make sure that nobody else runs Terraform at the same time. With remote state, Terraform writes the state data to a remote data store, which can then be shared between all members of a team, also storing state files remotely can offer additional security. Terraform supports storing state in Terraform Cloud, HashiCorp Consul, Amazon S3, Azure Blob Storage, Google Cloud Storage, Alibaba Cloud OSS, and more.Architecture Overview
Deploying an AKS cluster in a custom Virtual Network using Terraform and the CI will be achieved using Azure Pipeline. This scenario will be focused on using GitHub as version control system and integrating it with Azure Pipeline.Workflow:
-
A push to the
- The pipeline initializes Terraform and deploys the AKS cluster.
- The result is reported back to the user with the build and deployment status.
Azure VNET IP Allocation
For ensuring the traffic flow to an AKS Cluster you can define one of two CNI’s which can either be kubenet or Azure CNI. Here we are using Azure CNI. With Azure CNI, every pod gets an IP address from the subnet and can directly communicate with other pods and services. In CNI each node has a configuration parameter that decides the maximum number of pods it can hold. So, an equivalent number of IP addresses are reserved upfront for that node. This demands more planning while creating the Virtual Network, otherwise it will lead to IP address exhaustion or need to rebuild the clusters in a larger subnet as your application demands grow.Used Tools
- GitHub
- Azure Cloud
- Terraform
- Managed Kubernetes
- Azure Pipeline
Steps to Deploy AKS Cluster using Azure Pipeline and Terraform
1. Create a GitHub Repository
The entire code will be kept inside the GitHub repository. Example: tf files and pipeline file.
2. Configure Azure Service Principal With A Secret:
There are numerous scenarios where you want rights on Azure subscription but not as a user; rather as an application. Here comes the Azure service principal, this will provide access to the resources or resource group in Azure Cloud. For this deployment we recommend using Service Principal.
3. Creating An Azure Service Principal With Contributor Role:
$ az ad sp create-for-rbac --name "myApp" --role contributor
--scopes /subscriptions/<subscription-id>/resourceGroups/<resource-group>
--sdk-auth
Replace <subscription-id>, <resource-group> with the subscription id, resource group name (need to create a resource group). Using “–sdk-auth” will print the output that is compatible with the Azure SDK auth file. For more information
Note: The service principal also required Active Directory permission to read AD group information.
4. Setting Up An Azure Devops Project And A Service Connection
- Azure DevOps project provides a platform for users to plan, track progress, and collaborate on software development.
- To create a project:
- Go to Azure DevOps > New Project > Create.
- Azure Resource Manager service connection helps deploy applications to Azure Cloud, with options for various scenarios.
- Using Azure Resource Manager with an existing service principal:
- Go to Project Settings > Service Connections (within Pipelines Settings).
- Select New Service Connections > Azure Resource Manager > Service Principal (Manual).
- Create a service connection on the subscription (scope level) and provide the credentials of the service principal created previously.
5. Getting Terraform Files:
Checkout to a new branch and create tf files with respect to below tree structure. ├── azure-pipelines.yml
└── azure-tf
├── main.tf
├── sg-aks.tf
├── sg-resource-group.tf
├── sg-vnet.tf
├── terraform.tfvars
└── variables.tf
6. Terraform Contents:
Terraform will deploy an AKS cluster on a custom Azure Virtual network along with a cluster resource group
main.tf
terraform {
backend "azurerm" {
}
}
# Azure Provider Version #
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "2.99"
}
azuread = {
source = "hashicorp/azuread"
version = "2.21.0"
}
}
}
# Configure the Microsoft Azure Provider
provider "azurerm" {
features {}
}
sg-aks.tf
data "azuread_group" "admin-team" {
display_name = "<your-admin-group>"
}
module "aks" {
source = "Azure/aks/azurerm"
version = "4.14.0"
resource_group_name = azurerm_resource_group.sg_aks_rg.name
kubernetes_version = "1.22.6"
orchestrator_version = "1.22.6"
prefix = "${var.env}-${var.group}-${var.app}"
cluster_name = "${var.env}-${var.group}-${var.app}-cluster"
vnet_subnet_id = module.vnet.vnet_subnets[0]
network_plugin = "azure"
os_disk_size_gb = 50
sku_tier = "Paid" # defaults to Free
enable_role_based_access_control = true
rbac_aad_admin_group_object_ids = [data.azuread_group.admin-team.id]
rbac_aad_managed = true
//enable_azure_policy = true # calico.. etc
enable_auto_scaling = true
enable_host_encryption = true
agents_min_count = 1
agents_max_count = 2
agents_count = null # Please set `agents_count` `null` while `enable_auto_scaling` is `true` to avoid possible `agents_count` changes.
agents_max_pods = 100
agents_pool_name = "agentpool"
agents_availability_zones = ["1", "2", "3"]
agents_type = "VirtualMachineScaleSets"
agents_size = "Standard_B2ms"
agents_labels = {
"agentpool" : "agentpool"
}
agents_tags = {
"Agent" : "defaultnodepoolagent"
}
net_profile_dns_service_ip = var.net_profile_dns_service_ip
net_profile_docker_bridge_cidr = var.net_profile_docker_bridge_cidr
net_profile_service_cidr = var.net_profile_service_cidr
depends_on = [module.vnet]
}
Note: Replace <your-admin-group> with the name of your AD group (if present). Otherwise create an Active Directory Group and attach the users who wanted to have access to the AKS Cluster.
sg-resource-group.tf
resource "azurerm_resource_group" "sg_aks_rg" {
name = "${var.env}-${var.group}-${var.app}-rg"
location = var.region
tags = {
app = var.app
env = var.env
group = var.group
}
}
sg-vnet.tf
module "vnet" {
source = "Azure/vnet/azurerm"
version = "~> 2.6.0"
resource_group_name = azurerm_resource_group.sg_aks_rg.name
vnet_name = "${var.env}-${var.group}-${var.app}-${var.vnet_name}"
address_space = var.address_space
subnet_prefixes = var.subnet_prefixes
subnet_names = var.subnet_names
tags = {
env = var.env
group = var.group
app = var.app
}
depends_on = [azurerm_resource_group.sg_aks_rg]
}
terraform.tfvars
## Vnet Variables ##
address_space = ["10.10.0.0/16"]
subnet_prefixes = ["10.10.32.0/19", "10.10.64.0/19", "10.10.96.0/19"]
variables.tf
## Global Variables ##
variable "region" {
type = string
default = "uksouth"
}
variable "env" {
type = string
default = "poc"
}
variable "group" {
type = string
default = "devops"
}
variable "app" {
type = string
default = "aks"
}
## VNET variables ##
variable "vnet_name" {
description = "Name of the vnet to create"
type = string
default = "vnet"
}
variable "address_space" {
type = list(string)
description = "Azure vnet address space"
}
variable "subnet_prefixes" {
type = list(string)
description = "Azure vnet subnets"
}
variable "subnet_names" {
type = list(string)
description = "Azure vnet subnet names in order"
default = ["subnet1", "subnet2", "subnet3"]
}
## AKS Variables ##
variable "net_profile_service_cidr" {
description = "(Optional)
The Network Range used by the Kubernetes service. Changing this forces a new resource to be created.”
type = string
default = “10.0.0.0/16”
}
variable “net_profile_dns_service_ip” {
description = “(Optional) IP address within the Kubernetes service address range that will be used by cluster service discovery (kube-dns). Changing this forces a new resource to be created.”
type = string
default = “10.0.0.10”
}
variable “net_profile_docker_bridge_cidr” {
description = “(Optional) IP address (in CIDR notation) used as the Docker bridge IP address on nodes. Changing this forces a new resource to be created.”
type = string
default = “172.17.0.1/16”
}
7. Pipeline Setup:
Azure Pipeline will help to build and test the code automatically. Here, the trigger will be master.
azure-pipelines.yml
trigger:
- master
#variables:
# global_variable: value # this is available to all jobs
jobs:
- job: terraform_deployment
pool:
vmImage: ubuntu-latest
variables:
az_region: <region>
resource_group_name: <resource-group-name>
subscription: <service-connection-auth>
key_vault_name: <key-vault-name>
sa_prefix: <service-account-name>
sa_container_name: <blob-container-name>
tfstateFile: terraform.tfstate
steps:
– task: AzureCLI@2
inputs:
azureSubscription: ‘<service-connection-auth>’ #replace with your
service connection – azure resource manager service principal
scriptType: ‘bash’
scriptLocation: ‘inlineScript’
inlineScript: |
az group create -n $(resource_group_name) -l $(az_region)
VAULT_ID=$(az keyvault create –name “$(key_vault_name)” –resource-group “$(resource_group_name)” –location “$(az_region)” –query “id” -o tsv)
az storage account create –resource-group $(resource_group_name) –name “$(sa_prefix)” –sku Standard_LRS –encryption-services blob
az storage container create –name $(sa_container_name) –account-name “$(sa_prefix)” –auth-mode login
– task: TerraformInstaller@0
displayName: Terraform Installation
inputs:
terraformVersion: ‘latest’
– task: TerraformTaskV3@3
displayName: Terraform Init
inputs:
provider: ‘azurerm’
command: ‘init’
workingDirectory: ‘$(System.DefaultWorkingDirectory)/tf-files’
backendServiceArm: ‘<service-connection-auth>’
backendAzureRmResourceGroupName: ‘$(resource_group_name)’
backendAzureRmStorageAccountName: ‘$(sa_prefix)’
backendAzureRmContainerName: ‘$(sa_container_name)’
backendAzureRmKey: ‘$(tfstateFile)’
– task: TerraformTaskV3@3
displayName: Terraform Plan
inputs:
provider: ‘azurerm’
command: ‘plan’
workingDirectory: ‘$(System.DefaultWorkingDirectory)/tf-files’
commandOptions: ‘-out=tfplan’
environmentServiceNameAzureRM: ‘akrish-poc-sp’
– task: TerraformTaskV3@3
displayName: Terraform Apply
inputs:
provider: ‘azurerm’
command: ‘apply’
workingDirectory: ‘$(System.DefaultWorkingDirectory)/tf-files’
commandOptions: ‘tfplan’
environmentServiceNameAzureRM: ‘<service-connection-auth>’
Replace variables in the pipeline with appropriate values. Add service connection name in the <service-connection-auth>field.
