Use Terraform to connect ACR with Azure Web App

You can connect an Azure Web App to Docker Hub, Private Repository and also an Azure Container Registry(ACR). Using Terraform you can take it a step further and build your whole infrastructure environment at the same time as connecting these container registries. However, how do you connect them together in Terraform?

I am going to focus on the connection of an ACR, but you can also follow the same method for the other providers.

Why I am using this as an example, is when correcting the other methods they are a simple URL, username and password, but the Azure Container Registry within the portal has a different user interface where it connects natively in the Azure. Why I was learning to do this, I kept getting my ACR connecting like a private repository instead of an actual ACR. Therefore, the method below will have the desired outcome of within the Azure portal the Web App showing it is connected to an ACR.

I will go through the general setup I have got for a simple Web App connecting to an ACR with all of the supporting  elements. I am not showing best practice of having the variables and outputs in separate files as this is not the point of the post, but I would encourage people to do that.

First we will need to create the infrastructure to support the Web App, by connecting to the Azure Resource Manager provider in Terraform:

provider "azurerm" {
  version         = "=2.25.0"
  subscription_id = var.subscription_id
  features {}
}

This passes a ‘subscription_id’ variable to connect to the correct subscription. We then create the Resource Group to contain all the resources.

variable "resource_group_name" {
  type        = string
  description = "Azure Resource Group Name. "
}
variable "location" {
  type        = string
  description = "Azure Resource Region Location"
}

# Create a Resource Group
resource "azurerm_resource_group" "acr-rg" {
  name = var.resource_group_name
  location = var.location  
}

The next part is to create the Azure Container Registry with your chosen name and the SKU for the service level you would like. For this example we have use the ‘Standard’ to keep it cheap and simple, while using the same location as the Resource Group.

variable "container_registry_name" {
  type        = string
  description = "Azure Container Registry Name"
}

# Azure Container Regristry
resource "azurerm_container_registry" "acr" {
  name                     = var.container_registry_name
  resource_group_name      = azurerm_resource_group.acr-rg.name
  location                 = azurerm_resource_group.acr-rg.location
  sku                      = "Standard"
  admin_enabled            = true
}

For the Web App we will need an App Service Plan to contain the Web App and set the SKU Level. You can see this is the same as before using the same locations and also I am using Linux as the base operating system.

variable "app_plan_name" {
  type        = string
  description = "Azure App Service Plan Name"
}

# App Plan
resource "azurerm_app_service_plan" "service-plan" {
  name = var.app_plan_name
  location = azurerm_resource_group.acr-rg.location
  resource_group_name = azurerm_resource_group.acr-rg.name
  kind = "Linux"
  reserved = true  
  sku {
    tier = "Standard"
    size = "S1"
  }  
}

Now is where we declare the Web App itself, but first create the 3 variables we will need. The Web App name, your Registry name and the Tag assigned to your image.

variable "web_app_name" {
  type        = string
  description = "Azure Web App Name"
}
variable "registry_name" {
  type        = string
  description = "Azure Web App Name"
}
variable "tag_name" {
  type        = string
  description = "Azure Web App Name"
 default: 'latest'
}

To link to Docker Registries you need 3 App Settings configured ‘ DOCKER_REGISTRY_SERVER_URL’, ‘ DOCKER_REGISTRY_SERVER_USERNAME’, and ‘DOCKER_REGISTRY_SERVER_PASSWORD’.

These are used to gain the correct access to the registries.

For the ACR, the URL is the ‘Login Server’ and then the username/password is the Admin Username/Password.

These can be found here in the portal, if your ACR is already created.

For example:

    DOCKER_REGISTRY_SERVER_URL      = "https://myacr.azurecr.io"
    DOCKER_REGISTRY_SERVER_USERNAME = myacr
    DOCKER_REGISTRY_SERVER_PASSWORD = *********

A key part to see here is the URL is prefixed with the ‘https’ and it needs to be this, not http as it needs to be secure.

Instead of getting these details manually, we are using Terraform so we have access to these details from the created Azure Container Registry that we can use:

    DOCKER_REGISTRY_SERVER_URL              = "https://${azurerm_container_registry.acr.login_server}"
    DOCKER_REGISTRY_SERVER_USERNAME = azurerm_container_registry.acr.admin_username
    DOCKER_REGISTRY_SERVER_PASSWORD = azurerm_container_registry.acr.admin_password

We now have a connection to the ACR, but need to tell the Web App what registry and tag to look for. As we are using a Linux based server, we configure the ‘linux_fx_version’ in the site config with this pattern below, but for Windows you would use ‘windows_fx_version’.

"DOCKER|[RegistryName]:[TagName]"

For an example with a registry name MyRegistry and a tag name MyTag:

"DOCKER|MyRegistry:MyTag"

Below is the full example of the Web App generation in Terraform. With all these parts together you should have a Resource Group containing a ACR, App Service Plan and a Web App all connected.

# web App
resource "azurerm_app_service" "app-service" {
  name = var.web_app_name
  location = azurerm_resource_group.acr-rg.location
  resource_group_name = azurerm_resource_group.acr-rg.name
  app_service_plan_id = azurerm_app_service_plan.service-plan.id
  app_settings = {
    WEBSITES_ENABLE_APP_SERVICE_STORAGE = false
   
    # Settings for private Container Registires  
    DOCKER_REGISTRY_SERVER_URL      = "https://${azurerm_container_registry.acr.login_server}"
    DOCKER_REGISTRY_SERVER_USERNAME = azurerm_container_registry.acr.admin_username
    DOCKER_REGISTRY_SERVER_PASSWORD = azurerm_container_registry.acr.admin_password
 
  }
  # Configure Docker Image to load on start
  site_config {
    linux_fx_version = "DOCKER|${var.registry_name}:${var.tag_name}"
    always_on        = "true"
  }
  identity {
    type = "SystemAssigned"
  }
}

## Outputs
output "app_service_name" {
  value = "${azurerm_app_service.app-service.name}"
}
output "app_service_default_hostname" {
  value = "https://${azurerm_app_service.app-service.default_site_hostname}"
}

Authenticate Terraform with Azure CLI

Sometimes there are no error messages and they’re not helpful at all, but sometimes there are error message which are helpful for your debugging of the issues which are the best thing ever. Then again this is only helpful if the error message points you to the correct problem to fix. I stubbled across an issue recently when I could not add a Secret to an Azure Key Vault via Terraform, which the error message did not help at all.

To paint the picture around where I was at. I had used Terraform to create a Resource Group, Azure Container Instance and a Azure Key Vault. This had all deployed correctly, but the last part was to create a Secret in the Azure Key Vault. However, when doing this I was met with this error below:

Error: Error checking for presence of existing Secret “demo-container-registry-password” (Key Vault “https://demo-kv.vault.azure.net/”): keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 — Original Error: autorest/azure: Service returned an error. Status=403 Code=”Forbidden” Message=”The user, group or application ‘appid=00000000-8ddb-461a-bbee-02f9e1bf7b46;oid=00000000-5015-4074-9780-4907e90957a8;numgroups=1;iss=https://sts.windows.net/00000000-a490-4728-9c9d-1d1446b68e5e/’ does not have secrets get permission on key vault ‘demo-kv;location=uksouth’. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287″ InnerError={“code”:”AccessDenied”}

Now you would think this is to do with permissions, but I am logged in via my user with Owner permissions. Therefore, it couldn’t be permissions, plus I just created all these resources in Azure correctly.

After some intense Googling, I found the issue wasn’t being authenticated but how I was authenticated. There is a particular method to authenticating while using the Azure CLI, and my issue was the subscription I was using was not my default directory. Therefore, I could not access the secret from the default subscription it was using. I am not sure why all other processes worked fine and this didn’t, but sometimes you just don’t question the insanity.

Here is the details from Terraform on authenticating with the Azure CLI correctly: https://www.terraform.io/docs/providers/azurerm/guides/azure_cli.html

For a simple overview of what is said in there, you can follow these simple steps:

Sign in to Azure CLI using the ‘az’ command

az login

Once you are logged in then you can get the subscription details by listing the available subscriptions

az account list

From the response you can see what you have access to, so you can copy the Subscription ID from the response and set the Subscription context.

az account set --subscription="SUBSCRIPTION_ID"

E.g.

az account set --subscription="00000000-0000-0000-0000-000000000000"

After this you should have no issue connecting and executing the Terraform for Azure.

Azure DevOps Pipeline Templates and External Repositories

Working with Azure DevOps you can use YAML to create the build and deployment pipelines. To make this easier and more repeatable you can also use something called templates. However, if you want to use them in multiple repositories you don’t want to repeat yourself. There is a method to get these shared as I will demo below.

When I format my folders for holding the YAML files, I like to mirror how they were built in the UI editor in Azure DevOps website. That is with Tasks like DotNetCli and Group Tasks that are a collection of Tasks to complete a job like Build Dotnet Core Application.

DevOps
–Tasks
—-DotNetCli.yml
–GroupTasks
—-BuildDotnetApp.yml

In this method the ‘BuildDotnetApp.yml’ would inherit the ‘DotNetCli.yml’ and other Group Tasks could also inherit it as well. This makes them more reusable and dynamic, plus easier to upgrade if you need to change a Task version or add a new parameter.

This would be the Dot Net Core CLI Task:

parameters:
  diplayName: 'DotNetCoreCLI'
  projects: ''
  arguments: ''
  command: build
  customScript: ''
  continueOnError: false

steps:
- task: DotNetCoreCLI@2
  displayName: ${{parameters.diplayName}}
  inputs:
    publishWebProjects: false
    command: ${{parameters.command}}
    projects: ${{parameters.projects}}
    arguments: ${{parameters.arguments}}
    zipAfterPublish: false
    custom: ${{parameters.customScript}}
    continueOnError: ${{parameters.continueOnError}}

And can then be called in like below. Remember that the folder path is relative to where this file is hosted.

steps:
- template: ../Tasks/_DotNetCoreCLI.yml
  parameters:
    diplayName: 'Restore .NetCore Projects'
    projects:  '**/MicroServices/**/*.API.csproj'
    arguments: '--packages $(Build.SourcesDirectory)\packages'
    command: restore

- template: ../Tasks/_DotNetCoreCLI.yml
  parameters:
    diplayName: 'Build .NetCore Projects'
    projects:  '**/*.csproj'
    arguments: '--configuration $(BuildConfiguration) --output $(Build.SourcesDirectory)\bin\$(BuildConfiguration)'
    command: build

You can read more on using templates in the Azure DevOps Documentation.
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops

Now we have these great reusable templates, we don’t want them sitting in a multiple repository to then be maintained in multiple times.

The idea here would to move these files to a single repository for example ‘deployment-files’, which will contain all them files to then be referenced later.

The first thing we need to do is reference this new repository in the applications pipeline file. Below is a standard azure pipeline file for building the dotnet application. It has array of stages with the first stage being the CI Build, a single job and the default agent pool.

stages:
  - stage: 'CIBuild'
    displayName: 'CI  Service'
    jobs:
      - job: CI_Service
        displayName: CI Service
        continueOnError: false
        pool:
          displayName: "CI Service"
          name: Default
        workspace:
          clean: all
        timeoutInMinutes: 120
        cancelTimeoutInMinutes: 2
        steps:

To add a reference to another repository you will need to add the following to the top of the file.

This reference will have a alias name, type of repository, location to the repository and a reference to the git branch reference as below.

resources:
  repositories:
    - repository: DeploymentTemplates #alias name
      type: git #type of repository
      name: deployment-files #repository name
      ref: 'refs/heads/main' #git branch reference

This is making a reference to another Azure DevOps Repository in the same Organisation, which might work for some setup, but others might have them in different repositories or different vendors like GitHub. The other alternative to this method above is you might want to get the reference from a Pipeline Artifacts after a build, which you can also do by following the instructions in this documentation. https://docs.microsoft.com/en-us/azure/devops/pipelines/process/resources?view=azure-devops&tabs=schema

With this reference, it means you have access to the repository, but it doesn’t do a git pull as far as I could tell. This might just be for repositories in the same system like Azure DevOps, but it does make things simple as your not download more resources when running the pipeline.

Now you have access to the repository you can call upon the templates in the same method as you normally would with once slight change. You need to reference the file relative to the location it is in the deployment files repository, not the current applications. The other part is you need to add ‘@alias name’ to the end of the path, so it knows where to get the files from. For our example it would look like this.

steps:
- template: DevOps/Tasks/_DotNetCoreCLI.yml@DeploymentTemplates
  parameters:
    diplayName: 'Restore .NetCore Projects'
    projects:  '**/MicroServices/**/*.API.csproj'
    arguments: '--packages $(Build.SourcesDirectory)\packages'
    command: restore

- template: DevOps/Tasks/_DotNetCoreCLI.yml@DeploymentTemplates
  parameters:
    diplayName: 'Build .NetCore Projects'
    projects:  '**/*.csproj'
    arguments: '--configuration $(BuildConfiguration) --output $(Build.SourcesDirectory)\bin\$(BuildConfiguration)'
    command: build

Notice I am not using  the ‘../Task’, but directly referencing the  path ‘DevOps/Task’. Also I have added the ‘@DeploymentTemplates’ to the end of the path.

Here is the full example.

Deployment Files Repository:
Location = ‘DevOps/Tasks’

parameters:
  diplayName: 'DotNetCoreCLI'
  projects: ''
  arguments: ''
  command: build
  customScript: ''
  continueOnError: false

steps:
- task: DotNetCoreCLI@2
  displayName: ${{parameters.diplayName}}
  inputs:
    publishWebProjects: false
    command: ${{parameters.command}}
    projects: ${{parameters.projects}}
    arguments: ${{parameters.arguments}}
    zipAfterPublish: false
    custom: ${{parameters.customScript}}
    continueOnError: ${{parameters.continueOnError}}



Application Repository:
Location = ‘azurepipeline.yml’

resources:
  repositories:
    - repository: DeploymentTemplates #alias name
      type: git #type of repository
      name: deployment-files #repository name
      ref: 'refs/heads/main' #git branch reference
stages:
  - stage: 'CIBuild'
    displayName: 'CI  Service'
    jobs:
      - job: CI_Service
        displayName: CI Service
        continueOnError: false
        pool:
          displayName: "CI Service"
          name: Default
        workspace:
          clean: all
        timeoutInMinutes: 120
        cancelTimeoutInMinutes: 2
	steps:
	- template: DevOps/Tasks/_DotNetCoreCLI.yml@DeploymentTemplates
	  parameters:
	    diplayName: 'Restore .NetCore Projects'
	    projects:  '**/MicroServices/**/*.API.csproj'
	    arguments: '--packages $(Build.SourcesDirectory)\packages'
	    command: restore
	
	- template: DevOps/Tasks/_DotNetCoreCLI.yml@DeploymentTemplates
	  parameters:
	    diplayName: 'Build .NetCore Projects'
	    projects:  '**/*.csproj'
	    arguments: '--configuration $(BuildConfiguration) --output $(Build.SourcesDirectory)\bin\$(BuildConfiguration)'
	    command: build