Azure DevOps Pipeline Templates and External Repositories

Working with Azure DevOps you can use YAML to create the build and deployment pipelines. To make this easier and more repeatable you can also use something called templates. However, if you want to use them in multiple repositories you don’t want to repeat yourself. There is a method to get these shared as I will demo below.

When I format my folders for holding the YAML files, I like to mirror how they were built in the UI editor in Azure DevOps website. That is with Tasks like DotNetCli and Group Tasks that are a collection of Tasks to complete a job like Build Dotnet Core Application.

DevOps
–Tasks
—-DotNetCli.yml
–GroupTasks
—-BuildDotnetApp.yml

In this method the ‘BuildDotnetApp.yml’ would inherit the ‘DotNetCli.yml’ and other Group Tasks could also inherit it as well. This makes them more reusable and dynamic, plus easier to upgrade if you need to change a Task version or add a new parameter.

This would be the Dot Net Core CLI Task:

parameters:
  diplayName: 'DotNetCoreCLI'
  projects: ''
  arguments: ''
  command: build
  customScript: ''
  continueOnError: false

steps:
- task: DotNetCoreCLI@2
  displayName: ${{parameters.diplayName}}
  inputs:
    publishWebProjects: false
    command: ${{parameters.command}}
    projects: ${{parameters.projects}}
    arguments: ${{parameters.arguments}}
    zipAfterPublish: false
    custom: ${{parameters.customScript}}
    continueOnError: ${{parameters.continueOnError}}

And can then be called in like below. Remember that the folder path is relative to where this file is hosted.

steps:
- template: ../Tasks/_DotNetCoreCLI.yml
  parameters:
    diplayName: 'Restore .NetCore Projects'
    projects:  '**/MicroServices/**/*.API.csproj'
    arguments: '--packages $(Build.SourcesDirectory)\packages'
    command: restore

- template: ../Tasks/_DotNetCoreCLI.yml
  parameters:
    diplayName: 'Build .NetCore Projects'
    projects:  '**/*.csproj'
    arguments: '--configuration $(BuildConfiguration) --output $(Build.SourcesDirectory)\bin\$(BuildConfiguration)'
    command: build

You can read more on using templates in the Azure DevOps Documentation.
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops

Now we have these great reusable templates, we don’t want them sitting in a multiple repository to then be maintained in multiple times.

The idea here would to move these files to a single repository for example ‘deployment-files’, which will contain all them files to then be referenced later.

The first thing we need to do is reference this new repository in the applications pipeline file. Below is a standard azure pipeline file for building the dotnet application. It has array of stages with the first stage being the CI Build, a single job and the default agent pool.

stages:
  - stage: 'CIBuild'
    displayName: 'CI  Service'
    jobs:
      - job: CI_Service
        displayName: CI Service
        continueOnError: false
        pool:
          displayName: "CI Service"
          name: Default
        workspace:
          clean: all
        timeoutInMinutes: 120
        cancelTimeoutInMinutes: 2
        steps:

To add a reference to another repository you will need to add the following to the top of the file.

This reference will have a alias name, type of repository, location to the repository and a reference to the git branch reference as below.

resources:
  repositories:
    - repository: DeploymentTemplates #alias name
      type: git #type of repository
      name: deployment-files #repository name
      ref: 'refs/heads/main' #git branch reference

This is making a reference to another Azure DevOps Repository in the same Organisation, which might work for some setup, but others might have them in different repositories or different vendors like GitHub. The other alternative to this method above is you might want to get the reference from a Pipeline Artifacts after a build, which you can also do by following the instructions in this documentation. https://docs.microsoft.com/en-us/azure/devops/pipelines/process/resources?view=azure-devops&tabs=schema

With this reference, it means you have access to the repository, but it doesn’t do a git pull as far as I could tell. This might just be for repositories in the same system like Azure DevOps, but it does make things simple as your not download more resources when running the pipeline.

Now you have access to the repository you can call upon the templates in the same method as you normally would with once slight change. You need to reference the file relative to the location it is in the deployment files repository, not the current applications. The other part is you need to add ‘@alias name’ to the end of the path, so it knows where to get the files from. For our example it would look like this.

steps:
- template: DevOps/Tasks/_DotNetCoreCLI.yml@DeploymentTemplates
  parameters:
    diplayName: 'Restore .NetCore Projects'
    projects:  '**/MicroServices/**/*.API.csproj'
    arguments: '--packages $(Build.SourcesDirectory)\packages'
    command: restore

- template: DevOps/Tasks/_DotNetCoreCLI.yml@DeploymentTemplates
  parameters:
    diplayName: 'Build .NetCore Projects'
    projects:  '**/*.csproj'
    arguments: '--configuration $(BuildConfiguration) --output $(Build.SourcesDirectory)\bin\$(BuildConfiguration)'
    command: build

Notice I am not using  the ‘../Task’, but directly referencing the  path ‘DevOps/Task’. Also I have added the ‘@DeploymentTemplates’ to the end of the path.

Here is the full example.

Deployment Files Repository:
Location = ‘DevOps/Tasks’

parameters:
  diplayName: 'DotNetCoreCLI'
  projects: ''
  arguments: ''
  command: build
  customScript: ''
  continueOnError: false

steps:
- task: DotNetCoreCLI@2
  displayName: ${{parameters.diplayName}}
  inputs:
    publishWebProjects: false
    command: ${{parameters.command}}
    projects: ${{parameters.projects}}
    arguments: ${{parameters.arguments}}
    zipAfterPublish: false
    custom: ${{parameters.customScript}}
    continueOnError: ${{parameters.continueOnError}}



Application Repository:
Location = ‘azurepipeline.yml’

resources:
  repositories:
    - repository: DeploymentTemplates #alias name
      type: git #type of repository
      name: deployment-files #repository name
      ref: 'refs/heads/main' #git branch reference
stages:
  - stage: 'CIBuild'
    displayName: 'CI  Service'
    jobs:
      - job: CI_Service
        displayName: CI Service
        continueOnError: false
        pool:
          displayName: "CI Service"
          name: Default
        workspace:
          clean: all
        timeoutInMinutes: 120
        cancelTimeoutInMinutes: 2
	steps:
	- template: DevOps/Tasks/_DotNetCoreCLI.yml@DeploymentTemplates
	  parameters:
	    diplayName: 'Restore .NetCore Projects'
	    projects:  '**/MicroServices/**/*.API.csproj'
	    arguments: '--packages $(Build.SourcesDirectory)\packages'
	    command: restore
	
	- template: DevOps/Tasks/_DotNetCoreCLI.yml@DeploymentTemplates
	  parameters:
	    diplayName: 'Build .NetCore Projects'
	    projects:  '**/*.csproj'
	    arguments: '--configuration $(BuildConfiguration) --output $(Build.SourcesDirectory)\bin\$(BuildConfiguration)'
	    command: build


Azure REST API Scopes

When working with the Azure REST API you need to provide the scope in all API requests, so Azure knows where you are looking. However, throughout their documentation that although they ask for the scope they do not explain or link to an explanation of what a scope is and what the formats are. Therefore, I have collected them and got a simple explanation for each of them.

As mentioned above, the scope is like a search filter and also part of the permissions. For example, if you was getting a list of Resources you might use the Resource Group scope to get only them Resources, or you might go for the Subscription scope to get all Resources in the requested Subscription. This might also be due to permissions, if the Service Principle account you are using doesn’t have access to the whole Tenant, but does to specific Subscriptions.

Scopes

Subscription scope

subscriptions/{subscriptionId}

Example:

subscriptions/d7f90b53-af20-4061-8206-f05e31852a44

Resource Group scope

subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}

Example:

subscriptions/d7f90b53-af20-4061-8206-f05e31852a44/resourceGroups/my-rg-2020

Providers scope

These scopes can vary depending on what the scope is for. For example this is the scope for the Billing Account:

providers/Microsoft.Billing/billingAccounts/{billingAccountId}

You can find all of the scopes by following https://docs.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations

Tenant scope

The Tenant scope is easist, as you just don’t put anything. For example getting a list of Role Definitions

The official URL is:

GET https://management.azure.com/{scope}/providers/Microsoft.Authorization/roleDefinitions?api-version=2015-07-01

But to get the Role Definitions from the Tenant Level and below, you just remove the scope segment:

GET https://management.azure.com/providers/Microsoft.Authorization/roleDefinitions?api-version=2015-07-01

Deleting Recovery Service Vault Backup Server (MABS)

This is one of them quick shares to try help the world of Azure developers just a little bit.

While doing some Azure Development using the Microsoft Azure REST API, I got to a point to working in the Azure Recovery Service Vault. I was trying to delete a Back Up Server Agent using the API, which after a little bit of a hunt I found it is actually called ‘Registered Identities

This of course only has one endpoint to call, so it must just be that easy…
However when calling this API I would just get the following:

{
"error":
{
"code":"ServiceContainerNotEmptyWithBackendMessage",
"message":" There are backup items associated with the server. Stop protection and delete backups for each of the backup items to delete the server's registration. If the issue persists, contact support. Refer to https://aka.ms/DeleteBackupItems for more info. ",
"target":null,
"details":null,
"innerError":null
}
}

This was odd as I couldn’t find anything to imply a reason for this error. After then investigating the network traffic while deleting the resource in the Azure Portal, I found a slight difference which also fixed the issue. This was the version of the API.

In the documentation it is stated as ‘2016-06-01’ , but in the network traffic it had version ‘2019-05-13’. If you now change:

DELETE https://management.azure.com/Subscriptions/77777777-d41f-4550-9f70-7708a3a2283b/resourceGroups/BCDRIbzRG/providers/Microsoft.RecoveryServices/vaults/BCDRIbzVault/registeredIdentities/dpmcontainer01?api-version=2016-06-01

To

DELETE https://management.azure.com/Subscriptions/77777777-d41f-4550-9f70-7708a3a2283b/resourceGroups/BCDRIbzRG/providers/Microsoft.RecoveryServices/vaults/BCDRIbzVault/registeredIdentities/dpmcontainer01?api-version=2019-05-13

<p value="<amp-fit-text layout="fixed-height" min-font-size="6" max-font-size="72" height="80">It will now all work.It will now all work.

How to get Azure Backup Protection Containers?

When working with the Azure REST API, I had the requirement to get the Recovery Service Vaults Backup Protection Containers. Naturally, like many, I went to the Microsoft Azure REST API documentation website to learn how to call this API, but when following it I hit a few issues that was not clear and not part of the documentation. Here below are some of the issues I had, that I hope can solve yours… If your having any.

This is the link to the documentation if you don’t have it already to follow.
https://docs.microsoft.com/en-gb/rest/api/backup/backupprotectioncontainers/list

Get All Doesn’t Work

As you can see from the documentation you should be able to call the GET request for:

GET https://management.azure.com/Subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.RecoveryServices/vaults/{vaultName}/backupProtectionContainers?api-version=2016-12-01

This should be a non-filtered list of all the containers not matter their type. However, when calling this you will get the error saying you are missing required properties. With such a helpful message response, how could you not understand what is going wrong.

What I have found is you need to use the filter request, but it is still not that simple. From the example in the documentation you can use the Backup Management Type. However, if like me you want all the containers no matter the type then this isn’t the most helpful, so looking into the OData filters documentation you can use many other requests.

Instead of doing backupManagementType eq ‘AzureWorkload’ which is Backup Management Type equals Azure Workload, you can use ‘ne’ that is not equal to. With this you can put some random text in the string to say Get All Backup Protection Containers Where Backup Management Type Does Not Equal RandomWords. Makes sense yes? Well not for this API. This still results in the same error. I even tried to change the parameter it is searching on to something all valid containers should have like ‘registrationStatus’, but that didn’t work.

The only method I got this API call to work was to call it with the filter of Backup Management Type.

Trying to reduce code I added the Backup Models from the Nuget Package Microsoft.Azure.Management.RecoveryServices.Backup

Azure C# SDK missing options

As stated above you need to use the filter with Backup Management Type to get the Protected Containers, so to get all of them we need to call each type. As I don’t know all the types,  I found the Enum BackupManagementTypes in the above Nuget package that I can then loop through.

However, I am not sure if it is just out of date or that the Nuget package is in preview, but no all the types are in there. Therefore, I created my own using the values in the API Documentation.

/// <summary>
/// Azure Backup Management Types 
/// https://docs.microsoft.com/en-gb/rest/api/backup/backupprotectioncontainers/list#backupmanagementtype
/// </summary>
        public enum BackupManagmentTypes
        {
            AzureBackupServer,
            AzureIaasVM,
            AzureSql,
            AzureStorage,
            AzureWorkload,
            DPM,
            DefaultBackup,
            Invalid,
            MAB
        }

Then it can be  used like this:

foreach (var backupManagementTypeStr in Enum.GetValues(typeof(BackupManagmentTypes)))
{
	var urlWithFilter = $"{containerUrl}&$filter=backupManagementType eq '{backupManagementTypeStr}'";
}

Azure C# SDK incorrect format

With the Nuget package I thought I could the Desterilise the response into the class ProtectionContainerListResponse as this is what the documentation says the type is, but it failed as the model of this class follows ‘response.ItemList.ProtectionContainers’, whereas the expected response follows ‘response.values’

When looking at the response in the API Documentation I have also seen a pattern to the response types for the Recovery Service API. They all have a list with ‘nextlink’ and ‘values’, which values is a list of the response type. However, the types in the list also follow the same pattern with all the properties the same except what type the property ‘properties’ is. Therefore, you can make a generic response as below:

  /// <summary>
        /// API Response
        /// </summary>
        public class ResourceResponse<T> where T : class
        {
            /// <summary>
            /// The uri to fetch the next page of resources. Call ListNext() fetches next page of resources.
            /// </summary>
            public string nextLink { get; set; }
            /// <summary>
            /// List of resources.
            /// </summary>
            public List<ResourceDetails<T>> value { get; set; }
        }

        /// <summary>
        /// Resource Details
        /// </summary>
        public class ResourceDetails<T> where T : class
        {
            /// <summary>
            /// Optional ETag.
            /// </summary>
            public string eTag { get; set; }
            /// <summary>
            /// Resource Id represents the complete path to the resource.
            /// </summary>
            public string id { get; set; }
            /// <summary>
            /// Resource location.
            /// </summary>
            public string location { get; set; }
            /// <summary>
            /// Resource name associated with the resource.
            /// </summary>
            public string name { get; set; }
            /// <summary>
            /// resource properties
            /// </summary>
            public T properties { get; set; }
            /// <summary>
            /// Resource tags.
            /// </summary>
            public object tags { get; set; }
            /// <summary>
            /// Resource type represents the complete path of the form Namespace/ResourceType/ResourceType/...
            /// </summary>
            public string type { get; set; }
        }

Setting Bearer tokens in PowerShell

This is a quick post to put out how to set a Bearer Token using PowerShell for Invoke-RestMethod and Invoke-WebRequest as it was something I could not find a clear explaining post.

This is simply set in the headers of the request as below, where ‘$bearer_token’ is the token’s variable. I have put it in the ‘$headers’ variable, which is then used in the Invoke-RestMethod.

$headers = @{Authorization = "Bearer $bearer_token"}
$response = Invoke-RestMethod -ContentType "$contentType" -Uri $url -Method $method -Headers $headers -UseBasicParsing