Get Azure API Management Git Credentials using PowerShell

One of the many great features of Azure API Management is the fact that it has a built in Git repository for storing the current configuration as well as publishing new configurations.

ss_apim_gitrepository

This allows you to push updated Azure API Management configurations to this internal Git repository as a new branch and then Deploy the configuration to API Management.

The internal Git repository in Azure API Management is not intended to be used for a normal development workflow. You’ll still want to develop and store your Azure API management configuration in an external Git repository such as GitHub or TFS/VSTS and then copy configuration updates to the internal Git repository in Azure API Management using some sort of automated process (e.g. Continuous Integration/Continuous Delivery could be adopted for this).

The Internal Git Repository

To access the Internal Git Repository requires short lived (30 days maximum) Git credentials to be generated. This is fairly easy through the Azure API Management portal:

ss_apim_gitrepositorygeneratecreds

Unfortunately using the portal to get these credentials is a manual process and so would not be so good for an automated delivery process (e.g. CI/CD). You’d need to update these Git credentials in your CI/CD automation system every time they expired (every 30 days).

Get Git Credentials

A better approach to generating the Git Credentials is to use Azure PowerShell API Management cmdlets connected with a Service Principal to generate the Git credentials whenever you need them in your CI/CD pipeline.

This is not a completely straightforward process right now (which is unusual for the Azure PowerShell team), so I’ve created a simple PowerShell script that will take care of the nuts and bolts for you.

Requirements

To run this script you’ll need:

  1. PowerShell 5 (WMF 5.0) or greater.
  2. Azure PowerShell Modules installed (make sure you’ve got the latest versions – 4.0.3 at the time of writing this).

You’ll also need to supply the following parameters to the script:

  1. The Azure Subscription Id of the subscription containing the API Management instance.
  2. The name of the Resource Group where the API Management instance is installed to.
  3. The service name of the API Management instance.

You can also optionally supply which of the two internal API Management keys, primary or secondary, to use to generate the credential and also the length of time that the Git credential will be valid for (up to 30 days).

Steps

Download the Script

  1. Download the script Get-AzureRMApiManagementGitCredential.ps1 using the PowerShell command:
  2. Unblock the script using the PowerShell command:

Using the Script

  1. Use the Login-AzureRMAccount cmdlet to authenticate to Azure. This would normally be done using a Service Principal if using an automated process, but could be done interactively when testing.
  2. Execute the script providing the SubscriptionId, ResourceGroup and ServiceName parameters (and optionally the KeyType and ExpiryTimespan) using the following PowerShell command:

ss_apim_gitrepositoryinvoke

The script will return an object containing the properties GitUsername and GitPassword that can be provided to Git when cloning the internal Git repository.

The GitPassword is not escaped so can not be directly used within a Git Clone URL without replacing any / or @ with %2F and %40 respectively.

In the example above I generated an internal Git Credential using the Primary Secret Key that will expire in 4 hours.

Typically you’d assign the output of this script to a variable and use the properties to generate the URL to pass into the Git Clone. For example:

ss_apim_gitrepositoryclone

Tips

  • When cloning the internal Git Repository you’ll need the clone URL of the repository. This is always the name of your Azure API Management instance followed by with scm.azure-api.net appended to it E.g. https://myapimanagementinstance.scm.azure-api.net
  • Once you’ve uploaded a new Git branch containing a new or updated Azure API Management configuration you’ll need to use the Publish-AzureRmApiManagementTenantGitConfiguration cmdlet to tell Azure API Management to publish the configuration contained in the branch. I have not detailed this process here, but if there is interest I can cover the entire end-to-end process.
  • The Primary and Secondary Secret Keys that are used to generate the internal Git Credential can be re-generated (rolled) individually if a Git credential is compromised. However, this will invalidate all Git Credentials generated using that Secret Key.

The Script

If you wish to review the script itself, here it is:

So, hopefully that will be enough information to get anyone else started on building a CI/CD pipeline for deploying Azure API Management configurations.

 

Advertisements

Sonatype Nexus Containers with Persistent Storage in Azure Container Instances

On the back of yesterdays post on running Azure Container Instance containers with persistent storage, I thought I’d try a couple of other containers with my script.

Note: I don’t actually plan on running any of these apps, I just wanted to test out the process and my scripts to identify any problems.

I tried:

And here are the results of my tests:

Sonatype Nexus 2

Works perfectly and the container starts up quickly (under 10 seconds):

ss_aci_sonatypenexus2

I passed the following parameters to the script:

Note: The Nexus 2 server is only accessible on the path /nexus/.

Sonatype Nexus 3

Works perfectly but after takes at least a minute to be accessible after the container starts. But this is normal behavior for Nexus 3.

ss_aci_sonatypenexus3

I passed the following parameters to the script:

Jenkins

Unfortunately Jenkins does not work with a persistent storage volume from an Azure Share. It seems to be trying to set the timestamp of the file that will contain the InitialAdminPassword, which is failing:

ss_aci_jenkins

I passed the following parameters to the script:

So, this is still a little bit hit and miss, but in general Azure Container Instances look like a very promising way to run different types of services in containers without a lot of overhead. With a bit of automation, this could turn out to be a cost effective way to quickly and easily run some common services.

Persistent Storage in Azure Container Instances

Update 2017-08-06: I have improved the script so that it is idempotent (can be run more than once and will only create anything that is missing). The Azure Container Instance resource group can be deleted once you’ve finished with the container and then recreated again with this same script when you next need it. The storage will be preserved in the separate storage account resource group. The script can now be run with the -verbose parameter and will produce much better progress information.

Azure Container Instances (ACI) is a new resource type in Azure that allows you to quickly and easily create containers without the complexity or overhead of Azure Service Fabric, Azure Container Services or provisioning a Windows Server 2016 VM.

It allows you to quickly create containers that are billed by the second from container images stored in Docker Hub or your own Azure Container Registry (ACR). Even though this feature is still in preview, it is very easy to get up and running with it.

But this post isn’t about creating basic container instances, it is about running container instances where some of the storage must persist. This is a basic function of a container host, but if you don’t have access to the host storage then things get more difficult. That said, Azure Container Instances do support mounting Azure File Shares into the container as volumes. It is fairly easy to do, but requires quite a number of steps.

There is some provided documentation for persisting storage in a container instance, but it is quite a manual process and the example ARM templates are currently broken: there are some typos and missing properties. So this post aims to make the whole thing a lot simpler and automatable.

So in this post, I’m going to share a PowerShell function and Azure Resource Manager (ARM) template that will allow you to easily provision an Azure Container Instance with an Azure File Share mounted. The process defaults to installing a GoCD Server container (version 17.8.0 if you’re interested), but you could use it to install any other Linux Container that needs persistent storage. The script is parameterized so other containers and mount points can be specified – e.g. it should be fairly easy to use this for other servers like Sonatype Nexus or Jenkins Server.

Update 2017-08-06: I documented my findings trying out these other servers in my following blog post.

Requirements

To perform this process you will need the following:

  • PowerShell 5.0+ (PowerShell 4.0 may work, but I haven’t tested it).
  • The Azure PowerShell module installed.
  • Created an Application Service Principal – see below.

Azure Service Principal

Before you start this process you will need to have created an Application Service Principal in Azure that will be used to perform the deployment. Follow the instructions on this page to create an application and then get the Service Principal from it.

You will need to record these values as they will be provided to the script later on:

  • Application Id
  • Application Key
  • Tenant Id
  • Subscription Name

The Process

The process will perform the following tasks:

  1. The Service Principal is used to login to Azure to perform the deployment.
  2. An Azure Resource Group is created to contain a Azure Storage Account and Azure Key Vault.
  3. An Azure Storage Account is created and an Azure File Share is created in it.
  4. An Azure Key Vault is created to store the Storage Account Key and make it accessible to the Azure Container Instance.
  5. The Service Principal is granted permission to the Azure Key Vault to read and write secrets.
  6. The key to the Storage Account Key is added as a secret to the Azure Key Vault.
  7. The parameters are set in an ARM Template parameter file.
  8. An Azure Resource Group is created to contain the Azure Container Instance.

The Script

This is the content of the script:

The script requires a four parameters to be provided:

  • ServicePrincipalUsername – the Application Id obtained when creating the Service Principal.
  • ServicePrincipalPassword – the Application Key we got (or set) when creating the Service Principal.
  • TenancyId – The Tenancy Id we got during the Service Principal creation process.
  • SubscriptionName – the name of the subscription to install the ACI and other resources into.

There are also some other optional parameters that can be provided that allow the container image that is used, the TCP port the container listens on and mount point for the Auzre File Share. If you don’t provide these parameters will be used which will create a GoCD Server.

 

  • AppCode – A short code to identify this application. It gets added to the resource names and resource group names. Defaults to ‘gocd’.
  • UniqueCode – this string is just used to ensure that globally unique names for the resources can be created. Defaults to ‘zzz‘.
  • ContainerImage – this is the name and version of the container image to be deployed to the ACI. Defaults to ‘gocd/gocd-server:v17.8.0‘.
  • CPU – The number of cores to assign to the container instance. Defaults to 1.
  • MemoryInGB – The amount of memory (in GB) to assign to the container instance. Defaults to 1.5.
  • ContainerPort – The port that the container listens on. Go CD Server defaults to 8153.
  • VolumeName – this is a volume name that is used to represent the volume in the ARM template. It can really be set to anything. Defaults to ‘gocd‘.
  • MountPoint – this is the folder in the Container that the Azure File Share is mounted to. Defaults to ‘/godata/‘.

ARM Template Files

There are two other files that are required for this process:

  1. ARM templatethe ARM template file that will be used to install the ACI.
  2. ARM template parameters – this file will be used to pass in the settings to the ARM Template.

ARM Template

This file is called aci-azuredeploy.json and should be downloaded to the same folder as the script above.

ARM Template Parameters

This file is called aci-azuredeploy.parameters.json and should be downloaded to the same folder as the script above.

Steps

To use the script the following steps need to be followed:

  1. Download the three files above (the script and the two ARM template files) and put them into the same folder:ss_aci_filesrequires
  2. Open a PowerShell window.
  3. Change directory to the folder you place the files into by executing:
  4. CD <folder location>
  5. Execute the script like this (passing in the variables):ss_aci_executingscript
  6. The process will then begin and make take a few minutes to complete:ss_aci_creategocdNote: I’ve changed the keys to this service principal and deleted this storage account, so I using these Service Principal or Storage Account keys won’t work!
  7. Once completed you will be able to log in to the Azure Portal and find the newly created Resource Groups:ss_aci_resourcegroup
  8. Open the resource group *gocdacirg and then select the container group *gocdaci:ss_aci_getcontainerip
  9. The IP Address of the container is displayed. You can copy this and paste it into a browser window along with the port the container exposed. In the case of Go CD it is 8153:ss_aci_runninggocdserver
  10. The process is now completed.

The Azure Container Instance can now be deleted and recreated at will, to reduce cost or simply upgrade to a new version. The Azure File Share will persist the data stored by the container into the mounted volume:

ss_aci_storageexplorerfileshare

Hopefully this process will help you implement persisted storage containers in Azure Container Instances more easily and quickly.

Thanks for reading!

Publish an Azure RM Web App using a Service Principal in PowerShell

Introduction

Deploying an Azure Web App is almost stupidly simple. If I were to list the methods and tools I’d still be typing next week. The problem with many of these tools and process is that they do a whole lot of magic under the hood which makes the process difficult to manage in source control.

I’m a big believer that all code (including deployment code) should be in the application source repository so it can be run by any tool or release pipeline – including manually by development teams. This ensures that whatever deployment process is used, it is the same no matter who or what runs it – and we end up continuously testing the deployment code and process.

So I decided to go and find out how to deploy an Azure Web App using PowerShell using an Service Principal.

Where is Publish-AzureRMWebsiteProject?

If you look through the Azure PowerShell cmdlets you’ll find a service manager one called Publish-AzureWebsiteProject. This cmdlet looks like it should do the trick, but it isn’t suitable because it requires authentication by a user account instead of a service principal.

Only service principal accounts can be authenticated using automation. Therefore using Publish-AzureWebsiteProject would only work if a development team member was able to interactively login– which would prevent the same process being used for automation or our continuous delivery pipeline. The newer Azure Resource Manager cmdlets (*-AzureRM*) all support a login using a service principal, but the problem is that there is no Publish-AzureRMWebsiteProject cmdlet.

So, to work around this limitation I determined I had to use Web Deploy/MSDeploy. The purpose of this post is to share the PowerShell function/code and process I used to do this. This will work with and without Web App deployment slots.

Note: in my case our teams put all deployment code into a PowerShell PSake task in the application source code repository to make it trivial for anyone to run the deployment. The continuous delivery pipeline was also able to call the exact same task to perform the deployment. There is no requirement to use PowerShell PSake – just a simple PowerShell script will do.

The Code

So, I’ll start by just pasting the function that does performs the task:

Just save this file as Publish-AzureRMWebappProject.ps1 and you’re ready to start publishing (almost).

Before you can use this function you’ll need to get a few things sorted:

  1. Create a Service Principal with a password to use to deploy the web app using the instructions on this page.
  2. Make sure you have got the latest version of the Azure PowerShell Modules installed (I used v4.0.0). See this page for instructions.
  3. Make sure you’ve got MSDeploy.exe installed on your computer – see this page for instructions. You can pass the path to MSDeploy.exe into the Publish-AzureRMWebappProject.ps1 using the MSDeployPath parameter.
  4. Gather the following things (there are many ways of doing that – but I’ll leave it up to you to figure out what works for you):
    1. the Subscription Id of the subscription you’ll be deploying to.
    2. the Tenant Id of the Azure Active Directory containing your Service Principal.
    3. the Application Id that was displayed to you when you created the Service Principal.
    4. the Password you assigned when you created the Service Principal.

Once you have got all this information you can call the script above like this:

Note: You’ll need to make sure to replace the variables $SubscriptionId, $TenantId, $Password and $Username with the values for your Azure Subscription, Tenancy and Service Principal.

When everything is done correctly this is what happens when you run it (with -Verbose enabled):

ss_webappdeploy_publishazurermwebappproject

Note: in the case above I was installing to a deployment staging slot called offline, so the new version of my website wouldn’t have been visible in my production slot until I called the Swap-AzureRmWebAppSlot cmdlet to swap the offline slot with my production slot.

All in all, this is fairly robust and allows our development teams and our automation and continuous delivery pipeline to all use the exact same deployment code which reduces deployment failures.

If you’re interested in more details about the code/process, please feel free to ask questions.

Thanks for reading.

Using Azure Key Vault with PowerShell – Part 1

Azure Key Vault is used to safeguard and manage cryptographic keys, certificates and secrets used by cloud applications and services (you can still consume these on-premise though). This allows other application, services or users in an Azure subscription to store and retrieve these cryptographic keyscertificates and secrets.

Once cryptographic keys, certificates and secrets have been stored in a Azure Key Vault access policies can be configured to provide access to them by other users or applications.

Azure Key Vault also stores all past versions of a cryptographic key, certificate or secret when they are updated. So this allows easily rolling back if anything breaks.

This post is going to show how:

  1. Set up an Azure Key Vault using the PowerShell Azure Module.
  2. Set administration access policies on the Azure Key Vault.
  3. Grant other users or applications access to cryptographic keyscertificates or secrets.
  4. Add, retrieve and remove a cryptographic key from the Azure Key Vault.
  5. Add, retrieve and remove a secret from the Azure Key Vault.

Requirements

Before getting started there is a few things that will be needed:

  1. An Azure account. I’m sure you’ve already got one, but if not create a free one here.
  2. The Azure PowerShell module needs to be installed. Click here for instructions on how install it.

Install the Key Vault

The first task is to customize and install the Azure Key Vault using the following PowerShell script.

But first, the variables in the PowerShell script need to be customized to suit. The variables in the PowerShell script that needs to be set are:

  • $subscriptionName – the name of the Azure subscription to install the Key Vault into.
  • $resourceGroupName – the name of the Resource Group to create to contain the Key Vault.
  • $keyVaultName – the name of the Key Vault to create.
  • $location – the Azure data center to install the Key Vault to (use Get-AzureRMLocation to get a list of available Azure data centers).
  • $keyVaultAdminUsers – an array of users that will be given administrator (full control over cryptographic keys, certificates and secrets). The user names specified must match the full name of users found in the Azure AD assigned to the Azure tenancy.

ss_akv_create

It will take about 30 seconds for the Azure Key Vault to be installed. It will then show up in the Azure Subscription:

ss_akv_createcompleteportal

Assigning Permissions

Once the Azure Key Vault is setup and an administrator or two have been assigned, other access policies will usually need to be assigned to users and/or application or service principal.

To create an access policy to allow a user to get and list cryptographic keys, certificates and secrets if you know the User Principal Name:

Note: the above code assumes you still have the variables set from the ‘Install the Key Vault’ section.

If you only have the full name of the user then you’ll need to look up the Object Id for the user in the Azure AD:

Note: the above code assumes you still have the variables set from the ‘Install the Key Vault’ section.

To create an access policy to allow a service principal or application to get and list cryptographic keys if you know the Application Id (a GUID):

Note: the above code assumes you still have the variables set from the ‘Install the Key Vault’ section.

Changing the values of the PermissionsToKeys, PermissionsToCertificates and PermissionsToSecrets parameters in the cmdlets above allow different permissions to be set for each policy.

The available permissions for certificates, keys and secrets are:

An access policy can be removed from users or service principals using the Remove-AzureRmKeyVaultAccessPolicy cmdet:

Note: the above code assumes you still have the variables set from the ‘Install the Key Vault’ section.

Working with Secrets

Secrets can be created, updated, retrieved and deleted by users or applications that have been assigned with the appropriate policy.

Creating/Updating Secrets

To create a new secret, use the Set-AzureKeyVaultSecret cmdlet:

Note: the above code assumes you still have the variables set from the ‘Install the Key Vault’ section.

This will create a secret called MyAdminPassword with the value P@ssword!1 in the Azure Key Vault.

The secret can be updated to a new value using the same cmdlet:

Additional parameters can also be assigned to each version of a secret to control how it can be used:

  • ContentType – the type of content the secret contains (e.g. ‘txt’)
  • NotBefore – the date that the secret is valid after.
  • Expires – the date the secret is valid until.
  • Disable – marks the secret as disabled.
  • Tag – assigns tags to the secret.

For example:

ss_akv_secretupdatewithparameters

Retrieving Secrets

To retrieve the latest (current) version of a secret, use the Get-AzureKeyVaultSecret cmdlet:

This will assign the stored secret to the variable $secretText as a SecureString. This can then be passed to any other cmdlets that require a SecureString.

To list all the versions of a secret, add the IncludeVersions parameter:

ss_akv_secretallhistory

To retrieve a specific version of a secret, use the Get-AzureKeyVaultSecret cmdlet with the Version parameter specified:

Removing Secrets

Finally, to remove a secret use the Remove-AzureKeyVaultSecret cmdlet:

That pretty much covers managing and using secrets in Azure Key Vault using PowerShell.

Cryptographic keys and Certificates

In the next part of this series I’ll cover using Azure Key Vault to use and manage cryptographic keys and certificates. Thanks for sticking with me this far.