Use Pester to Test Azure Resource Manager Templates for Best Practices

Recently I came across the amazing Secure DevOps Kit for Azure (AzSK). This contains a really useful AzSK PowerShell Module that contains cmdlets for performing different types of security scanning on Azure Resources, Subscriptions and Resource Manager Templates.

The feature of this module that I was most interested in for my current project was being able to scan ARM templates for best practice violations. The module contains several

To install the module, open a PowerShell Window and run:

Important: At the time of writing this post, the AzSK module has dependencies on the AzureRM.Profile and other AzureRM.* PowerShell modules. As of December 2018, the AzureRM.* PowerShell Modules are going to be renamed to Az.* (see this post). The AzureRM and Az modules can not be installed side-by-side, so if you’ve installed the Az PowerShell modules on your system then the installation of AzSK will fail because the AzureRM modules will also be installed and a conflict will occur.

The cmdlet we’re most interested in is the Get-AzSKARMTemplateSecurityStatus. It can be used to scan one or more ARM templates or entire folders of ARM templates for best practice violations:


This will scan the ARM templates and produce a CSV report in a folder Microsoft\AzSKLogs\ARMChecker within your $ENV:LOCALAPPDATA folder and open the folder in Explorer. This isn’t ideal for automation scenarios or using during Continuous Integration or Continuous Delivery pipelines. I’ve raised an issue with the AzSK team on GitHub to see if this can be improved.

In my case, I wanted to be able to use the PowerShell Pester Module, a PowerShell testing framework, to execute tests on the output and then use the nUnit output Pester generates to publish into a Continuous Integration pipeline. To do that I needed to create a custom test script that would take the CSV report, count the failures of each level (High, Medium or Low) and fail if any are counted in the specific level.

This is what the script looks like:

You can download the script from GitHub Gist directly or get it from the PowerShell Gallery by running:

Install-Script -Name AzSKARMTemplateSecurityStatus.Test

To use it you will need to install Pester 4.3.0 and AzSK 3.6.1 modules:

Once that is done, you can use Invoke-Pester and pass in the TemplatePath and Severity parameters to the test script:

This will execute the Pester tests in the file above on the specified ARM template. The tests will fail when there are any best practice violations with the specified Severity or above. If you didn’t pass in a Severity then it will default to failing on Medium and High.


If you use the OutputFile and OutputFormat parameters to cause Pester to output an NUnit format file that most Continuous Integration tools will happily accept and use to display the output of the tests.

If you installed the script from the PowerShell Gallery, you can also run the tests like this:

AzSKARMTemplateSecurityStatus.Test -TemplatePath D:\101-webapp-basic-windows\azuredeploy.json

Finally, if you’re using Azure DevOps, you can also get this function as part Secure DevOps Kit (AzSK) CICD Extensions for Azure in the Azure DevOps Marketplace.

Which ever way you choose to consume AzSK, it is a great module and well worth including in your CI/CD pipelines to ensure your ARM templates meet best practices.



Test Website SSL Certificates Continuously with PowerShell and Pester

One of the most common problems that our teams deal with is ensuring that SSL certificates are working correctly. We’ve all had that urgent call in telling us that the web site is down or some key API or authentication function is offline – only to find out it was caused by an expired certificate.

An easy way of preventing this situation would have been to set up a task that continuously tests your SSL endpoints (internal and external web apps and sites, REST API’s etc.) and warns us if:

  • The certificate is about to expire (with x days).
  • The SSL endpoint is using safe SSL protocols (e.g. TLS 1.2).
  • The certificate is using SHA256.

This seemed like a good task for Pester (or Operation Validation Framework). So, after a bit of digging around I found this awesome blog post from Chris Duck showing how to retrieve the certificate and SSL protocol information from an SSL endpoint using PowerShell.

Chris’ post contained this PowerShell cmdlet:

So that was the hard part done, all I needed was to add this function to some Pester tests.

Note: If you are running these tests on an operating system older than Windows 10 or Windows Server 2016 then you will need to install the PowerShell Pester module by running this command in an Administrator PowerShell console:

Install-Module -Name Pester

So after a little bit of tinkering I ended up with a set of tests that I combined into the same file as Chris’ function from earlier. I called the file SSL.tests.ps1. I used the file extension .tests.ps1 because that is the file extension Pester looks for when it runs.

The tests are located at the bottom of the file below the Test-SslProtocol function.

So, now to test these SSL endpoints all I need to do is run in a PowerShell console with the current folder set to the folder containing my SSL.tests.ps1 file:

cd C:\SSLTests\

This is the result:


This shows that all the SSL endpoint certificates being used by, and are all valid SHA-256 certificates and aren’t going to expire in 14 days.

All I would then need to do is put this in a task to run every hour or so and perform some task when the tests fail:

At this point you will still need to use some mechanism to notify someone when they fail. One method could be to write an event into the Windows Event Log and then use Microsoft Operations Management Suite (or SCOM) to monitor for this event and send an e-mail or other alert to the appropriate administrators.

For an example showing how to use OMS to monitor custom events created by failed Pester and OVF tests, see my previous article here.

Potential Improvements

There are a number of ways you could go about improving this process, which our teams have in fact implemented. If you’re considering implementing this process then you might want to also consider them:

  1. Put the Test-SSLProtocol cmdlet into a PowerShell Module that you can share easily throughout your organization.
  2. Put your tests into source control and have the task clone the tests directly from source control every time they are run – this allows tests to be stored centrally and can be change tracked.
  3. Parameterize the tests so that you don’t have to hard code the endpoints to test in the script file. Parameters can be passed into Pester tests fairly easily.
  4. Use something like Jenkins, SCOM or Splunk to run the tests continuously.
  5. Run the tests in an Azure Automation account in the Cloud.

Really, the options for implementing this methodology are nearly limitless. You can engineer a solution that will work for you and your teams, using whatever tools are at your disposal.

At the end of the day, the goal here should be:

  • Reduce the risk that your internal or external applications or websites are using bad certificates.
  • Reduce the risk that an application or website will be deployed without a valid certificate (write infrastructure tests before you deploy your infrastructure – TDD for operations).
  • Reduce the risk you’ll get woken up in the middle of the night with an expired certificate.

So, in this holiday season, I hope this post helps you ensure your certificates won’t expire in the next two weeks and you won’t get called into fix a certificate problem when you should be lying on a beach in the sun (in the southern hemisphere anyway).

Have a good one!

Creating Professional DSC Resources – Part 6

The purpose of this series of articles is to try and document a few of the lessons I learned while releasing new DSC resources as well as contributing to the existing Microsoft Community DSC resources. These articles are not intended to tell you how to write DSC resources from a programming perspective, but to give you some ideas on what might be expected of a DSC resource you’re releasing to the public. For example, unit and integration tests (don’t worry if you aren’t familiar with those terms).

These articles are also not intended to tell you what you must do to release your resource, but more document what will help your resource be easier to use and extend by other people. Some of these these things are obvious for people who have come from the development community, but may be quite new to operations people.

If you missed any previous articles you can find them here:



In the last couple of articles I covered the importance of automated testing and covered unit testing in particular (I’ll get to integration testing later). I had covered creating new unit tests using the unit test templates that are available here (although they will probably move here). I also covered how to complete the Pester Test Initialization and the Get-TargetResource and Set-TargetResource function areas of the unit test.


Unit Testing Completion

The final task in completing the unit tests is to complete the Set-TargetResource  in tests and also optionally tests for any other supporting functions your DSC Resource may have required.

In these unit tests I am using a DSC Resource for creating iSCSI Virtual Disks to illustrate the process. You don’t need to know anything about iSCSI Virtual Disks to understand these articles or resources, but if you’re interested to know the cmdlets I’m using for these, see this page. I’m using the *_iSCSIVirtualDisk cmdlets in this DSC Resource.

Function Test-TargetResource

This area will contain the actual Pester tests that test the Test-TargetResource function. These are fairly similar to the Set-TargetResource except we will be checking these two things:

  1. The output of the Test-TargetFunction is correct. E.g. it returns false if changes are required, which will cause Set-TargetFunction to be called.
  2. The expected Mocks are called by the Function.

This area may contain a large number of tests depending on the complexity of your DSC Resource. In most cases, you should expect there to create the tests from the following list, but often you will need even more for 100% code coverage:

  • Does the function return false when the resource being configured does exist and should, but one of the configured parameters does not match the current values? This test is usually repeated for each parameter in the DSC Resource.
  • Does the function return true when the resource being configured does exist and should, and all the configured parameters match the current values?
  • Does the function return false when the resource being configured does not exist but should?
  • Does the function return false when the resource being configured does exist but should not?
  • Does the function return true when the resource being configured does not exist and should not?

The bottom four of these tests are very similar. So I’ll only show examples of the top two contexts here.

Context ‘Virtual Disk exists and should but has a different …’

In this scenario we Mock the Get-iSCSIVirtualDisk cmdlet to return the object we defined in the Pester Test Initialization section. This is the behavior we’d expect if the resource being configured does exist:

This context will perform two tests:

  1. Should return false – The Test-TargetResource should return false because we are changing the Description parameter so that the resource will require changes (e.g. Set-TargetResource should be called).
  2. Should call the expected mocks – The Test-TargetResource should call the mocked cmdlets the expected number of times. In all contexts in this function this will always be just once.

The purpose of cloning the $TestVirtualDisk object is so we can modify the properties to simulate a property difference without modifying the $TestVirtualDisk object.

You should expect to repeat this context for each parameter that might be different.


Context ‘Virtual Disk does not exist but should’

In this scenario we Mock the Get-iSCSIVirtualDisk cmdlet to return nothing. This is the behavior we’d expect if the resource being configured does not exist:

As you can see, there is not too much different with these tests and you shouldn’t have any problems figuring out the remaining ones. Just remember, the goal is always to get 100% code coverage.


Unit Testing Supporting Functions

It is quite common that you might have implemented some supporting functions in your DSC Resource. These supporting functions are usually called by your standard *-TargetResource functions. If that is the case there are two important things you should do:

  1. Write unit tests that cover all code paths in your supporting functions.
  2. Add mocks to your *-TargetResource unit tests that prevent any constructive/destructive cmdlets that exist in your supporting functions from being called.

The first item is fairly self explanatory. For example, I often implement a get-* function in my DSC Resources which is used to pull the actual objects that will be used by the *-TargetResource functions (e.g. Get-VirtualDisk):

To unit test this function I’d write unit tests that tested the following contexts:

  1. Context ‘Virtual Disk does not exist’
  2. Context ‘Virtual Disk does exist’

For example:

As you can see, there isn’t much to it.

Earn a Chocolate Fish: If you look at the above supporting function and unit tests carefully, you’ll notice that I haven’t got 100% code coverage on it!

To get 100% code coverage I would have had to implement a unit test that covered the situation where the Get-iSCSIVirtualDisk function threw an exception that wasn’t a [Microsoft.Iscsi.Target.Commands.IscsiCmdException] exception.

In case you’re wondering, the Get-iSCSIVirtualDisk function throws a [Microsoft.Iscsi.Target.Commands.IscsiCmdException] when the cmdlet is called with the path parameter set to a path that does not contain a valid iSCSI Virtual Hard Disk file.


Unit Testing Exceptions

When creating unit tests you’ll often need to test a scenario where the function that is being tested is expected to throw an exception. If you read the Pester documentation, you’d might write a test for an exception like this:

This would of course will work. It will ensure that the code throws an exception in this situation. The problem is we aren’t really sure if it is the exception that we expected it to throw. It could have been thrown by some other part of our code.

So to improve on this we need to do things:

  1. Customize the exception that is thrown.
  2. Change the unit test so that it checks for the customized exception.


Customize the Exception

To create a custom exception we need to create a new exception object containing our custom error message. The exception object is then used to create a custom Error Record:

In the above code, you just need to customize the $errorId and $errorMessage variables. The $errorId should just contain a simply string identifier for this particular type of error, but the $errorMessage can contain a full description of the error, including related parameters.

Once you’ve created the $errorRecord object you can call the ThrowTerminatingError method of the $PSCmdLet object, passing the $errorRecord object as the parameter.

Important: the $PSCmdLet object is only available in Functions that include the [CmdletBinding()] function attribute. So ensure your *-TargetResource and supporting functions include this attribute if you want to be able to access this object.


Test for the Customized Exception

To test for the custom exception object we need to create an identical object in the unit test and test for it:

The above code creates an identical exception object to the one produced by the exception in our DSC Resource code. The exception object can then be passed to the should throw cmdlet. If a different exception is thrown by the code then the test will fail – it will only pass if the exception object is exactly the same.

Important: Make sure both the $errorId and $errorMessage variables are exactly the same as what would be produced by the code when your unit test calls it. This includes ensuring that if your $errorMessage contains any parameters that the unit test $errorMessage contains the same parameter values.


That about completes creating unit tests. After you’ve implemented a few unit tests you’ll no doubt come up with your own method of implementing them, but hopefully this has given you a place to start.


Up Next – Integration Tests

In the next article, I’ll cover the integration tests. There are often the most difficult to implement, but if you can take the time to implement them then your DSC Resources are guaranteed to be extremely robust and bugs are far less likely to slip through.

Further parts in this series:

Pester as an Operation Validation Framework

In this latest video on Channel 9 Jeffrey Snover (the grand wizard of PowerShell) is suggesting might be on the horizon in Windows Server 2016. In it he is saying they are looking at using Pester (or a form of it) to allow you to create Operational Validation tests for your servers and environment so that after any environmental changes are made the environment is validated automatically. This sound like a fantastic idea to me and such an obvious fit to Pester. After doing a bit of digging around it seems like this idea has been around for a while – see this post here for an example of how it can be used in practice.

Of course there does feel like there is a little bit of an overlap here with DSC, but I’m sure the implementation will play well with DSC. All of these new ideas technologies (Nano, Containers, DSC, Operational Pester tests etc) are just more tools in the “Infrastructure as Code” tool belt. So I’m very happy.

I suggest watching the whole video (found here) as it is really interesting, but if you want to just jump to the bit about Pester, it starts at about 11:48. I am really eager to see where Microsoft is going with this stuff in Windows Server 2016. Roll on TP4!

Comparing Objects using JSON in PowerShell for Pester Tests

Recently I spent the good part of a weekend putting together Pester Tests (click here if you aren’t familiar with Pester) for my LabBuilder PowerShell module- a module to build a set of Virtual Machines based on an XML configuration file. In the module I have several cmdlets that take an XML configuration file (sample below) and return an array of hash tables as well as some hash table properties containing other arrays – basically a fairly complex object structure.

A Pester Test config file for the LabBuilder module

A Pester Test config file for the LabBuilder module

In the Pester Tests for these cmdlets I wanted to ensure the object that was returned exactly matched what I expected. So in the Pester Test I programmatically created an object that matched what the Pester Test should expect the output of the cmdlets would be:

$ExpectedSwtiches = @( 
  @{ name="General Purpose External"; type="External"; vlan=$null; adapters=[System.Collections.Hashtable[]]@(
    @{ name="Cluster"; macaddress="00155D010701" },
    @{ name="Management"; macaddress="00155D010702" },
    @{ name="SMB"; macaddress="00155D010703" },
    @{ name="LM"; macaddress="00155D010704" }
  @{ name="Pester Test Private Vlan"; type="Private"; vlan="2"; adapters=@() },
  @{ name="Pester Test Private"; type="Private"; vlan=$null; adapters=@() },
  @{ name="Pester Test Internal Vlan"; type="Internal"; vlan="3"; adapters=@() },
  @{ name="Pester Test Internal"; type="Internal"; vlan=$null; adapters=@() }

What I needed to do was try and make sure the objects were the same. At first I tried to use the Compare-Object cmdlet – this actually wasn’t useful in this situation as it doesn’t do any sort of deep property comparison. What was needed was to serialize the objects and then perform a simple string comparison. The ConvertTo-JSON cmdlet seemed to be just what was needed. I also decided to use the [String]::Compare() method instead of using the PowerShell -eq operator because the -eq operator seems to have issues with Unicode strings.

The Pester test that I first tried was:

Context "Valid configuration is passed" {
  $Switches = Get-LabSwitches -Config $Config
  It "Returns Switches Object that matches Expected Object" {
    [String]::Compare(($Switches | ConvertTo-Json),($ExpectedSwtiches | ConvertTo-Json)) | Should Be 0

This initially seemed to work, but if I changed any of the object properties below the root level (e.g. the adapter name property) the comparison still reported the objects were the same when they weren’t. After reading the documentation it states that the ConvertTo-JSON cmdlet provides a Depth property that defaults to 2 – which limits the depth that an object structure would be converted to. In my case the object was actually 4 levels deep. So I needed to add a Depth parameter to the ConvertTo-JSON calls:

[String]::Compare(($Switches | ConvertTo-Json -Depth 4),($ExpectedSwtiches | ConvertTo-Json -Depth 4)) | Should Be 0

This then did pretty much exactly what I wanted. However, I also needed the comparison to be case-insensitive, so I added a boolean parameter to the [String]::Compare static call:

[String]::Compare(($Switches | ConvertTo-Json),($ExpectedSwtiches | ConvertTo-Json),$true) | Should Be 0

The end result was an deep object comparison between a reference object and the object the cmdlet being tested returned. It is by no means perfect as if the properties or contents of any arrays in the object are out of order the comparison will report that there are differences, but because we control the format of these objects this shouldn’t be a problem and should enable some very test strict cmdlet tests.

How the the Final Pester Test in Visual Studio 2015 (with POSH tools)

How the the Final Pester Test in Visual Studio 2015 (with POSH tools)

Edit: after writing a number of Pester tests using the approach I realized it could be simplified slightly by replacing the generation of the comparison object with the actual JSON output produced by the reference object embedded inline in a variable. For example:

Performing the object comparison using JSON in a variable in the test.

Performing the object comparison using JSON in a variable in the test.

The JSON can be generated manually by hand (before writing the function itself) to stick to the Test Driven Design methodology or it can be generated from the object the function being tested created (once it it working correctly) and then written to a file using:

Set-Content -Path "$($ENV:Temp)\Switches.json" -Value ($Switches | ConvertTo-Json -Depth 4)

The $switches  variable contains the actual object that is produced by the  working command being tested.

A Word of Caution about CRLF

I have noticed that when opening the JSON file in something like Notepad++ and copying the JSON to the clipboard (to paste into my Pester test) that an additional CRLF appears at the bottom. You need to ensure you don’t include this at the bottom of your variable too – otherwise the comparison will fail and the objects will appear to be different (when they aren’t).

This is what the end of the JSON variable definition should look like:

Good JSON CRLF Formatting

And this is what it should not look like (the arrow indicates the location of the extra CRLF that should be removed):

Good JSON CRLF formatting

Note: I could have used the Export-CliXML and Import-CliXML CmdLets instead to perform the object serialization and comparison, but these cmdlets write the content to disk and also generate much larger strings which would take much longer to compare and ending up with a more complicated test.

Well, hopefully someone else will find this useful!