ss_dockerdsc_installing

Install Docker on Windows Server 2016 using DSC

Windows Server 2016 is now GA and it contains some pretty exciting stuff. Chief among them for me is support for containers by way of Docker. So, one of the first things I did was start installing Windows Server 2016 VM’s (Server Core and Nano Server naturally) and installing Docker on them so I could begin experimenting with Docker Swarms and other cool stuff.

Edit: If you’re looking for a DSC configuration for setting up Docker on a Windows 10 Anniversary Edition machine, see the Windows 10 AE section below.

At first I started using the standard manual instructions provided by Docker, but this doesn’t really suit any kind of automation or infrastructure as code methodology. This of course was a good job for PowerShell Desired State Configuration (DSC).

So, what I did was put together a basic DSC config that I could load into a DSC Pull Server and build out lots of Docker nodes quickly and easily. This worked really nicely for me to build out lots of Windows Server 2016 Container hosts in very short order:

ss_dockerdsc_installing

If you don’t have a DSC Pull server or you just want a simple script that you can use to quickly configure a Windows Server 2016 (Core or Core with GUI only) then read on.

Note: This script and process is really just an example of how you can configure Docker Container hosts with DSC. In a real production environment you would probably want to use a DSC Pull Server.

Get it Done

Edit: After a suggestion from Michael Friis (@friism) I have uploaded the script to the PowerShell Gallery and provided a simplified method of installation. The steps could be simplified even further into a single line, but I’ve kept them separate to show the process.

Using PowerShell Gallery

On a Windows Server 2016 Server Core or Windows Server 2016 Server Core with GUI server:

  1. Log on as a user with Local Administrator privileges.
  2. Start an Administrator PowerShell console – if you’re using Server Core just enter PowerShell at the command prompt:ss_dockerdsc_console
  3. Install the Install-DockerOnWS2016UsingDSC.ps1 script from the PowerShell Gallery using this command:

    You may be asked to confirm installation of these modules, answer yes to any confirmations.
    ss_dockerdsc_consolegetscript
  4. Run the Install-DockerOnWS2016UsingDSC.ps1 script using:

    ss_dockerdsc_consolerunscriptfromgallery

The script will run and reboot the server once. Not long after the reboot the Docker service will start up and you can get working with containers:

ss_dockerdsc_consoledockerdetails

You’re now ready to start working with Containers.

The Older Method (without PowerShell Gallery)

On a Windows Server 2016 Server Core or Windows Server 2016 Server Core with GUI server:

  1. Log on as a user with Local Administrator privileges.
  2. Start an Administrator PowerShell console – if you’re using Server Core just enter PowerShell at the command prompt:ss_dockerdsc_console
  3. Install the DSC Resources required for the DSC configuration by executing these commands:

    You may be asked to confirm installation of these modules, answer yes to any confirmations.
    ss_dockerdsc_consoleinstallresources
  4. Download the Docker installation DSC script by executing this command:

    ss_dockerdsc_consoledownloadscript
  5. Run the Docker installation DSC script by executing this command:

    ss_dockerdsc_consolerunscript

The script will run and reboot the server once. Not long after the reboot the Docker service will start up and you can get working with containers:

ss_dockerdsc_consoledockerdetails

You’re now ready to start working with Containers.

What the Script Does

In case you’re interested in what the script actually contains, here are the components:

  1. Configuration ContainerHostDsc – the DSC configuration that configures the node as a Docker Container host.
  2. Configuration ConfigureLCM – the LCM meta configuration that sets Push Mode, allows the LCM to reboot the node if required and configures ApplyAndAutoCorrect mode.
  3. ConfigData – a ConfigData object that contains the list of node names to apply this DSC Configuration to – in this case LocalHost.
  4. ConfigureLCM – the call to the Configuration ConfigureLCM to compile the LCM meta configuration MOF file.
  5. Set-DscLocalConfigurationManager – this applies the compiled LCM meta configuration MOF file to LocalHost to configure the LCM.
  6. ContainerHostDsc – the call to the Configuration ContainerHostDsc to compile the DSC MOF file.
  7. Start-DSCConfiguration – this command starts the LCM applying the DSC MOF file produces by the ContainerHostDsc.

The complete script can be found here. Feel free to use this code in anyway that makes sense to you.

What About Windows 10 AE?

If you’re looking for a DSC configuration that does the same thing for Windows 10 Anniversary edition, Ben Gelens (@bgelens) has written an awesome DSC config that will do the trick. Check it out here.

 

Happy containering!

ss_hqrmreview_codeofconductgood

Tips for HQRM DSC Resources

I’ve spent a fair amount of time recently working on getting some of my DSC Resources (SystemLocaleDsc, WSManDsc, iSCSIDsc and FSRMDsc) accepted into the Microsoft DSC Community Resource Kit. Some are nearly there (SystemLocaleDsc and WSManDsc), whereas others have a way to go yet.

I’ve had one resource already accepted (xDFS) into the DSC Community Resource kit, but this was before the High Quality Resource Module (HQRM) guidelines became available. The HQRM guidelines are a set of standards that DSC modules must meet and maintain to be considered a High Quality Resource Module. Once they meet these requirements they may be eligible to have the ‘x’ moniker removed with ‘Dsc‘ being added to the name.

More information: If you want to read a bit more about the HQRM standards, you can find the HQRM Guidelines here.

Any modules being submitted for inclusion into the DSC Community Resource kit will be expected to meet the HQRM standards. The process of acceptance requires three reviewers from the Microsoft DSC team to review the module.

I thought it might be helpful to anyone else who might want to submit a DSC Resource into the DSC Community Resource kit to get a list of issues the reviewers found with my submissions. This might allow you to fix up your modules before the review process – which will help the reviewers out (they hate having to be critical of your code as much as you do). This enables the submission process to go much faster as well.

More information: If you want to read more about the submission process, you can find the documentation here.

I’ll keep this post updated with any new issues the reviewers pick up. Feel free to ask for clarifications on the issues.

So here is my list of what I have done wrong (so far):

Missing Get-Help Documentation

Every function (public or private) within the DSC resource module must contain a standard help block containing at least a .SYNOPSIS and .PARAMETER block:

This will get rejected:

ss_hqrmreview_gethelpbad

This is good:

ss_hqrmreview_gethelpgood

Examples Missing Explanation

All examples in the Examples folder and the Readme.md must contain an explanation of what the example will do.

This is bad:

ss_hqrmreview_exampledescriptionbad

This is good:

ss_hqrmreview_exampledescriptiongood

Old or Incorrect Unit/Integration Test Headers

There is a standard method of unit and integration testing DSC Resources. Your DSC resources should use these methods where ever possible. Any tests should therefore be based on the latest unit test templates and integration test templates. You should therefore ensure your tests are based on the latest practices and contain the latest header.

This is probably the hardest thing to get right if you’re not paying close attention to the current DSC community best practices around testing. So feel free to ask me for help.

This is bad:

ss_hqrmreview_testheaderbad

This is good:

ss_hqrmreview_testheadergood

Incorrect Capitalization of Local Variables

Local variables must start with a lower case letter. I needed to correct this on several occasions.

Note: this is for local variables. Parameter names should start with Uppercase.

This is bad:

ss_hqrmreview_localparameterbad

This is good:

ss_hqrmreview_localparametergood

Spaces around = in Localization Files

In any localization files you should make sure there is a space on either side of the = sign. This greatly improves message readability.

This is bad:

ss_hqrmreview_localizationbad.png

This is good:

ss_hqrmreview_localizationgood

Missing code of Conduct in Readme.md

All modules that are part of the DSC Resource Kit must contain this message in the Readme.md:

This project has adopted the Microsoft Open Source Code of Conduct.
For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

This is bad:

ss_hqrmreview_codeofconductbad

This is good:

ss_hqrmreview_codeofconductgood

Missing Localization file indent

All strings in localization files should be indented.

This is bad:

ss_hqrmreview_localizationdatabad

This is good:

ss_hqrmreview_localizationdatagood

 

Final Words

There were some other issues raised which I will also document, however I am still in discussion with the DSC team over the best methods to use to solve the issues (specifically the use of InModuleScope in unit tests).

The main thing you can do to help speed this process up and reduce the load on the reviewers however is to implement all the best practices and guidelines listed.

I hope this helps someone out there.

fi_dsc_allthethings

cDFS is dead, long live xDFS

The xDFS DSC resource module has been officially released to the PowerShell Gallery thanks to the awesome review efforts of the Microsoft PowerShell Team. The cDFS DSC Resource has now been unlisted from the PowerShell Gallery. So now is the time to update any DSC configuration scripts to use xDFS.

ss_xdfs_releasepsgallery

Important: There were some minor changes to xDFS when it was converted from cDFS. For information on what you’ll need to change to convert to xDFS see my earlier post.

 

cDFS moving to the PowerShell Team

Just a Friday afternoon heads up – if you’re using the cDFS DSC Resource I created to manage Windows Server Distributed File System (Replication and Namespaces), it has now been accepted into the PowerShell Community resources and will be under the control of the PowerShell Team.

This means that the GitHub source code repository will be moving over to the PowerShell organization in the next few days. This also means that any future releases of this resource module won’t be provided by me as cDFS, but will be released by the PowerShell team as xDFS.

So I recommend that when this happens you switch over to using the xDFS resource. I will put another post up here when the change over officially occurs. The first official release version under the new xDFS name will be 3.0.0.x. I won’t make any further changes or bug fixes to the cDFS resources.

It is also worth noting that as part of this move some minor changes were made to the DSC Resource modules. These are breaking changes and you will most likely need to update any DSC Configurations depending on this, but you would have to do this anyway because of the name change.

The changes are:

  • Resource xDFSRepGroup renamed to xDFSReplicationGroup
  • Resource xDFSRepGroupConnection renamed to xDFSReplicationGroupConnection
  • Resource xDFSRepGroupFolder renamed to xDFSReplicationGroupFolder
  • Resource xDFSRepGroupMembership renamed to xDFSReplicationGroupMembership
  • xDFSReplicationGroupConnection:
    • Changed DisableConnection parameter to EnsureEnabled.
    • Changed DisableRDC parameter to EnsureRDCEnabled.

These changes should only require minor changes to your configuration scripts to implement.

Thanks for reading and have a great Friday~

 

 

Install Jenkins using DSC – Part 2

In my previous post I showed how to create a PowerShell script that would install a Jenkins CI Master server onto a Windows Server Core installation. The obvious next step for such a script was to convert it into a DSC configuration file.

In this post I’m assuming WMF 5.0 is installed onto the server that will be converted into a Jenkins Master. You could manage this without WMF 5.0, but you’d need to manually install the DSC Resource modules that the configuration will use.

Once again, the full DSC Configuration script can be found at the end of the post.

Requirements

You’ll need:

  • A physical or virtual machine running Windows Server 2012 R2 Core (or Full) – it should be a completely clean install with WMF 5.0 installed on it.
  • An administrator login to the server.
  • An internet connection to the server.

Resource Modules

This DSC Configuration requires the use of three DSC Resources:

  • cChoco – this community resource is used to install Chocolatey and Jenkins.
  • xNetworking – this resource is used to configure the networking on the server if required.
  • PSDesiredStateConfiguration – this resource comes with PowerShell by default and is used to provide the Script resource.

The easiest way to install these resource modules is by executing these commands on the Jenkins server:

# Make sure the DSC Resource modules are downloaded
Install-Module -Name cChoco -Force
Install-Module -Name xNetworking -Force

However, if you’re using a Pull server or compiling the DSC MOF on a development machine (rather than the Jenkins node) you would need to use other methods of ensuring the modules are available.

The Configuration Components

The DSC Configuration needs to do the following things:

  • Configure Networking (optional)
  • Install .NET 3.5 Framework
  • Install Chocolatey
  • Install JDK 8
  • Install Jenkins
  • Configure Jenkins Port (optional)

Configure Networking

I like to use the xNetwoking DSC resource to configure the IPv4 and IPv6 settings on the Network adapter to have a static configuration. However, you won’t need to do this if you’re using DHCP or manual configuration. Note, in my case my adapter was called “Ethernet”.

xIPAddress IPv4_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv4'
    IPAddress      = '192.168.128.20'
    SubnetMask     = '24'
}
xDefaultGatewayAddress IPv4G_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv4'
    Address        = '192.168.128.19'
}
xDnsServerAddress IPv4D_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv4'
    Address        = '192.168.128.10'
}
xIPAddress IPv6_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv6'
    IPAddress      = 'fd53:ccc5:895a:bc00::14'
    SubnetMask     = '64'
}
xDefaultGatewayAddress IPv6G_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv6'
    Address        = 'fd53:ccc5:895a:bc00::13'
}
xDnsServerAddress IPv6D_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv6'
    Address        = 'fd53:ccc5:895a:bc00::a'
}

Install .NET 3.5 Framework

Jenkins requires the .NET 3.5 Framework, so I’m going to use the WindowsFeature DSC Resource to install it:

WindowsFeature NetFrameworkCore
{
    Ensure    = "Present"
    Name      = "NET-Framework-Core"
}

Install Chocolatey

Next up, I’m going to use the cChocoInstaller resource in the cChoco resource module (available on PowerShell Gallery here) to install the Chocolatey package manager:

# Install Chocolatey
cChocoInstaller installChoco
{
    InstallDir = "c:\choco"
    DependsOn = "[WindowsFeature]NetFrameworkCore"
}

Install JDK 8 and Jenkins

The cChocoPackageInstaller resource module is the used to install JDK 8 and Jenkins

# Install JDK8
cChocoPackageInstaller installJdk8
{
    Name = "jdk8"
    DependsOn = "[cChocoInstaller]installChoco"
}

# Install Jenkins
cChocoPackageInstaller installJenkins
{
    Name = "Jenkins"
    DependsOn = "[cChocoInstaller]installChoco"
}

Configure Jenkins Port

The last step of the configuration is optional. By default Jenkins is configured to listen on port 8080, however I want to change it to 80. So this next part uses the Script resource to change the “–httpPort” setting in the Jenkins.xml file. I use Regex to do this:

# Set the Jenkins Port
Script SetJenkinsPort
{
	SetScript = {
		Write-Verbose -Verbose "Setting Jenkins Port to $Using:JenkinsPort"
		$Config = Get-Content `
			-Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
		$NewConfig = $Config `
			-replace '--httpPort=[0-9]*\s',"--httpPort=$Using:JenkinsPort "
		Set-Content `
			-Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml" `
			-Value $NewConfig `
			-Force
		Write-Verbose -Verbose "Restarting Jenkins"
		Restart-Service `
			-Name Jenkins
	}
	GetScript = {
		$Config = Get-Content `
			-Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
		$Matches = @([regex]::matches($Config, "--httpPort=([0-9]*)\s", 'IgnoreCase'))
		$CurrentPort = $Matches.Groups[1].Value
		Return @{
			'JenkinsPort' = $CurrentPort
		}
	}
	TestScript = {
		$Config = Get-Content `
			-Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
		$Matches = @([regex]::matches($Config, "--httpPort=([0-9]*)\s", 'IgnoreCase'))
		$CurrentPort = $Matches.Groups[1].Value

		If ($Using:JenkinsPort -ne $CurrentPort) {
			# Jenkins port must be changed
			Return $False
		}
		# Jenkins is already on correct port
		Return $True
	}
	DependsOn = "[cChocoPackageInstaller]installJenkins"
}

Create the MOF

The final thing to do is download the cChoco and xNetworking DSC Resources,create the MOF and then ask the LCM to apply it:

$ConfigData = @{
    AllNodes =
    @(
        @{
            NodeName = "LocalHost"
        }
    )
}

JENKINS_CI -JenkinsPort 80 -ConfigurationData $ConfigData

Start-DscConfiguration -Path .\JENKINS_CI -Wait -Verbose

The Complete DSC Configuration

Here is the complete DSC Configuration file. You just need to copy it to the Server and run it. It will compile the configuration into a MOF and tell the LCM to apply it. Just remember to ensure required DSC Resource modules are installed.

Within five to ten minutes the Jenkins server will be configured and ready to go.

Creating Professional DSC Resources – Part 7

The purpose of this series of articles is to try and document a few of the lessons I learned while releasing new DSC resources as well as contributing to the existing Microsoft Community DSC resources. These articles are not intended to tell you how to write DSC resources from a programming perspective, but to give you some ideas on what might be expected of a DSC resource you’re releasing to the public. For example, unit and integration tests (don’t worry if you aren’t familiar with those terms).

These articles are also not intended to tell you what you must do to release your resource, but more document what will help your resource be easier to use and extend by other people. Some of these these things are obvious for people who have come from the development community, but may be quite new to operations people.

If you missed any previous articles you can find them here:

 

Recap

In the last couple of articles I covered the importance of automated testing with unit testing in particular. I had covered creating new unit tests using the unit test templates that are now available here. I also covered how to complete the Pester Test Initialization and the Get-TargetResource, Set-TargetResource and Test-TargetResource function areas of the unit test.

 

Integration Testing

Integration testing is a great way of catching many errors that can’t be easily picked up by Unit testing. It effectively tests your DSC Resource by actually using it in a DSC configuration file and applying it to a computer and checking the results. So this is as close to real-life testing as you can get.

Integration testing of a PowerShell DSC resource should be performed after unit testing. When a PowerShell DSC Resource is integration tested the following process occurs:

  1. A DSC configuration file using the DSC resource to be integration tested is compiled into a MOF.
  2. The MOF file is applied to the test machine.
  3. The parameters current DSC Configuration of this DSC Resource on the test machine is obtained.
  4. The parameters of the current DSC Configuration are compared with what was set in the DSC configuration file in step 1.

Just like unit testing we use Pester to test the above steps and ensure that errors don’t occur and the output is as expected.

 

 

Sometimes Integration Tests are not Possible

Integration testing is not always possible on a resource. Some resources may rely on external servers being available or they might be destructive to the machine performing the tests.

For example, integration tests could not be implemented for the MSFT_xIPAddress resource in the xNetworking DSC Resource module because it would have caused the network to disconnect during testing which would have resulted in a failure of the AppVeyor CI machine running the tests.

But, if there is a reasonable way of implementing integration tests for a resource in a non-destructive manor, then I’d strongly recommend it – especially as it is usually really easy.

 

Don’t Be Destructive!

Unlike unit testing, integration testing actually changes configuration on the machine performing the tests. If you’re using a continuous integration service like AppVeyor to perform your tests then this isn’t such a problem as the test machine is “destroyed” after your tests are run.

However, many people also run any integration tests on their local machines before committing code, therefore, your integration tests should always leave the machine in the state that it was before running them. This means that any changes that will be made applying the integration tests should be undone at the completion of your integration tests script.

 

Integration Test Files

Integration tests for a DSC resource actually consist of two different files:

ss_dsc_inttestfiles

  1. *.config.ps1 – The DSC Configuration file that will use the DSC Resource being tested.
  2. *.Integration.Tests.ps1 – The Integration Test script file containing the Pester tests.

These files should be stored in the Tests\Integration folder in the DSC Resource module:

ss_dsc_inttestfolders

You must also ensure that the names of these files exactly matches the name of the resource itself. For example, if your DSC Resource is called BMD_MyResource then these files must be called:

 

  1. BMD_MyResource.config.ps1
  2. BMD_MyResource.Integration.Tests.ps1

 

 

Creating a New Integration Test

Luckily, a good amount of the work in implementing integration tests is already done for you. Like unit tests, templates for the two integration files are available in the DscResources repository in GitHub:

ss_dsc_inttesttemplatesrepo

You need to copy the integration test template files and rename them to match your DSC Resource.

The easiest way to do this is to clone the repository containing the test template files and copy the integration_template.ps1 and integration_config_template.ps1 files to your Tests/Integration folder:

ss_dsc_createnewinttestfromtemplate

You’ll now have two new integration test files that you can open in your PowerShell editor of choice.

 

Modifying the Config File

The first file I usually edit is the *.config.ps1 file:

ss_dsc_newinttestconfigtemplate

Next, you’ll want to change any <ResourceName> occurrences in this file to the name of your resource. I also like to remove the #TODO bits at the same time so I know what I’ve completed:

Next, we need to configure the config file with the parameters we want to use as tests of the resource.

The best way of doing this is actually to create a hash table object at the beginning of the file with the parameters that we’re going to set. This is so that we can use this hash table object in the other integration file (*.Integration.Tests.ps1) when we’re comparing the values that are expected to be set.

As you can see in the example above, I create a $VirtualDisk hash table that contains all the parameters and values that will be used to test this DSC Resource. The $VirtualDisk object is then also accessible in the *.Integration.Tests.ps1 file.

 

Modifying the Integration Tests File

Now that the integration tests config file has been completed it is time to move on to the integration test script (*.Integration.Tests.ps1) itself, so open it in your editor of choice:

ss_dsc_newinttesttemplate

Next, customize the TODO area in the header with the your DSC Resource Name and DSC Module Name:

ss_dsc_inttestsheader

Feel free to remove the TODO comments if you want (I always do).

 

Initialization Code

After customizing the header we need to add any code that might be required to set this machine up to actually perform these integration tests. The first thing I like to do is add code to check that these integration tests can actually be performed on this machine. In my example resource, the iSCSI Virtual Disk resource will require the iSCSI Target Server feature to be installed, which also means the OS must be a Server OS. So, first thing in the try/catch block I add these checks:

This will cause the try/catch block to be exited straight away if these tests can’t actually be performed on this machine.

Note: The cleanup code in the finally block will still be called if we exit with a break command.

After this you might also need to add code to configure anything that these integration tests might depend on. For example, if you were implementing integration tests for testing an iSCSI Server Target, you’d need to make sure that there was an iSCSI Virtual Disk available to use, so you’d need to create one at this point. However, in the integration tests for the iSCSI Virtual Disk resource I don’t need anything else.

 

Testing the Resource Was Applied

Next, we need to add the tests that check that after the DSC Configuration has been applied to the machine that the changes have actually been made and that the parameters match those set by the Configuration:

To do this, we complete this section:

ss_dsc_inttestsvalidate

In this case, I’ve changed it to:

What this code does is gets the iSCSI Virtual Disk that is at the path specified in the $VirtualDisk.path into a variable $VirtualDiskNew.

The parameters in $VirtualDiskNew are then matched to ensure they are the same as those in the $VirtualDisk hash table object that was created in the DSC Configuration script (*.config.ps1).

 

Cleaning Up

It is important that after the tests have been run that any changes that were made to the testing computer are reverted. So, after the end of the last test I add any clean up code. In my case, I want to remove the iSCSI Virtual Disk that was created:

The above code just removes the iSCSI Virtual Disk and then also makes sure that the VHD file was also deleted. This is also very important because if the clean up does not occur and the tests are run again on the same computer they may fail.

 

And We’re Done!

Now, that may all seem like quite a bit of work, but it becomes second nature after creating a few of them. They will also save you far more time in addressing future issues with the resource every time you make a simple change to the MOF (but forget to change the resource code). These tests will give users and other maintainers much more confidence in your resources as well.

This series actually ended up being a bit longer than I intended, but hopefully you’ve stuck with it and it has helped in some small way. If you’ve got this far and you’re wanting to know what to do next, why not head over to the PowerShell DSCResources GitHub repository and see if you could help out on some resources. You could start off adding some small but useful parameter to an existing resource, fixing a bug or contribute an entire new resource to an existing module. There are numerous issues that need to be addressed on these resources, many of which are requests for new features or resources.

If you have an idea for a new resource in an existing module, raise an issue in the DSC Resource Module repository and offer to create the new resource. You may find that someone is already working on one, but if not, then this is a great opportunity to get started. It is quite a rewarding feeling the first time one of your contributions gets published in the official community DSC Resources!

So, thanks again for reading.