Managing Users & Permissions in CosmosDB with PowerShell

If you’re just getting started with CosmosDB, you might not have come across users and permissions in a CosmosDB database. However, there are certain use cases where managing users and permissions are necessary. For example, if you’re wanting to be able to limit access to a particular resource (e.g. a collection, document, stored procedure) by user.

The most common usage scenario for users and permissions is if you’re implementing a Resource Token Broker type pattern, allowing client applications to directly access the CosmosDB database.

Side note: The CosmosDB implementation of users and permissions only provides authorization – it does not provide authentication. It would be up to your own implementation to manage the authentication. In most cases you’d use something like Azure Active Directory to provide an authentication layer.

But if you go hunting through the Azure Management Portal CosmosDB data explorer (or Azure Storage Explorer) you won’t find any way to configure or even view users and permissions.

ss_cdb_cosmosdbdataexplorer

To manage users and permissions you need to use the CosmosDB API directly or one of the SDKs.

But to make CosmosDB users and permissions easier to manage from PowerShell, I created the CosmosDB PowerShell module. This is an open source project hosted on GitHub. The CosmosDB module allows you to manage much more than just users and permissions, but for this post I just wanted to start with these.

Requirements

This module works on PowerShell 5.x and PowerShell Core 6.0.0. It probably works on PowerShell 3 and 4, but I don’t have any more machines running this version to test on.

The CosmosDB module does not have any dependencies, except if you call the New-CosmosDbConnection function with the ResourceGroup parameter specified as this will use the AzureRM PowerShell modules to read the Master Key for the connection directly from your CosmosDB account. So I’d recommend installing the Azure PowerShell modules or if you’re using PowerShell 6.0, install the AzureRM.NetCore modules.

Installing the Module

The best way to install the CosmosDB PowerShell module is from the PowerShell Gallery. To install it for only your user account execute this PowerShell command:

install-module -Name CosmosDB -Scope CurrentUser

ss_cdb_cosmosdbinstallmodulecurrentuser

Or to install it for all users on the machine (requires administrator permissions):

install-module -Name CosmosDB

ss_cdb_cosmosdbinstallmoduleallusers

Connection Variable

Before you get down to the process of working with CosmosDB resources, you’ll need to configure the information required to connect. This requires the following information:

  1. The CosmosDB Account name
  2. The CosmosDB Database name
  3. The Master Key for the account (you can have the CosmosDB PowerShell module get this directly from your Azure account if you wish).

To create the connection variable we just use the New-CosmosDbConnection:

ss_cdb_cosmosdbnewconnection

If you do not wish to specify your master key, you can have the New-CosmosDbConnection function pull your master key from the Azure Management Portal directly:

ss_cdb_cosmosdbnewconnectionviaportal

Note: This requires the AzureRM.Profile and AzureRM.Resoures module on Windows PowerShell 5.x or AzureRM.Profile.NetCore and AzureRM.Resources.NetCore on PoweShell Core 6.0.0.

Managing Users

To add a user to the CosmosDB Database use the New-CosmosDBUser function:

New-CosmosDbUser -Connection $connection -Id 'daniel'

ss_cdb_cosmosdbnewuser

To get a list of users in the database:

Get-CosmosDbUser -Connection $connection

ss_cdb_cosmosdbgetusers

To get a specific user:

Get-CosmosDbUser -Connection $connection -Id 'daniel'

ss_cdb_cosmosdbgetuser

To remove a user (this will also remove all permissions assigned to the user):

Remove-CosmosDbUser -Connection $connection -Id 'daniel'

ss_cdb_cosmosdbremoveuser

Managing Permissions

Permissions in CosmosDB are granted to a user for a specific resource. For example, you could grant a user access to just a single document, an entire collection or to a stored procedure.

To grant a permission you need to provide four pieces of information:

  1. The Id of the user to grant the permission to.
  2. An Id for the permission to create. This is just string to uniquely identify the permission.
  3. The permission mode to the permission: All or Read.
  4. The Id of the resource to grant access to. This can be generated from one of the Get-CosmosDb*ResourcePath functions in the CosmosDB PowerShell module.

In the following example, we’ll grant the user daniel all access to the TestCollection:

ss_cdb_cosmosdbnewpermission

Once a permission has been granted, you can use the Get-CosmosDbPermission function to retrieve the permission and with it the Resource Token that can be used to access the resource for a limited amount of time (between 10 minutes and 5 hours).

Note: as you have the Master Key already, using the Resource Token isn’t required.

For example, to retrieve all permissions for the user with Id daniel and a resource token expiration of 600 seconds:

Get-CosmosDbPermission -Connection $connection -UserId 'daniel' -TokenExpiry '600' |
fl *

ss_cdb_cosmosdbgetpermission

You can as expected delete a permission by using the Remove-CosmosDbPermission function:

Remove-CosmosDbPermission -Connection $connection -UserId 'daniel' -Id 'AccessTestCollection'

ss_cdb_cosmosdbremovepermission

Final Thoughts

So this is pretty much all there is to managing users and permissions using the CosmosDB PowerShell module. This module can also be used to manage the following CosmosDB resources:

 

  • Attachments
  • Collections
  • Databases
  • Documents
  • Stored procedures
  • Triggers
  • User Defined Functions

You can find additional documentation and examples of how to manage these resources over in the CosmosDB PowerShell module readme file on GitHub.

Hopefully this will help you in any CosmosDB automation tasks you might need to implement.

 

Advertisements

Configure Azure SQL Server Automatic Tuning with PowerShell

One thing I’ve found with configuring Azure services using automation (e.g. Azure PowerShell Modules, Azure Resource Manager template) is that the automation features are a little bit behind the feature set. For example, the Azure PowerShell modules may not yet implement settings for new or preview features. This can be a an issue if you’re strictly deploying everything via code (e.g. infrastructure as code). But if you run into a problem like this, all is not lost. So read on for an example of how to solve this issue.

Azure REST APIs

One of the great things about Azure is that everything is configurable by making direct requests to the Azure REST APIs, even if it is not available in ARM templates or Azure PowerShell.

Depending on the feature/configuration you can sometimes use the Set-AzureRmResource cmdlets to make calls to the REST APIs. But this cmdlet is limited to using an HTTP method of POST. So if you need to use PATCH, you’ll need to find an alternate way to make the call.

So, what you need then is to use the Invoke-RestMethod cmdlet to create a custom call to the REST API. This is the process I needed to use to configure the Azure SQL Server Automatic Tuning settings and what I’ll show in my script below.

The Script

The following script can be executed in PowerShell (of course) and requires a number of parameters to be passed to it:

  • SubscriptionId – the subscription Id of the Azure subscription that contains the Azure SQL Server.
  • ResourceGroupName – The name of the resource group containing SQL Server or
    database.
  • ServerName – The name of the Azure SQL Server to set the automatic tuning
    options on.
  • DatabaseNameThe name of the Azure SQL Database to set the automatic tuning options on. If you pass this parameter then the automatic tuning settings are applied to the Azure SQL Database, not the server.
  • Mode – This defines where the settings for the automatic tuning are
    obtained from. Inherit is only valid if the DatabaseName is specified.
  • CreateIndexEnable automatic tuning for creating an index.
  • DropIndexEnable automatic tuning for dropping an index.
  • ForceLastGoodPlan – Enable automatic tuning for forcing last good plan.

Requirements: You need to have the installed the AzureRM.Profile PowerShell module (part of the AzureRM PowerShell Modules) to use this script. The script also requires you to have logged into your Azure Subscription using Add-AzureRmAccount (as a user or Service Principal).

Example Usage

To apply custom automatic tuning to an Azure SQL Server:

.\Set-AzureRMSqlServerAutotuning.ps1 -SubscriptionId '<Subscription Id>' -ResourceGroupName '<Resource Group name>' -ServerName '<Azure SQL server name>' -Mode Custom -CreateIndex On -DropIndex On -ForceLastGoodPlan Off

ss_sqlserver_serverautotuning

To apply custom automatic tuning to an Azure SQL Database:

.\Set-AzureRMSqlServerAutotuning.ps1 -SubscriptionId '<Subscription Id>' -ResourceGroupName '<Resource Group name>' -ServerName '<Azure SQL server name>' -DatabaseName '<Azure SQL database name>' -Mode Custom -CreateIndex On -DropIndex On -ForceLastGoodPlan Off

ss_sqlserver_databaseautotuning

Conclusion

I’ve not yet encountered something in Azure that I can’t configure via the Azure REST APIs. This is because the Azure Management Portal uses the same APIs – so if it is available in the portal then you can do it via the Azure REST APIs. The biggest challenge is determining the body, header and methods available if the APIs are not yet documented.

If the API you need is not documented then you can raise a question in the Microsoft Azure Forums or on Stack Overflow. Failing that you can use the developer tools in your browser of choice to watch the API calls being made to the portal – I’ve had to resort to this many times, but documenting that process is something I’ll save for another day.

 

Stop, Start or Restart all Web Apps in Azure using PowerShell

Here is a short (and sometimes handy) single line of PowerShell code that can be used to restart all the Azure Web Apps in a subscription:

ss_azurecloudshell_restartallwebapps

Note: Use this with care if you’re working with production systems because this _will_ restart these Web Apps without confirming first.

This would be a handy snippet to be able to run in the Azure Cloud Shell. It could also be adjusted to perform different actions on other types of resources.

To stop all Web Apps in a subscription use:

To start them all again:

The key part of this command is the GetEnumerator() method because most Azure Cmdlets don’t return an array of individual objects into the pipeline like typical PowerShell cmdlets. Instead returning a System.Collections.Generic.List object, which requires a slight adjustment to the code. This procedure can be used for most Azure Cmdlets to allow the results to be iterated through.

ss_azurecloudshell_systemcollections

Thanks for reading.

Install Nightly Build of Azure CLI 2.0 on Windows

The Azure PowerShell cmdlets are really first class if you’re wanting to manage Azure with PowerShell. However, they don’t always support the very latest Azure components and features. For example, at the time of writing this there is no Azure PowerShell module for managing Azure Container Instances.

The solution to this is to install the Nightly Build of Azure CLI 2.0. However, on Windows it is not entirely clear the easiest way to do this. So, in this post I’ll provide a PowerShell script that will:

  1. Install Python 3.x using Chocolatey
  2. Use PIP (Python package manager) to install the latest nightly build packages
  3. Update the Environment Path variable so that you can use Azure CLI 2.0.

Note: If you have the stable build of Azure CLI 2.0 installed using the MSI then you’ll need to configure your Environment Path variable to find the Az command that you’d like to use by default. I personally removed the stable build of Azure CLI 2.0 to make it easier.

Performing the Install

Make sure you’ve got Chocolatey installed. If you aren’t sure what Chocolatey is, it is a package management system for Windows – not unlike Apt-Get or Yum for Linux. It is free and awesome. In this process we’ll use Chocolatey to install Python for us. If you haven’t got Chocolatey installed, see this page for instructions.

Next, download and run this PowerShell script in a PowerShell Administrator Console:

You could save the content of this script into a PS1 file and then execute it like this:

ss_azurecli_installnightlybuild

It will then download and install Python, then use PIP to install the current nightly build packages. After a few minutes the installation will complete:

ss_azurecli_installnightlybuildcompete

You can then run:

Az Login

To get started.

If you’re a bit new to Azure CLI 2.0, then another great way is to use Azure CLI Interactive:

Az Interactive

ss_azurecli_interactive

If you need to update to a newer nightly build, just run the script again and it will update your packages.

Easy as that! Now you can experiment with all the latest automation features in Azure without needing to wait for a new version of Azure CLI 2.0 or for latest Azure PowerShell cmdlets.

Edge Builds

If you want to install even more “bleeding edge” builds (built straight off the master branch on every merge to master) then you can make a small adjustment to the script above:

On line 34 change the URL of the feed from:

https://azureclinightly.blob.core.windows.net/packages

To:

https://azurecliprod.blob.core.windows.net/edge

Thanks for reading!

 

 

Get Azure API Management Git Credentials using PowerShell

One of the many great features of Azure API Management is the fact that it has a built in Git repository for storing the current configuration as well as publishing new configurations.

ss_apim_gitrepository

This allows you to push updated Azure API Management configurations to this internal Git repository as a new branch and then Deploy the configuration to API Management.

The internal Git repository in Azure API Management is not intended to be used for a normal development workflow. You’ll still want to develop and store your Azure API management configuration in an external Git repository such as GitHub or TFS/VSTS and then copy configuration updates to the internal Git repository in Azure API Management using some sort of automated process (e.g. Continuous Integration/Continuous Delivery could be adopted for this).

The Internal Git Repository

To access the Internal Git Repository requires short lived (30 days maximum) Git credentials to be generated. This is fairly easy through the Azure API Management portal:

ss_apim_gitrepositorygeneratecreds

Unfortunately using the portal to get these credentials is a manual process and so would not be so good for an automated delivery process (e.g. CI/CD). You’d need to update these Git credentials in your CI/CD automation system every time they expired (every 30 days).

Get Git Credentials

A better approach to generating the Git Credentials is to use Azure PowerShell API Management cmdlets connected with a Service Principal to generate the Git credentials whenever you need them in your CI/CD pipeline.

This is not a completely straightforward process right now (which is unusual for the Azure PowerShell team), so I’ve created a simple PowerShell script that will take care of the nuts and bolts for you.

Requirements

To run this script you’ll need:

  1. PowerShell 5 (WMF 5.0) or greater.
  2. Azure PowerShell Modules installed (make sure you’ve got the latest versions – 4.0.3 at the time of writing this).

You’ll also need to supply the following parameters to the script:

  1. The Azure Subscription Id of the subscription containing the API Management instance.
  2. The name of the Resource Group where the API Management instance is installed to.
  3. The service name of the API Management instance.

You can also optionally supply which of the two internal API Management keys, primary or secondary, to use to generate the credential and also the length of time that the Git credential will be valid for (up to 30 days).

Steps

Download the Script

  1. Download the script Get-AzureRMApiManagementGitCredential.ps1 using the PowerShell command:
  2. Unblock the script using the PowerShell command:

Using the Script

  1. Use the Login-AzureRMAccount cmdlet to authenticate to Azure. This would normally be done using a Service Principal if using an automated process, but could be done interactively when testing.
  2. Execute the script providing the SubscriptionId, ResourceGroup and ServiceName parameters (and optionally the KeyType and ExpiryTimespan) using the following PowerShell command:

ss_apim_gitrepositoryinvoke

The script will return an object containing the properties GitUsername and GitPassword that can be provided to Git when cloning the internal Git repository.

The GitPassword is not escaped so can not be directly used within a Git Clone URL without replacing any / or @ with %2F and %40 respectively.

In the example above I generated an internal Git Credential using the Primary Secret Key that will expire in 4 hours.

Typically you’d assign the output of this script to a variable and use the properties to generate the URL to pass into the Git Clone. For example:

ss_apim_gitrepositoryclone

Tips

  • When cloning the internal Git Repository you’ll need the clone URL of the repository. This is always the name of your Azure API Management instance followed by with scm.azure-api.net appended to it E.g. https://myapimanagementinstance.scm.azure-api.net
  • Once you’ve uploaded a new Git branch containing a new or updated Azure API Management configuration you’ll need to use the Publish-AzureRmApiManagementTenantGitConfiguration cmdlet to tell Azure API Management to publish the configuration contained in the branch. I have not detailed this process here, but if there is interest I can cover the entire end-to-end process.
  • The Primary and Secondary Secret Keys that are used to generate the internal Git Credential can be re-generated (rolled) individually if a Git credential is compromised. However, this will invalidate all Git Credentials generated using that Secret Key.

The Script

If you wish to review the script itself, here it is:

So, hopefully that will be enough information to get anyone else started on building a CI/CD pipeline for deploying Azure API Management configurations.

 

Sonatype Nexus Containers with Persistent Storage in Azure Container Instances

On the back of yesterdays post on running Azure Container Instance containers with persistent storage, I thought I’d try a couple of other containers with my script.

Note: I don’t actually plan on running any of these apps, I just wanted to test out the process and my scripts to identify any problems.

I tried:

And here are the results of my tests:

Sonatype Nexus 2

Works perfectly and the container starts up quickly (under 10 seconds):

ss_aci_sonatypenexus2

I passed the following parameters to the script:

Note: The Nexus 2 server is only accessible on the path /nexus/.

Sonatype Nexus 3

Works perfectly but after takes at least a minute to be accessible after the container starts. But this is normal behavior for Nexus 3.

ss_aci_sonatypenexus3

I passed the following parameters to the script:

Jenkins

Unfortunately Jenkins does not work with a persistent storage volume from an Azure Share. It seems to be trying to set the timestamp of the file that will contain the InitialAdminPassword, which is failing:

ss_aci_jenkins

I passed the following parameters to the script:

So, this is still a little bit hit and miss, but in general Azure Container Instances look like a very promising way to run different types of services in containers without a lot of overhead. With a bit of automation, this could turn out to be a cost effective way to quickly and easily run some common services.

Persistent Storage in Azure Container Instances

Update 2017-08-06: I have improved the script so that it is idempotent (can be run more than once and will only create anything that is missing). The Azure Container Instance resource group can be deleted once you’ve finished with the container and then recreated again with this same script when you next need it. The storage will be preserved in the separate storage account resource group. The script can now be run with the -verbose parameter and will produce much better progress information.

Azure Container Instances (ACI) is a new resource type in Azure that allows you to quickly and easily create containers without the complexity or overhead of Azure Service Fabric, Azure Container Services or provisioning a Windows Server 2016 VM.

It allows you to quickly create containers that are billed by the second from container images stored in Docker Hub or your own Azure Container Registry (ACR). Even though this feature is still in preview, it is very easy to get up and running with it.

But this post isn’t about creating basic container instances, it is about running container instances where some of the storage must persist. This is a basic function of a container host, but if you don’t have access to the host storage then things get more difficult. That said, Azure Container Instances do support mounting Azure File Shares into the container as volumes. It is fairly easy to do, but requires quite a number of steps.

There is some provided documentation for persisting storage in a container instance, but it is quite a manual process and the example ARM templates are currently broken: there are some typos and missing properties. So this post aims to make the whole thing a lot simpler and automatable.

So in this post, I’m going to share a PowerShell function and Azure Resource Manager (ARM) template that will allow you to easily provision an Azure Container Instance with an Azure File Share mounted. The process defaults to installing a GoCD Server container (version 17.8.0 if you’re interested), but you could use it to install any other Linux Container that needs persistent storage. The script is parameterized so other containers and mount points can be specified – e.g. it should be fairly easy to use this for other servers like Sonatype Nexus or Jenkins Server.

Update 2017-08-06: I documented my findings trying out these other servers in my following blog post.

Requirements

To perform this process you will need the following:

  • PowerShell 5.0+ (PowerShell 4.0 may work, but I haven’t tested it).
  • The Azure PowerShell module installed.
  • Created an Application Service Principal – see below.

Azure Service Principal

Before you start this process you will need to have created an Application Service Principal in Azure that will be used to perform the deployment. Follow the instructions on this page to create an application and then get the Service Principal from it.

You will need to record these values as they will be provided to the script later on:

  • Application Id
  • Application Key
  • Tenant Id
  • Subscription Name

The Process

The process will perform the following tasks:

  1. The Service Principal is used to login to Azure to perform the deployment.
  2. An Azure Resource Group is created to contain a Azure Storage Account and Azure Key Vault.
  3. An Azure Storage Account is created and an Azure File Share is created in it.
  4. An Azure Key Vault is created to store the Storage Account Key and make it accessible to the Azure Container Instance.
  5. The Service Principal is granted permission to the Azure Key Vault to read and write secrets.
  6. The key to the Storage Account Key is added as a secret to the Azure Key Vault.
  7. The parameters are set in an ARM Template parameter file.
  8. An Azure Resource Group is created to contain the Azure Container Instance.

The Script

This is the content of the script:

The script requires a four parameters to be provided:

  • ServicePrincipalUsername – the Application Id obtained when creating the Service Principal.
  • ServicePrincipalPassword – the Application Key we got (or set) when creating the Service Principal.
  • TenancyId – The Tenancy Id we got during the Service Principal creation process.
  • SubscriptionName – the name of the subscription to install the ACI and other resources into.

There are also some other optional parameters that can be provided that allow the container image that is used, the TCP port the container listens on and mount point for the Auzre File Share. If you don’t provide these parameters will be used which will create a GoCD Server.

 

  • AppCode – A short code to identify this application. It gets added to the resource names and resource group names. Defaults to ‘gocd’.
  • UniqueCode – this string is just used to ensure that globally unique names for the resources can be created. Defaults to ‘zzz‘.
  • ContainerImage – this is the name and version of the container image to be deployed to the ACI. Defaults to ‘gocd/gocd-server:v17.8.0‘.
  • CPU – The number of cores to assign to the container instance. Defaults to 1.
  • MemoryInGB – The amount of memory (in GB) to assign to the container instance. Defaults to 1.5.
  • ContainerPort – The port that the container listens on. Go CD Server defaults to 8153.
  • VolumeName – this is a volume name that is used to represent the volume in the ARM template. It can really be set to anything. Defaults to ‘gocd‘.
  • MountPoint – this is the folder in the Container that the Azure File Share is mounted to. Defaults to ‘/godata/‘.

ARM Template Files

There are two other files that are required for this process:

  1. ARM templatethe ARM template file that will be used to install the ACI.
  2. ARM template parameters – this file will be used to pass in the settings to the ARM Template.

ARM Template

This file is called aci-azuredeploy.json and should be downloaded to the same folder as the script above.

ARM Template Parameters

This file is called aci-azuredeploy.parameters.json and should be downloaded to the same folder as the script above.

Steps

To use the script the following steps need to be followed:

  1. Download the three files above (the script and the two ARM template files) and put them into the same folder:ss_aci_filesrequires
  2. Open a PowerShell window.
  3. Change directory to the folder you place the files into by executing:
  4. CD <folder location>
  5. Execute the script like this (passing in the variables):ss_aci_executingscript
  6. The process will then begin and make take a few minutes to complete:ss_aci_creategocdNote: I’ve changed the keys to this service principal and deleted this storage account, so I using these Service Principal or Storage Account keys won’t work!
  7. Once completed you will be able to log in to the Azure Portal and find the newly created Resource Groups:ss_aci_resourcegroup
  8. Open the resource group *gocdacirg and then select the container group *gocdaci:ss_aci_getcontainerip
  9. The IP Address of the container is displayed. You can copy this and paste it into a browser window along with the port the container exposed. In the case of Go CD it is 8153:ss_aci_runninggocdserver
  10. The process is now completed.

The Azure Container Instance can now be deleted and recreated at will, to reduce cost or simply upgrade to a new version. The Azure File Share will persist the data stored by the container into the mounted volume:

ss_aci_storageexplorerfileshare

Hopefully this process will help you implement persisted storage containers in Azure Container Instances more easily and quickly.

Thanks for reading!