Azure Automation Hybrid Runbook Workers: A First Look

Azure Automation is Microsoft’s cloud-based workflow engine that executes PowerShell-based runbooks to orchestrate complex, repetitive, or time-consuming tasks. From its release until now, your options for automation using the tool were limited, because it could only operate against Azure cloud resources. It began life as a tool for automating virtual machine or website provisioning in Azure, but not for managing your existing on-premises infrastructure.

That story has begun to change with Microsoft’s release of a new feature for Azure Automation named Hybrid Runbook Workers. Think of it as a server or (highly-available) group of servers in your datacenter that act on behalf of your cloud runbooks, performing tasks locally just as you would run a script or System Center Orchestrator runbook on-premises. This is an exciting and important advance that opens up a variety of scenarios, allowing you to reach into your existing infrastructure anywhere from a highly-reliable, secure, and centralized location, creating a truer “hybrid” cloud/on-premises infrastructure. This diagram from the documentation visualizes the architecture:

 

 

In this post we’ll kick the tires on this new feature and walk through the setup experience from scratch to getting our first local runbook execution from an Azure subscription in the cloud.

Azure Subscription and Automation Account

Azure Automation has the concept of an “account” which contains our runbooks and the data they use. It is also the place from which we’ll set up our Hybrid Runbook Worker server. Assuming you have an Azure subscription in hand, head into the Azure Portal to create a new Automation Account. Once created you’ll find it in the Automation Accounts section of the resource list:

 

 

Operational Insights Account Setup

Somewhat confusingly, the Hybrid Runbook Worker is implemented as an agent of the Operational Insights service within Azure, a.k.a. the Operation Management Suite (OMS). Executing runbooks as a Hybrid Runbook Worker is just one of the various things the agent can do, by way of “intelligence packs” that are selected for deployment to it. In other words, you start with an Operational Insights agent, and then make it into a Hybrid Runbook Worker by pushing the Automation pack to it. This makes the setup more difficult to wrap one’s head around, even if it fits within a larger strategy. The actual procedure isn’t too bad though, as you’ll see.

First, we need to add Operational Insights to our Azure subscription in the form of a workspace. The workspace starts as free, so it won’t increasse the bill to get started. As of this writing, you can’t create an Operational Insights workspace from the “new” Azure portal, so head over to the “full” old one for this. Make sure to select a globally unique name.

 

 

Once created, open the workspace in the portal and look for the “Manage” button. Click this link to go to the workspace management page, which is yet another portal experience (Microsoft Operations Management Suite) separate from the other two Azure portals.

 

 

Inside the workspace, we add the Azure Automation component. This may be referred to in the documentation as an “intelligence pack”, but appears under the “Solutions Gallery” tile in the OMS page. From the list of solutions, find Automation and add it.

 

 

 

After it’s added, you’ll see a new tile on the home screen of the OMS portal. Open it and use the Configure button at the bottom of the page to ensure the correct Automation Account is selected.

 

 

With that, the service side of the equation is in place. Now we need an agent to become our Hybrid Runbook Worker.

Installing the Agent

We’ll be using the manually installed agent for now. You’ll notice the ability to connect to a System Center Operations Manager instance, which is related to the other features of Operational Insights that analyze data from SCOM and other log sources. For purposes of automation, we will ignore this aspect, except to recognize that the agent we’re installing is actually the same as the SCOM agent, so it won’t allow itself to be installed on a server where an existing SCOM agent lives. I didn’t look at the SCOM integration for this post, and decided to create a fresh, dedicated on-premises Windows Server 2012 R2 server to become the Hybrid Runbook Worker.

When you download and install the Windows agent, you’ll be prompted for the Workspace ID and Key that come from the same settings page as the download link:

 

 

 

 

When installation completes, you will find a new Control Panel entry if you need to update the settings:

 

 

 

At this point, we’ve pointed our new agent up to our cloud workspace, which serves as its management server. Assuming all goes well, we should see it soon connect and start asking for and receiving any assigned management/intelligence packs that have been added to the workspace. In our case, we want to see the Azure Automation pack appear. We can watch for this in the newly-created Operations Manager event log:

 

 

This pack transforms the newborn agent into a useful member of society, otherwise known as a Hybrid Runbook Worker. If successful, you should now be able to find a new folder named “AzureAutomationFiles” under the installation directory, by default C:\Program Files\Microsoft Monitoring Agent\Agent.

Once downloaded, the folder should contain a new PowerShell module, “HybridRegistration”. We use this to register the Hybrid Runbook Worker with the automation account, letting it know the new agent is alive and ready to receive runbook jobs. To accomplish this, you’ll need to run a bit of PowerShell from an elevated console or ISE window (run as administrator required). The below shows example values, but you’ll need to update with your own:

Import-Module HybridRegistration

$settings = @{
    Name = "OnPremLabRBWG1"
    EndPoint = "https://eus2-agentservice-prod-1.azure-automation.net/accounts/bd12f612-91bc-4119-802c-5b6cd8841fc8"
    Token = "AjAoGmqk8drLWVIkBJTRedsfTUIRZlY2b/iJnTHPMFPUdTAm397Ry0ROauNPXsEzvoPpKf4w89T6VC9cxrFQ=="
}

Add-HybridRunbookWorker @settings

 

Here, “Name” is any name you want to use for your Hybrid Runbook Worker group. This is the name of a new group of Hybrid Runbook Worker servers, not just this one, which will be the group’s first member. “EndPoint” and “Token” values come from your Azure Automation account, which when accessed through the new Azure portal, contains a key icon revealing a “URL” (EndPoint) and a “Primary Access Key” (Token).

 

 

When you run the PowerShell commands above with your updated values, you should see some console windows briefly flash by, but unfortunately no output or result displayed even if things worked correctly. Hopefully Microsoft will fix this or provide a simple utility to do this setup.

How do you know it worked? You should soon see a Hybrid Runbook Worker entry appear from the list you see when clicking the Hybrid Workers tile in your Automation Account view.

 

 

You should also see two new Application logs in the event viewer (after re-opening). The “Microsoft-SMA” operational log shows events related to the Hybrid Runbook Worker connecting to the Azure service endpoint.

 

 

Without too much pain, we’ve installed ourselves a fancy new Hybrid Runbook Worker server that is connected to an Azure datacenter in a far-off land, ready to do our bidding! You may be asking, don’t I have to go groveling to the network and security teams to allow an exception through the firewall? Not in this case, unless your infrastructure blocks outbound HTTPS by default, which is what the agent uses for its one-way communication to the cloud. All communication is initiated locally, so no inbound access is required.

Invoking Azure Runbook Locally

Now let’s kick the tires and see what this thing can do. For testing purposes, I chose the very exciting scenario of copying files between two file shares in my datacenter. Because these files are hypothetically very large, I don’t want them going up to the cloud and back – they should copy locally, and the Hybrid Runbook Worker is perfect for that. So we will have a runbook defined in Azure that asks for the source and destination path, which we run from Azure (imagine as part of a larger process workflow, or on a schedule). Once run, the Hybrid Runbook Worker will immediately notice the new runbook job in Azure, then download and execute it, reporting the results back to Azure when finished.

So, I start with a local server with a file share containing some files. My goal will be to copy these files to a second local server file share.

 

 

A Simple File Copy Runbook

We’ll need to create a runbook that copies files. This is pretty straightforward, as a single PowerShell cmdlet, Copy-Item, does what we want. We just wrap this in the required PowerShell workflow structure, and save/publish. I won’t cover runbook creation here, but you can read more about that here or look at this larger example.

My sample runbook, “Copy-LocalFiles”, contains the following:

 

workflow Copy-LocalFiles
{
	Param
	(
		[parameter(Mandatory=$true)]
		[String] $SourcePath,

		[parameter(Mandatory=$true)]
		[String] $DestinationPath
 	)
 
 	Write-Output "Executing runbook on hybrid runbook worker: $env:ComputerName"
	Write-Output "Source: $SourcePath"
	Write-Output "Destination: $DestinationPath"
	$result = InlineScript
 	{
  		try
  		{
   			Copy-Item -Path "$using:SourcePath" -Destination "$using:DestinationPath"
  		}
  		catch
  		{
   			$errorMessage = $error[0].Exception.Message
  		}
  
 		if($errorMessage -eq $null)
		{
			return "Successfully copied files"
		}
		else
		{
			return "Failed: Encountered error(s) while copying files. Error message=[$errorMessage]"
		}
 	}
 
	Write-Output $result
	Write-Output "Execution finished"
}

 

After I save and publish, I can click the Start button. I am then prompted to supply a source and destination path, and choose where I want to run the runbook. I should see my new Hybrid Runbook Worker group as a choice here. Clicking OK, the job is immediately queued.

 

 

Essentially immediately, we can see the Hybrid Runbook Worker pick up the job, as recorded in an event log 15007 message. It then spawns a new “Sandbox” in which the downloaded runbook is executed. You should soon see that process appear as “Orchestrator.Sandbox”.

 

 

 

 

After a short time, the runbook finishes and reports its status back to Azure, where we should see the output produced by the script.

 

 

And what about the target share? Files copied!

 

 

Pitfalls to Avoid

OK, the above didn’t exactly go as smoothly as presented. I ran into a couple issues that are worth watching out for. Here they are, free of charge.

PowerShell Execution Policy Bug?

When first attempting to start runbooks, I was seeing an exception thrown by the Hybrid Runbook Worker when trying to create the sandbox. The error event 3712 reported “An unhandled exception has reached the sandbox’s main entry point. The sandbox will exit immediately…exceptionMessage=System.Security.SecurityException: Security error…The Zone of the assembly that failed was: MyComputer”. After troubleshooting, I was able to identify the cause as the Group Policy setting “Turn on Script Execution” in my domain that configured domain-wide PowerShell execution policy to be Unrestricted. Something about the application of the policy itself causes the security exception, and the only workaround I’ve found so far is to prevent the Group Policy setting from applying to the Hybrid Runbook Worker server. Hopefully Microsoft will address this in an update.

 

 

The setting is configured here in Group Policy:

 

 

Constrained Delegation of Hybrid Runbook Worker Computer Account

When the Hybrid Runbook Worker agent is installed, it takes the form of the Microsoft Monitoring Agent service, which runs under the Local System account. This means that runbooks will execute with the identity of the Hybrid Runbook Worker computer when talking to other systems on the network. A nice advantage to this is that the runbook doesn’t need to know a local domain username and password – indeed, we’d rather not have credentials for our infrastructure floating around in the cloud if such can be avoided. Whenever the runbook encounters an authentication challenge with any of the Windows authentication protocols, it should be able to seamlessly authenticate as the computer, assuming the Hybrid Runbook Worker computer account is given the required permissions on protected resources.

 

 

In my case, I had two file shares, so I granted my Hybrid Runbook Worker computer, HW1, permission to both shares. Yet I was still seeing “access denied” when trying to read the shares. I ended up finding a workaround in this case to be configuring the Hybrid Runbook Worker computer account to allow Kerberos delegation. Once this was configured, authentication worked as expected. You can read more about this “most confusing dialog box in Active Directory”. For different scenarios, it may be necessary to configure the third option to allow additional protocols.

Another possibility is configuring the service to run as a service account rather than Local System. Perhaps Microsoft will provide additional guidance on this approach going forward.

Configure by opening the computer account properties in Active Directory Users and Computers:

 

 

Wrapping Up

The Hybrid Runbook Worker addition is an important and powerful capability that will go a long way to making Azure Automation a platform for orchestrating workloads across the datacenter and the cloud. And it’s only one of several new features including graphical runbook authoring and webhooks, which present an external API for invoking runbooks from outside Azure. With these components, the reach of Azure Automation is effectively unlimited.

In future posts I’ll look at more of these features and ways to leverage them for automation goodness.

What do you think? Let me know in the comments_.