In the ever-evolving landscape of cloud computing, efficiency, and automation are not just buzzwords but essential strategies for managing complex cloud environments. Azure Automation emerges as a pivotal tool in this realm, offering a robust, cloud-based automation service that helps you focus on work that adds business value. By automating frequent, time-consuming, and error-prone cloud management tasks, Azure Automation enables you to reduce operational overhead, ensure consistency across your environments, and implement scalable, reliable automation solutions.

One of the key features of Azure Automation is the use of Hybrid Runbook Workers. These workers allow you to run automation tasks securely and flexibly not just within Azure but also in your on-premises environments, extending the reach and capability of your cloud automation. This feature is handy for scenarios where you need to interact with local resources or manage Azure resources from a local context.

Understanding Azure Automation and Hybrid Runbook Workers

Azure Automation is a cloud-based service within Microsoft Azure that provides the capability to automate repetitive and complex tasks across Azure and non-Azure environments. This service helps in orchestrating processes using Runbooks — scripts written in PowerShell or Python — to manage resources and automate workflows. The primary goal of Azure Automation is to improve operational efficiency, reduce errors, and maintain consistency across environments by automating tasks that are typically performed manually.

Key features of Azure Automation include:

• Process Automation: Automate frequent, time-consuming, and error-prone management tasks.
• Update Management: Manage updates and patches for your VMs.
• Configuration Management: Ensure desired state configurations for services and environments.
• Inventory: Track changes and inventory in your environment for visibility and control.

Hybrid Runbook Workers: Extending Automation Beyond Azure

Hybrid Runbook Workers play a crucial role in the Azure Automation ecosystem. They are feature components of Azure Automation that allow you to run Runbooks directly on the machine hosting the role and against resources in the environment to manage local resources. Hybrid Runbook Workers can be deployed in different environments, including on-premises data centers or other cloud environments, not just Azure.

Advantages of using Hybrid Runbook Workers include:

• Flexibility: Execute automation tasks across both Azure and non-Azure environments.
• Security: Perform tasks on machines without exposing them to the public internet.
• Direct Interaction: Manage local resources and execute tasks that require direct access to on-premises resources.

Utilizing Hybrid Runbook Workers for VM and Data Management

Hybrid Runbook Workers are particularly effective for scenarios involving virtual machines (VMs) and data handling. Here’s how they can be leveraged:

1. Managing VMs: Hybrid Runbook Workers can start, stop, monitor, and manage VMs in Azure and on-premises. This capability is crucial for scenarios where VMs need to be dynamically managed based on workload, time, or usage, ensuring efficient use of resources.
2. Handling Data: With the ability to run scripts directly on a machine, Hybrid Runbook Workers can be used to automate data transfer tasks, such as using AzCopy for moving data between Azure Storage accounts. This is particularly useful for backup, replication, or migration tasks where large amounts of data need to be moved securely and efficiently.

By understanding and utilizing Azure Automation and Hybrid Runbook Workers, organizations can significantly streamline their cloud management tasks, ensuring a more secure, efficient, and cost-effective cloud environment.

Real-world use case of Hybrid Runbook

In this blog, we’ll dive into a practical, real-world application of Azure Automation, harnessing the power of Hybrid Runbook Workers. We’ll explore a three-step automated process to streamline your virtual machine (VM) management and data handling tasks. The process involves:

1. Starting an Azure VM: Automating the VM start-up process, and preparing the environment for subsequent tasks.
2. Executing AzCopy with PowerShell: Utilizing a PowerShell script within a runbook to transfer files between Azure Storage accounts using AzCopy – a command-line utility optimized for speed and reliability.
3. Shutting Down the VM: Gracefully shutting down the VM after the data transfer, ensuring cost-efficiency and resource optimization.

This sequence demonstrates not only the versatility and power of Azure Automation but also how it can be tailored to specific, practical needs in managing cloud resources. Whether you’re new to Azure Automation or looking to expand your existing knowledge, this guide aims to provide valuable insights into creating a more efficient, automated cloud environment.

Where this scenario can be useful?

To demonstrate the Azure Hybrid runbook scenario we will understand a problem scenario where a hybrid runbook could be useful.. Here we have two data lakes. Private Data Lake and Public Data Lake. The private data lake is only accessible from inside a VNet. This VNet has multiple subnets hosting private endpoints for Azure services (Azure Synapse) and a dedicated subnet for Virtual machines. Services deployed inside VNets can access the Private Data lake. While public data lake is accessed from the internet. We have created this setup here because the Reltio data dumping process can only access the ADLS data lake exposed to the public internet. Now in this scenario, if you want to copy the data from a public data lake to a private data lake there is no connectivity between these data lakes. Only the VM which is hosted in the workstation subnet can access the Private data lake. This VM is connected to the internet so this VM can connect to a public data lake as well. So we will use this VM to copy the data from the Public lake to the Private lake.

Setting the Stage: Preparing Your Azure Environment

To leverage the full potential of Azure Automation and Hybrid Runbook Workers, it’s essential to set up and configure your Azure environment properly. This preparation involves creating an Azure Automation account, configuring Hybrid Runbook Workers, and ensuring your Azure Virtual Machines (VMs) are ready for automation tasks. Here’s a step-by-step guide to get you started:

1. Setting Up an Azure Automation Account

An Azure Automation account is the foundational element where all your automation components and resources will reside. To create one:

1. Log in to the Azure Portal: Start by signing into your Azure Portal.
2. Create a New Automation Account:
   • Navigate to All Services and select Automation Accounts.
   • Click on + Add to create a new account.
   • Fill in the details such as name, subscription, resource group, and location. Choose a name that reflects the purpose of the account for easier management.
3. Grant contributor permission to subscription
   • After creation, grant the contributor permission to the subscription.

Grant necessary permission to system identity of Automation account to subscription. Or if you do not want to give permission to the entire subscription you can grant to specific resources.

Since AZCopy will be executed from the VM grant the VM identity storage blob data contributor permission to the storage accounts.

2. Configuring the Environment for Hybrid Runbook Workers

Once your Automation account is set up, the next step is to configure Hybrid Runbook Workers:

1. Install the Hybrid Runbook Worker:
   • On a machine that you want to use as a Hybrid Runbook Worker, ensure it meets the prerequisites in terms of OS, PowerShell version, and network connectivity. In our use case, I have downloaded AzCopy v10 and copied it into a folder path c:\runbook1
2. In the Azure Portal, under your Automation account, navigate to the Hybrid Worker Groups section and follow the instructions to add a new Hybrid Runbook Worker to your environment.

After creating the hybrid worker it will look like this:

3. Preparing Azure VMs for Automation Tasks

To ensure your Azure VMs are ready for automation:

1. VM Configuration:
   • Verify that the VMs you intend to manage are running and configured correctly. This includes checking the OS, installed applications, network settings, and security configurations.
   • Ensure that the VMs have the necessary modules and tools installed, such as Azure PowerShell modules and AzCopy, if they are required by your runbooks.
2. Network and Security Settings:
   • Configure the network settings to allow communication between the Azure Automation service and your VMs. This might involve setting up Network Security Group (NSG) rules or configuring firewalls.
   • Ensure that the necessary ports and endpoints are open for communication and that your security settings align with your organization’s policies.
3. Testing Connectivity:
   • Before deploying automation tasks, test the connectivity and configuration by running a simple test runbook. This helps validate that the Hybrid Runbook Workers can successfully communicate with and manage the VMs.

I have copied this PowerShell code to a folder in the VM

---Powershell code-----
# Define the command to run
$cmd = @"
c:\runbook1\azcopy.exe login --identity 
c:\runbook1\azcopy.exe cp "https://publicadls.blob.core.windows.net/inbound/export/" --recursive=true "https://privateadls.blob.core.windows.net/testing/azcopy_test/" --overwrite=true
"@

# Execute the command
$output=Invoke-Expression $cmd

Write-Host Invoke Command Completed!!
$output

I have used this code to test if the automation account can connect remotely to the VM and execute the PowerShell code deployed in the VM:

#Runbook code
 
$ScriptToRun = "C:\runbook1\azcopyScript.ps1"
Out-File -InputObject $ScriptToRun -FilePath ScriptToRun.ps1 
Connect-AzAccount -Identity
Set-AzContext -Subscription "XXXXX"
$result = Invoke-AzVMRunCommand -ResourceGroupName 'RG-1' -VMName 'vm1' -CommandId 'RunPowerShellScript' -ScriptPath ScriptToRun.ps1 
$result.Value[0].Message
Remove-Item -Path ScriptToRun.ps1

Now you need to test the runbook by selecting the hybrid worker group created earlier.

Now you have already created the runbook and it has been tested successfully. We will add three code snippets:

1. Code to start the VM
2. Invoke the PowerShell code on the remote VM.
3. Shutdown the VM

Here is the entire code:

Master runbook

# Ensure that the runbook does not inherit an AzContext
Disable-AzContextAutosave -Scope Process

# Connect to Azure with system-assigned managed identity
$AzureContext = (Connect-AzAccount -Identity).context

# set and store context
$AzureContext = Set-AzContext -SubscriptionName $AzureContext.Subscription -DefaultProfile $AzureContext

 

Start-AzAutomationRunbook `
    -AutomationAccountName 'TestHybridAccount' `
    -Name 'startVM' `
    -ResourceGroupName 'RG-1' `
    -DefaultProfile $AzureContext `
    -Wait

write-host "VM Started...."    

Start-Sleep -Seconds 60

write-host "Copy is going to start"
Start-AzAutomationRunbook `
    -AutomationAccountName 'TestHybridAccount' `
    -Name 'Azcopy_in_Private_ADLS' `
    -ResourceGroupName 'RG-1' `
    -DefaultProfile $AzureContext `
    -Wait
    
     
 
write-host "VM stopping...."
Start-AzAutomationRunbook `
    -AutomationAccountName 'TestHybridAccount' `
    -Name 'stopVM' `
    -ResourceGroupName 'RG-1' `
    -DefaultProfile $AzureContext `
    -Wait

Here are the two automation runbooks one for starting the VM and one for shutting down the VM. These runbooks are called by the Master runbook code provided above.

-----------------------
Connect-AzAccount -Identity
Set-AzContext -Subscription "xxx"

#Start the VM
Start-AzVM -Name "vm1" -ResourceGroupName "RG-1"
 
-------------------------


Connect-AzAccount -Identity
Set-AzContext -Subscription "xxx"

#Stop the VM
Stop-AzVM -Name "vm1" -ResourceGroupName "RG-1" -force

If you want to learn more about Azure Automation please refer to this blog post this blog post

Conclusion

Azure Automation, with its Hybrid Runbook Workers, stands out as a powerful and versatile tool in the cloud computing landscape. This blog has illustrated how Azure Automation can be effectively used to automate and streamline complex tasks across both Azure and non-Azure environments. The use of Hybrid Runbook Workers enables organizations to extend their automation capabilities beyond the Azure cloud, reaching into on-premises data centers and other cloud platforms. This flexibility is particularly beneficial in scenarios requiring direct interaction with local resources or managing Azure resources from a local context.

The practical application of this technology, as demonstrated through the VM and data management scenario, underscores its utility in real-world situations. By automating tasks like VM start-up, data transfer using AzCopy, and VM shutdown, Azure Automation ensures operational efficiency, security, and cost-effectiveness. The ability to manage and automate these tasks across diverse environments not only saves time and reduces errors but also enhances the overall agility of IT operations. By leveraging these tools, organizations can achieve greater control, efficiency, and flexibility in their cloud operations, ultimately driving better business outcomes. Whether you are new to Azure Automation or looking to enhance your existing setup, this guide provides a solid foundation for exploring the vast potential of cloud automation.